The British Library UK Web Domain Research Project

Looking through the stats on my website this morning, I noticed an entry I had not seen before — a crawler from the bl.uk domain:

Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.bl.uk/aboutus/stratpolprog/digi/domresproj/index.html)

Investigating, it seems that the British Library is attempting to "assess the feasibility of archiving the UK domain," in a project

This seems like a huge project — to capture and store *everything* stored on a website with a UK domain pointing to it.

The British Library goes on to say that:

This work is undertaken in anticipation of forthcoming Legal Deposit regulations that will make it the Library’s statutory responsibility to collect, preserve and provide long-term access to the UK’s online intellectual and cultural heritage.

Is something really the "the UK’s online intellectual and cultural heritage" simply because it has a UK domain pointing to it? I note that my primary web server is not in the UK, but has a .uk domain name, so it seems like the country in which the material is hosted is (currently) unimportant. If I were to register example.co.uk and point it to a site hosted in — say — South Korea, does this make that South Korean site part of the UK's heritage? When one considers the rules around "targeting" and making something available in another market over the Internet, the Alpenhof case sets out several factors to be considered, of which the choice of URL is perhaps one, but that perhaps overlooks that anyone can buy a domain name and point it anywhere they like, rather than needing to control the webserver to which the DNS resolves.

It is perhaps also interesting that crawling websites for content is now a key part of storing our heritage — huge swathes of information, potentially to be held on an ongoing basis, rather than a more selective harvesting of material.

I was not aware of this project, but it sounds very interesting indeed.