This operation is performed in accordance with an international agreement between BnF and the Library of Congress concerning official publications.
BnF uses a spider called Heritrix (http://crawler.archive.org) to harvest websites.
The robot’s identification field is "User-Agent : Mozilla/5.0 (compatible; bnf.fr_bot)".
It does not follow exclusion rules specified in the robots.txt protocol. However it always applies high politeness rules (delays between two requests) in order not to stress the producers’ servers.
If despite this the performance of your website is affected by this operation, please report it by email to firstname.lastname@example.org. We will propose a solution as soon as possible.
Tuesday, May 31, 2011
For more info
ContactCourriel : email@example.com