Sorry to say, archive.org is under a ddos attack. The data is not affected, but most services are unavailable.
We are working on it & will post updates in comments.
Brought to you by the Department of Erasing History.
TBH I can understand that it’s a problem for people who aren’t expecting it. If they disregard instructions not to index things then that’s also a problem. The only real way to prevent scrapers from replicating content is to place it behind a registration wall.
Can you elaborate on the last part?
TBH I can understand that it’s a problem for people who aren’t expecting it. If they disregard instructions not to index things then that’s also a problem. The only real way to prevent scrapers from replicating content is to place it behind a registration wall.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/
Does this answer the question?