🔗 linkblog: my thoughts on 'The rise and fall of robots.txt'

Fascinating read on web crawlers and robots.txt

link to “The rise and fall of robots.txt”

• link
• Communities
• web crawlers
• generative AI
• OpenAI
• internet
• web scraping
• Internet Archive

You can click on the < button in the top-right of your browser window to read and write comments on this post with Hypothesis. You can read more about how I use this software here.

Any Webmentions will also be displayed below: