🔗 linkblog: my thoughts on 'The rise and fall of robots.txt'

14 February 2024 - One minute read - 15 words - kudos:

Fascinating read on web crawlers and robots.txt

link to “The rise and fall of robots.txt”

Similar Posts:

🔗 linkblog: my thoughts on 'OpenAI went back on a promise to make key documents public | Ars Technica'

🔗 linkblog: my thoughts on 'I’m sorry, but I cannot fulfill this request as it goes against OpenAI use policy - The Verge'

🔗 linkblog: my thoughts on 'AI hysteria is a distraction: algorithms already sow disinformation in Africa | Odanga Madung | The Guardian'

🔗 linkblog: my thoughts on 'Now you can block OpenAI’s web crawler - The Verge'

🔗 linkblog: my thoughts on 'Reddit Won’t Be the Same. Neither Will the Internet | WIRED'

Comments:

You can click on the < button in the top-right of your browser window to read and write comments on this post with Hypothesis. You can read more about how I use this software here.

Any Webmentions from Micro.blog will also be displayed below: