To interact with this page you must login.
Signup
technology
understand
What are some funny examples of robots.txt files?
Facebook has boring messages, for example.
technologyunderstand
technologyunderstand
Compesh is a question and answer (and debate) website, so before you make a debate, you better learn what an assertion is. I suppose you already know what a question is, and that you've typed it in the box. ;)
An assertion, is basically a statement you can make, that is either true or false.
Richer people have better health.
The question for that would be, Do richer people have better health?
And don't forget to make your assertion, match your question.
The idea is that every website should have a humans.txt for humans to read instead of crawling robots. The website also gives you examples in the wild.
http://humanstxt.org/