Everything You Ever Wanted To Know About Robots.txt

…But were afraid to ask, or didn’t care enough to… Google has written a few posts today concerning robots.txt that should give you all the information you could possibly need about this subject.

First, this post contains helpful links about robots.txt and how Google actually deals with the existence of such a file.

Vanessa Fox also penned a comprehensive post about robots.txt to continue with the theme. At this point I’m thinking a “Why I love my Robots.txt File” would be quite a fun contest for them to add.

Personally, I’ve long been a fan of the robots.txt file. The crawlers that actually follow the directives are much better trained with some exclusions than if you just let them run amok with no special commands.

If you haven’t boned up lately on your robots.txt, you’re about to embark on an exciting adventure.