Monday, January 30

Fun with robots.txt

I cut off Robots.txt.

For those of you that don't know what I am talking about, every crawling bot in the world is supposed to adhear to a file in your directory called "robots.txt" which will allow you to tell all the indexers what to access, and what not to access.

I cut them all off. No Search engine may now index my website at all. Of course I'll get less hits. But it should be a good experiment to see just how much traffic to my website 500-600+ hits a day) is search engine generated traffic.

No comments: