My logs show attempts by GoogleBot et al to access a robots.txt file so I've decided not to disappoint them any more and have provided them with this:
User-agent: * Disallow:
This is telling them to scan away, I'm open.
Googling for robots.txt it is interesting to see the fifth entry is for http://www.whitehouse.gov/robots.txt i.e. the white house's robots.txt file. It seems to do a lot of disallowing. I hope this isn't going to be interpreted as political commentry.
I didn't need a mod_rewrite rule to get this working, it Just Worked.
Peter's Moblog
