Reputation: 15355
is there any point in having a robots.txt with just one line of content like below
User-agent: *
compared to not having it at all ?
Upvotes: 1
Views: 253
Reputation: 12230
No, google (and let's not beat around the bush, Google is the search engine you should care about) by default will crawl everything unless you tell it not to.
If you want to help Google, use google webmaster tools and submit a sitemap in the correct format
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184
Upvotes: 0