CODEWITHSUNDEEP
codeigniterseo
Jimmathy
Jimmathy

Reputation: 327

Does robots.txt stop Google from indexing my site or the files CodeIgniter uses?

I have a site built in CodeIgniter and I am trying to utilize Google's webmaster tools which tell me to setup a robots.txt file. I want Google to index the whole site but not necessarily the files which make up the site. So I don't want good to look at the /system/ files or the /application/config/ files but I do want every page to be indexed. Should I list out each file for the Google not to index or tell it to index all or tell it to index nothing? Thanks!

Upvotes: 0

Views: 665

Answers (1)

John Conde
John Conde

Reputation: 219884

Google only see the pages/URLs your website makes available. So you don't block files, you block pages. So, your robots.txt should contain the URLs you don't want indexed. The files behind the scenes are irrelevant.

Upvotes: 3

Related Questions