CODEWITHSUNDEEP
web-crawlerrobots.txt
wilsonlego
wilsonlego

Reputation: 97

robots.txt - Does it work with removed pages?

I have been using my robots.txt file to excluded certain pages from being crawled and found on my website.

I have found out that there are a couple of old pages (now deleted from the server) which can still found through search engines.

Q) Would I need to recreate the page on the server so I can use robots.txt for this?

Upvotes: 0

Views: 91

Answers (1)

Jim Mischel
Jim Mischel

Reputation: 133995

No, you don't need to recreate a page in order to mention it in robots.txt. That is, you could have:

User-Agent: *
Disallow: /foo/bar.html

It doesn't matter if /foo/bar.html actually exists. Polite robots will not attempt to access that page.

Note that it won't prevent browsers from trying to access the page if they got the link from a search engine.

Removing the old pages from search engines' indexes is a more difficult problem. If they re-crawl your site, then they'll likely replace their snapshot with your new layout and the page will no longer appear. But if they find a link to your old page from some other site, they might still list the removed page in their indexes. It depends on the search engine.

For the larger search engines, you can submit removal requests. How that's done depends on the individual site.

Upvotes: 2

Related Questions