CODEWITHSUNDEEP
php.htaccessrobots.txt
likeforex.com
likeforex.com

Reputation: 117

can robots.txt block one type of file in a directory only?

I have a directory with 3 kind of files

1. thisdir/some/phpfile.php

2. thisdir/some/after-url-rewrite/1 (all numbers)

3. thisdir/something.htm (block any file end with .htm in thisdir/

I just want to block the 3rd type of files, since there are not existing any more, but search engine spider keep asking for them

Thanks for your help.

Upvotes: 0

Views: 83

Answers (2)

anubhava
anubhava

Reputation: 785068

If that file doesn't exist then you can just redirect all requests of that 3rd file to a valid link with 301 code.

RewriteEngine On

RewriteRule ^thisdir/something\.htm$ /thisdir/some/phpfile.php? [L,NC,R=301]

Upvotes: 0

sectus
sectus

Reputation: 15464

You could write in your robots.txt

disallow: thisdir/*.htm$

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Upvotes: 1

Related Questions