CODEWITHSUNDEEP
web-servicesrobots.txt
Emanuele Ciriachi
Emanuele Ciriachi

Reputation: 2274

Robots.txt: disallow a folder's name, regardless at which depth it may show up

So I have to disallow search engines from indexing our REST web service responses (it's a Sitecore website); all of them have the same name in the URL but show up at different levels in the server hierarchy, and I was wondering if I can write a "catch all" entry in our robots file or if I am doomed to write an extensive list.

Can I add something like

Disallow: */ajax/*

to catch all folders named "ajax" regardless of where they appear?

Upvotes: 1

Views: 250

Answers (2)

Marek Musielak
Marek Musielak

Reputation: 27132

robots.txt specification doesn't say anything about wildcards but Google (Google Robots.txt Specifications) and Bing allow the use of wildcards in robots.txt files.

Disallow: */ajax/*

Your disallow is valid for all the /ajax/ urls no matter what is the nesting level of /ajax/.

Upvotes: 2

Ian Graham
Ian Graham

Reputation: 3216

You should be able to just use Disallow: /*ajax. Similar question over here: How to disallow service api and multilingual urls in robots.txt

Upvotes: 1

Related Questions