CODEWITHSUNDEEP
wordpressrobots.txt
mAsT3RpEE
mAsT3RpEE

Reputation: 1838

How to set up robots.txt file for WordPress

[UPDATE 2013]

I can't find an authoritative page with a format for robots.txt file for WordPress. I promise to maintain one on my site but I want one here on stack overflow.

If you know what your doing please check current draft here:

http://mast3rpee.tk/?p=127

Everyone else comment on this:

robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Sitemap: http://domain.com/sitemap.xml
Crawl-delay: 4

Upvotes: 5

Views: 30690

Answers (4)

Dan Knauss
Dan Knauss

Reputation: 11

It's not safe to block much in your robots.txt nowadays single Google tries to load all assets to determine "mobile friendliness." At minimum you can block /wp-admin. Here's a more detailed, current answer to the question at the StackExchange forum for WordPress.

Upvotes: 0

kdev
kdev

Reputation: 2767

As with all things SEO, things change. I think that the current advice is to have a a very minimal robots.txt file.

Ignoring wp-admin, wp-includes, wp-content, etc. may prevent Google from rendering pages correctly, which it doesn't like.

Check out this article by Yoast: https://yoast.com/wordpress-robots-txt-example/.

Upvotes: 1

Waqas Ahmed
Waqas Ahmed

Reputation: 300

User-agent: *
Allow: /
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /e/
Disallow: /show-error-*
Disallow: /xmlrpc.php
Disallow: /trackback/
Disallow: /comment-page-
Allow: /wp-content/uploads/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Sitemap: http://yoursite.com/sitemap.xml

I think this code is very authentic for robots.txt file, Just go to Public_HTML and create file with robots.txt and paste above code. You can make in your Notepad, just copy above code and paste into notpad but remember file name should robots.txt and upload to your public_HTML.

Upvotes: 10

inactive user
inactive user

Reputation: 1

Create in notepad robots.txt and upload it to public_html in CPANEL .

*remember rename your file notepad to robots before you upload it to public_html

Upvotes: 0

Related Questions