CODEWITHSUNDEEP
.htaccessrobots.txt
Ahmed Rashid
Ahmed Rashid

Reputation: 19

How to disallow all files & folder and allow for some in robots.txt?

I need robots.txt commends doing the next:

  1. disallow all files and folders
  2. allow just to index.php & msub.php

I have edit the links for msub.php i did RewriteReuls from .htaccess to be like:

domain.com/p/subject122

domain.com/p/subject104

So i want make the links show up in search engines just like: domain.com/p/subject122 .

How can i do that?

I have tried this commands but not works as well.. The links : domain.com/p/subject104 not show up in search engines

User-agent: *
Disallow: /
Allow: /index.php
Allow: /msub.php

.htaccess

Options -Indexes

RewriteEngine On
RewriteBase /

RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://www.example.com/$1 [R,L]

RewriteCond %{REQUEST_URI} ^/404/$
RewriteRule ^(.*)$ /error.html [L]

RewriteRule home /index.php 
RewriteRule p/(.*)$ msub.php?page=$1

Upvotes: 1

Views: 709

Answers (1)

anubhava
anubhava

Reputation: 785058

robots.txt should contain URIs that are seen by crawlers, no matter how you handle them internally. So your robots.txt should have:

User-agent: *
Disallow: /
Allow: /p/subject122
Allow: /p/subject104

Your .htaccess can be refactored as well:

Options -Indexes

RewriteEngine On
RewriteBase /

RewriteCond %{SERVER_PORT} 80 
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,NE,L]

RewriteRule ^404/?$$ error.html [L,NC]

RewriteRule ^home/?$ index.php [L,NC]

RewriteRule p/(.*)$ msub.php?page=$1 [L,NC,QSA]

Upvotes: 1

Related Questions