CODEWITHSUNDEEP
google-crawlers
oyvey
oyvey

Reputation: 601

Crawlers visiting my cronjobs?

I have been building websites for several years now, mostly in php. Several of the sites have cronjobs, that typically run once a day. The php files that the cronjobs run are stored on the server, along with the files that deliver the site pages.

I know that various crawlers, legitimate and not, visit various pages of my sites. Now if a crawler would visit one of my cronjob files, this would activate the cronjob, sometimes with undesirable results.

I'm pretty sure that this has never happened, and, although I'm grateful for that, I'm trying to understand why. Of course there are no links anywhere to any of my cronjob url's, but I'm pretty sure that various crawlers have visited other pages even though they were never linked to.

What do other developers do to address this issue? Put a line in the robots.txt file? Set the permissions of the cronjob-relate php files?

Thanks in advance.

Upvotes: 0

Views: 110

Answers (2)

ChrisLively
ChrisLively

Reputation: 88062

Along with @Jeff's great answer:

The only way a search engine will crawl your page is if there is something linking to it. This might be another page on your site, a page on someone else's site, or your own sitemap.

Regardless your cron job should never be directly accessible from the outside.

Upvotes: 0

Jeff Busby
Jeff Busby

Reputation: 2011

Don't store any cron scripts in a publicly accessible directory.

Upvotes: 9

Related Questions