janesconference
Reputation: 6473
Do import.io crawlers obey robots.txt?
When you run an import.io crawler, does it obey to the robots.txt file?
Upvotes: 0
Views: 329
Answers (2)
janesconference
Reputation: 6473
According to the faq, it turns out it does.
Do you adhere to robots.txt? Yes.
Upvotes: 3
Related Questions
- Listing extractors from import.io
- Crawling data using import IO
- Data crawler or something else
- Crawling recursively with import.io
- Is it possible to to rerun a web-crawler in import.io?
- import.io web crawler with drop downs menus
- Import.io (web-crawler) keeps querying without output
- Xpath to url for import.io
- Web Crawler - Ignore Robots.txt file?
- What rules govern in absence of robots.txt?