CODEWITHSUNDEEP
searchscrapyweb-crawler
Himanshu Bhandari
Himanshu Bhandari

Reputation: 1879

How to create a search engine, without any starting URL

I am learning scrapy to create a search engine. As per my understanding if I want to create a search engine than the first thing I need to do would be to crawl the Internet and create cache it to my servers. Now in order to crawl I need a URL through which I will go to the next, next and so on. But what if I don't have a starting URL and I want to search for what I want?

Upvotes: 0

Views: 295

Answers (2)

Himanshu Bhandari
Himanshu Bhandari

Reputation: 1879

With this question I found that You can't hide your IP address on the internet. They aren't secret.

I wrote a script and loop through IP addresses, and found website names to pass to Scrapy. That's what I needed.

Upvotes: 0

Aydin
Aydin

Reputation: 15304

If you don't have a starting URL, an idea might be to choose one. To me it sounds like you're rather lost.

To scrape content you need to make HTTP requests to other websites, you can't make HTTP requests to other websites if you don't have a URL.

It's like trying to map every road that is located around you in the physical world. If you don't specify a starting point, what can you expect?

Upvotes: 1

Related Questions