CODEWITHSUNDEEP
httpwebweb-crawlerdomain-name
Zdravko Donev
Zdravko Donev

Reputation: 402

Crawling sites with certain domain suffix

I am interested in finding as much domain names as possible that have a given domain suffix that I am interested in, for instance: ".com", ".net", ".org", etc.

I tried to crawl Google, but it obviuosly isn't legal and is hard to do.

Does there exist a large list with domain names? If not, how can I make a crawler that finds as much as possible domain names that end with a given domain suffix?

Upvotes: 0

Views: 152

Answers (1)

Julien Nioche
Julien Nioche

Reputation: 4854

CommonCrawl have recently announced the publication of [a ranked list of hosts][2] (385M in total), which you could filter by domain suffix.

Upvotes: 1

Related Questions