Reputation: 396
I want to list all subdomain of a given second level domain.top level domain (2LD.1LD) , such as .edu.uk
, or .gov.us
by python
I use dns
package in python
import dns.resolver
answers = dns.resolver.query('stackoverflow.com', 'MX')
for rdata in answers:
print('Host', rdata.exchange, 'has preference', rdata.preference)
But it can't work with dns.resolver.query('edu.uk', 'MX')
I also use googlesearch
from googlesearch import search
res = search("site:edu.uk",num_results=1000)
print((res))
But it only responses 100 results.
I want a result that have all subdomain (I think it's a very big result). How can I do? Thank all
Upvotes: 1
Views: 1908
Reputation: 95
Some websites have a /robot.txt to list urls for google to crawl through. If you are allowed a alternative is using burpsuite and a subdomain list to scan through them.
Upvotes: 0
Reputation: 22272
The simple answer: you can't. There is no way to query an entire domain's contents, regardless of level. Some (only a few, like the root) allow you to transfer the zone using AXFR transfers (try: dig @b.root-servers.net axfr .
if you have the bind utilities install). But in general, the DNS protocol does now allow you to always get the contents of a tree of DNS records. You can guess at some of them, for example you figure above that there might be MX
records inside some zones which is frequently true (exception example: no TLD has an MX
record). You can guess that most domains do have a www
prefix with an A
or AAAA
record, but even that can't be assured.
In short: your issue isn't with python, its with the DNS protocol itself. There is no way, in any language, to get a complete list of zone contents if the owner of that zone doesn't want you to.
[side note: DNSSEC signed zones with NSEC proof of non-existence records allow you to "walk" a zone for the names of the records, but I don't suggest you generally try to abuse this feature -- we run a DNS scanner and even we don't use this approach when looking for signed domains -- it would cause a lot of legitimate hate email]
Upvotes: 2