How can I automate Nmap scans using Python for recon while parsing and storing results in a database?

Question

I am working on a project to automate network reconnaissance using Nmap and Python. My goal is to:

• Perform Nmap scans on a range of IP addresses.
• Parse the scan results programmatically.
• Store the parsed results in a database for further analysis.

However, I’m struggling with the following:

• How to parse the scan results (which seem to be a nested dictionary) in an efficient way to extract useful information, such as open ports, services, and versions.
• The best way to structure a database schema for storing the results. Should I use SQL (e.g., SQLite/PostgreSQL) or NoSQL (e.g., MongoDB)?
• How to handle large scans efficiently if the target range is large (e.g., multiple subnets).

I used the python-nmap library to initiate an Nmap scan on a local subnet (192.168.1.0/24) using the following code:

import nmap

nm = nmap.PortScanner()
scan_result = nm.scan(hosts='192.168.1.0/24', arguments='-sV')

print(scan_result)

I expected the scan results to be returned as a dictionary containing detailed information about open ports, services, and their versions for each host in the subnet.

The Nmap scan successfully runs and returns a dictionary, but it’s quite complex and nested. I'm struggling to parse the dictionary to extract specific details such as open ports, services, and versions in an easy-to-read format.

Answer 1

The best thing to do is just start at the top level and look at the available keys and the data they contain.

At the top level, there are two keys:

>>> scan_results.keys()
dict_keys(['nmap', 'scan'])

The nmap key has global metadata about the scan:

>>> scan_results['nmap'].keys()
dict_keys(['command_line', 'scaninfo', 'scanstats'])

• command_line shows the options passed to nmap
• scaninfo shows any errors generated during the scan (in errors) and information about the scan method and ports scanned (in tcp, assuming a tcp scan)
• scanstats contains overall statistics about the scan (e.g., {'timestr': 'Tue Dec 24 14:46:23 2024', 'elapsed': '186.75', 'uphosts': '3', 'downhosts': '253', 'totalhosts': '256'})

The scan key contains the scan results:

>>> scan_results['scan'].keys()
dict_keys(['192.168.124.1', '192.168.124.62', '192.168.124.83'])

There is apparently one key for each discovered host. For each host, there are several keys:

>>> scan_results['scan']['192.168.124.62'].keys()
dict_keys(['hostnames', 'addresses', 'vendor', 'status', 'tcp'])

• hostnames and addresses have information about the hostnames and addresses associated with the host.

• vendor has information about the discovered operating system vendor, if nmap was able to determine one.

• status has the high level status for the host (up or down)

• tcp has details for the port scan of this host. There is one key per port. For example, on a host with only ssh open, we see:

  "tcp": {
    "22": {
      "state": "open",
      "reason": "syn-ack",
      "name": "ssh",
      "product": "OpenSSH",
      "version": "9.9",
      "extrainfo": "protocol 2.0",
      "conf": "10",
      "cpe": "cpe:/a:openbsd:openssh:9.9"
    }
  }
  

  Whereas on a host with two ports (ssh and a web server) we see:

  {
    "22": {
      "state": "open",
      "reason": "syn-ack",
      "name": "ssh",
      "product": "OpenSSH",
      "version": "8.7",
      "extrainfo": "protocol 2.0",
      "conf": "10",
      "cpe": "cpe:/a:openbsd:openssh:8.7"
    },
    "80": {
      "state": "open",
      "reason": "syn-ack",
      "name": "http",
      "product": "",
      "version": "",
      "extrainfo": "",
      "conf": "10",
      "cpe": ""
    }
  }
  

I expected the scan results to be returned as a dictionary containing detailed information about open ports, services, and their versions for each host in the subnet.

As we see from the above exploration, the returned information contains exactly that, in scan_results['scan']['a.b.c.d']['tcp'], where a.b.c.d is the IP address of one of the hosts in the scan.

We could iterate over the results like this:

for addr, info in scan_result["scan"].items():
    print(f"{addr} open ports:")
    for port, portinfo in info["tcp"].items():
        extra = ""
        if portinfo.get("product") and portinfo.get("version"):
            extra = f' (product={portinfo['product']}, version={portinfo['version']}'
        print(f"  - {port}{extra}")
    print()

Which for my example produces:

192.168.124.1 open ports:
  - 22 (product=OpenSSH, version=9.6
  - 53 (product=dnsmasq, version=2.90

192.168.124.62 open ports:
  - 22 (product=OpenSSH, version=9.9

192.168.124.83 open ports:
  - 22 (product=OpenSSH, version=8.7
  - 80