CODEWITHSUNDEEP
python
Stephen Malone
Stephen Malone

Reputation: 63

How to store and use a 300KB lookup table in a Python app?

I'm currently writing a port scanner and I'd like the program to show a description for each port in the output (e.g.: "http", "ssh", etc). I plan to use the IANA port list, but it would be ~300 kilobytes as a Python dictionary.

What are my options here? Should I just use the giant dict, or compress it in some way, or embed it as a SQLite database? I'd appreciate any and all suggestions as to how I can fit a giant data pack inside a Python program.

(Obviously the easiest solution is to just not use the IANA list, but I would like to at least try it first. Plus, I'm genuinely curious as to how this could be accomplished.)

Upvotes: 0

Views: 96

Answers (1)

kmcodes
kmcodes

Reputation: 857

There is an excellent module called klepto, which helps you to both access the data via cache and an archive for storing the data. It stores the data to a DB but uses interface like a dictionary for your uses.

On a different note, the filesize is small enough that there shouldnt be too much of a performance penalty even if you use normal python dicts and pickle them after transacting.

Upvotes: 1

Related Questions