Python: Sorting ip ranges which are dictionary keys

Question

I have a dictionary which has IP address ranges as Keys (used to de-duplicate in a previous step) and certain objects as values. Here's an example

Part of the dictionary sresult:

10.102.152.64-10.102.152.95 object1:object3
10.102.158.0-10.102.158.255 object2:object5:object4
10.102.158.0-10.102.158.31  object3:object4
10.102.159.0-10.102.255.255 object6

There are tens of thousands of lines, I want to sort (correctly) by IP address in keys

I tried splitting the key based on the range separator - to get a single IP address that can be sorted as follows:

ips={}
for key in sresult:
    if '-' in key:
        l = key.split('-')[0]
        ips[l] = key
    else:
        ips[1] = key

And then using code found on another post, sorting by IP address and then looking up the values in the original dictionary:

sips = sorted(ipaddress.ip_address(line.strip()) for line in ips)
for x in sips:
    print("SRC: "+ips[str(x)], "OBJECT: "+" :".join(list(set(sresult[ips[str(x)]]))), sep=",")

The problem I have encountered is that when I split the original range and add the sorted first IPs as new keys in another dictionary, I de-duplicate again losing lines of data - lines 2 & 3 in the example

 line 1 10.102.152.64 -10.102.152.95
 line 2 10.102.158.0  -10.102.158.255
 line 3 10.102.158.0  -10.102.158.31
 line 4 10.102.159.0  -10.102.255.25

becomes

line 1 10.102.152.64 -10.102.152.95
line 3 10.102.158.0  -10.102.158.31
line 4 10.102.159.0  -10.102.255.25

So upon rebuilding the original dictionary using the IP address sorted keys, I have lost data

Can anyone help please?

Answer 1

EDIT This post now consists of three parts:

1) A bit of information about dictionaries that you will need in order to understand the rest. 2) An analysis of your code, and how you could fix it without using any other Python features. 3) What I would consider the best solution to the problem, in detail.

1) Dictionaries

Python dictionaries are not ordered. If I have a dictionary like this:

dictionary = {"one": 1, "two": 2}

And I loop through dictionary.items(), I could get "one": 1 first, or I could get "two": 2 first. I don't know.

Every Python dictionary implicitly has two lists associated with it: a list of it's keys and a list of its values. You can get them list this:

print(list(dictionary.keys()))
print(list(dictionary.values()))

These lists do have an ordering. So they can be sorted. Of course, doing so won't change the original dictionary, however.

Your Code

What you realised is that in your case you only want to sort according to the first IP address in your dictionaries keys. Therefore, the strategy that you adopted is roughly as follows:

1) Build a new dictionary, where the keys are only this first part. 2) Get that list of keys from the dictionary. 3) Sort that list of keys. 4) Query the original dictionary for the values.

This approach will, as you noticed, fail at step 1. Because as soon as you made the new dictionary with truncated keys, you will have lost the ability to differentiate between some keys that were only different at the end. Every dictionary key must be unique.

A better strategy would be:

1) Build a function which can represent you "full" ip addresses with as an ip_address object.

2) Sort the list of dictionary keys (original dictionary, don't make a new one).

3) Query the dictionary in order.

Let's look at how we could change your code to implement step 1.

def represent(full_ip):
    if '-' in full_ip:
        # Stylistic note, never use o or l as variable names.
        # They look just like 0 and 1.
        first_part = full_ip.split('-')[0]
        return ipaddress.ip_address(first_part.strip())

Now that we have a way to represent the full IP addresses, we can sort them according to this shortened version, without having to actually change the keys at all. All we have to do is tell Python's sorted method how we want the key to be represented, using the key parameter (NB, this key parameter has nothing to do with key in a dictionary. They just both happened to be called key.):

# Another stylistic note, always use .keys() when looping over dictionary keys. Explicit is better than implicit.

sips = sorted(sresults.keys(), key=represent)

And if this ipaddress library works, there should be no problems up to here. The remainder of your code you can use as is.

Part 3 The best solution

Whenever you are dealing with sorting something, it's always easiest to think about a much simpler problem: given two items, how would I compare them? Python gives us a way to do this. What we have to do is implement two data model methods called

__le__

and

__eq__

Let's try doing that:

class IPAddress:
    def __init__(self, ip_address):
        self.ip_address = ip_address # This will be the full IP address

    def __le__(self, other):
        """ Is this object less than or equal to the other one?"""
        # First, let's find the first parts of the ip addresses
        this_first_ip = self.ip_address.split("-")[0]
        other_first_ip = other.ip_address.split("-")[0]
        # Now let's put them into the external library
        this_object = ipaddress.ip_address(this_first_ip)
        other_object = ipaddress.ip_adress(other_first_ip)
        return this_object <= other_object

    def __eq__(self, other):
        """Are the two objects equal?"""
        return self.ip_address == other.ip_adress

Cool, we have a class. Now, the data model methods will automatically be invoked any time I use "<" or "<=" or "==". Let's check that it is working:

test_ip_1 = IPAddress("10.102.152.64-10.102.152.95")
test_ip_2 = IPAddress("10.102.158.0-10.102.158.255")

print(test_ip_1 <= test_ip_2)

Now, the beauty of these data model methods is that Pythons "sort" and "sorted" will use them as well:

dictionary_keys = sresult.keys()
dictionary_key_objects = [IPAddress(key) for key in dictionary_keys]
sorted_dictionary_key_objects = sorted(dictionary_key_objects)
# According to you latest comment, the line below is what you are missing
sorted_dictionary_keys = [object.ip_address for object in sorted_dictionary_key_objects]

And now you can do:

for key in sorted_dictionary_keys:
    print(key)
    print(sresults[key])

The Python data model is almost the defining feature of Python. I'd recommend reading about it.