CODEWITHSUNDEEP
pythonpython-3.xdnspython-requestspython-logging
Andremoniy
Andremoniy

Reputation: 34900

How can I see in debug logs which IP address is used for connection by Python's request?

Let us say I am sending a GET request to an URL which's DNS record contains multiple IP addresses (DNS Round-Robin). For example, to the https://www.twitter.com. If we do dig of it, we see:

;; ANSWER SECTION:
twitter.com.        60  IN  A   104.244.42.193
twitter.com.        60  IN  A   104.244.42.129

I have tried to use DEBUG logging as described here: https://stackoverflow.com/a/16630836/1479414 however it does not show which IP address it uses.

Furthermore, I looked also at this answer: https://stackoverflow.com/a/22513161/1479414 about how to see the destination IP address, but it is not what I actually want to see -- this one shows me the finally used IP address.

I want rather to see which IP addresses are tried during all attempts, while the library performs DNS failover. Something like:

Trying to connect to 104.244.42.193... failed, trying to connect using another IP address

Is it possible to see somehow?

Upvotes: 3

Views: 1624

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1123440

The urllib3 library is responsible for this, in the urllib3.util.connection.create_connection() function. There's a loop there that checks each result from socket.getaddrinfo():

for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
    #  ....
    try:
        sock = socket.socket(af, socktype, proto)
        # ...
        sock.connect(sa)
        return sock

    except socket.error as e:
        # ...

Unfortunately, there is no logging used in this section. If you must have logging, you'd have to copy the function source, add in the logging you need, then monkeypatch the urllib3 code to use your version instead.

The only location that uses that function uses it as an attribute on the imported module, so at least monkeypatching is as simple as assigning it back to original function location:

import urllib3.util.connection

def create_connection_with_logging(
    address,
    timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
    source_address=None,
    socket_options=None,
):
    # ....

urllib3.util.connection.create_connection = create_connection_with_logging

Upvotes: 3

Related Questions