CODEWITHSUNDEEP
pythonasynchronousdnstwisted
sullivanmatt
sullivanmatt

Reputation: 740

DNS server using Python and Twisted - issues with asynchronous operation

I'm attempting to create a database-driven DNS server (specifically to handle MX records only, and pass everything else upstream) in Twisted using Python 2.7. The code below works (in terms of getting a result), but is not operating asynchronously. Instead, any DNS requests coming in block the entire program from taking any other requests until the first one has been replied to. We need this to scale, and at the moment we can't figure out where we've gone wrong. If anyone has a working example to share, or sees the issue, we'd be eternally grateful.

import settings
import db    

from twisted.names import dns, server, client, cache
from twisted.application import service, internet
from twisted.internet import defer


class DNSResolver(client.Resolver):
    def __init__(self, servers):
        client.Resolver.__init__(self, servers=servers)

    @defer.inlineCallbacks
    def _lookup_mx_records(self, hostname, timeout):

        # Check the DB to see if we handle this domain.
        mx_results = yield db.get_domain_mx_record_list(hostname)
        if mx_results:
            defer.returnValue(
                [([dns.RRHeader(hostname, dns.MX, dns.IN, settings.DNS_TTL,
                              dns.Record_MX(priority, forward, settings.DNS_TTL))
                    for forward, priority in mx_results]),
                (), ()])

        # If the hostname isn't in the DB, we forward
        # to our upstream DNS provider (8.8.8.8).
        else:
            i = yield self._lookup(hostname, dns.IN, dns.MX, timeout)
            defer.returnValue(i)

    def lookupMailExchange(self, name, timeout=None):
        """
        The twisted function which is called when an MX record lookup is requested.
        :param name: The domain name being queried for (e.g. example.org).
        :param timeout: Time in seconds to wait for the query response. (optional, default: None)
        :return: A DNS response for the record query.
        """

        return self._lookup_mx_records(name, timeout)


# App name, UID, GID to run as. (root/root for port 53 bind)
application = service.Application('db_driven_dns', 1, 1)

# Set the secondary resolver
db_dns_resolver = DNSResolver(settings.DNS_NAMESERVERS)

# Create the protocol handlers
f = server.DNSServerFactory(caches=[cache.CacheResolver()], clients=[db_dns_resolver])
p = dns.DNSDatagramProtocol(f)
f.noisy = p.noisy = False

# Register as a tcp and udp service
ret = service.MultiService()
PORT=53

for (klass, arg) in [(internet.TCPServer, f), (internet.UDPServer, p)]:
    s = klass(PORT, arg)
    s.setServiceParent(ret)

# Run all of the above as a twistd application
ret.setServiceParent(service.IServiceCollection(application))

EDIT #1

blakev suggested that I might not be using the generator correctly (which is certainly possible). But if I simplify this down a little bit to not even use the DB, I still cannot process more than one DNS request at a time. To test this, I have stripped the class down. What follows is my entire, runnable, test file. Even in this highly stripped-down version of my server, Twisted does not accept any more requests until the first one has come in.

import sys
import logging

from twisted.names import dns, server, client, cache
from twisted.application import service, internet
from twisted.internet import defer


class DNSResolver(client.Resolver):
    def __init__(self, servers):
        client.Resolver.__init__(self, servers=servers)

    def lookupMailExchange(self, name, timeout=None):
        """
        The twisted function which is called when an MX record lookup is requested.
        :param name: The domain name being queried for (e.g. example.org).
        :param timeout: Time in seconds to wait for the query response. (optional, default: None)
        :return: A DNS response for the record query.
        """
        logging.critical("Query for " + name)

        return defer.succeed([
          (dns.RRHeader(name, dns.MX, dns.IN, 600,
              dns.Record_MX(1, "10.0.0.9", 600)),), (), ()
        ])

# App name, UID, GID to run as. (root/root for port 53 bind)
application = service.Application('db_driven_dns', 1, 1)

# Set the secondary resolver
db_dns_resolver = DNSResolver( [("8.8.8.8", 53), ("8.8.4.4", 53)] )

# Create the protocol handlers
f = server.DNSServerFactory(caches=[cache.CacheResolver()], clients=[db_dns_resolver])
p = dns.DNSDatagramProtocol(f)
f.noisy = p.noisy = False

# Register as a tcp and udp service
ret = service.MultiService()
PORT=53

for (klass, arg) in [(internet.TCPServer, f), (internet.UDPServer, p)]:
    s = klass(PORT, arg)
    s.setServiceParent(ret)

# Run all of the above as a twistd application
ret.setServiceParent(service.IServiceCollection(application))


# If called directly, instruct the user to run it through twistd
if __name__ == '__main__':
    print "Usage: sudo twistd -y %s (background) OR sudo twistd -noy %s (foreground)" % (sys.argv[0], sys.argv[0])

Upvotes: 4

Views: 2424

Answers (1)

richardw
richardw

Reputation: 768

Matt,

I tried your latest example and it works fine. I think you may be testing it wrong.

In your later comments you talk about using time.sleep(5) in the lookup method to simulate a slow response.

You can't do that. It will block the reactor. If you want to simulate a delay, use reactor.callLater to fire the deferred

eg

def lookupMailExchange(self, name, timeout=None):
    d = defer.Deferred()
    self._reactor.callLater(
        5, d.callback, 
        [(dns.RRHeader(name, dns.MX, dns.IN, 600, 
                       dns.Record_MX(1, "mail.example.com", 600)),), (), ()]
    )
    return d

Here's how I tested:

time bash -c 'for n in "google.com" "yahoo.com"; do dig -p 10053 @127.0.0.1 "$n" MX +short +tries=1 +notcp +time=10 & done; wait'

And the output shows that both responses came back after 5 seconds

1 10.0.0.9.
1 10.0.0.9.

real    0m5.019s
user    0m0.015s
sys 0m0.013s

Similarly, you need to make sure that calls to your database don't block:

Some other points:

Upvotes: 2

Related Questions