CODEWITHSUNDEEP
dnslocustttl
Sébastien Allamand
Sébastien Allamand

Reputation: 198

How to make Locust respect DNS TTL

I'm distributed locust using the provided terraform on AWS. Between the load, I update the route53 weighted records of my target to point to another version.

What I see is that locust don't update the initial DNS resolution, and keep targetting my first version.

How can I make locust to recalculate the dns resolution during the load ?

Upvotes: 0

Views: 425

Answers (2)

tmokmss
tmokmss

Reputation: 538

Locust internally maintains a connection pool for each user. It will prevent from updating DNS resolution, because DNS resolution happens only when a connection is established, and the connection will be reused for the requests sent from the user.

As every HttpUser creates new HttpSession, every user instance has its own connection pools. https://docs.locust.io/en/stable/writing-a-locustfile.html#connection-pooling

Any updates to DNS records will not be reflected until the connection is deleted from the user's pool and a new connection is established.

There are several ways to disable connection pooling:

  1. Override pool_manager instance

You can override pool_manager for a User class (shared within the same class) with num_pools=0 pool

from locust import HttpUser, task
from urllib3 import PoolManager

class SampleUser(HttpUser):
    # add this
    pool_manager = PoolManager(num_pools=0)

    @task
    def hello_world(self):
        self.client.get("/")
  1. Add Connection: close to the requests sent from a user

When you explicitly add Connection: close HTTP header, the connection is disconnected as soon as a request is processed, which effectively disables connection pooling.

from locust import HttpUser, task

class SampleUser(HttpUser):
    @task
    def hello_world(self):
        self.client.get("/", headers={"Connection": "close"})
  1. Use FastHttpUser and disable connection pooling

FastHttpUser exposes an API to disable connection pooling explicitly.

By default a user will reuse the same TCP/HTTP connection if possible. To more realistically simulate new users connecting to your application this connection can be manually closed. https://docs.locust.io/en/stable/increase-performance.html#connection-handling

from locust import FastHttpUser, task

class SampleUser(FastHttpUser):
    @task
    def hello_world(self):
        self.client.client.clientpool.close()
        self.client.get("/")
  1. Manually resolve a DNS record

As suggested in the other answer, manually resolving DNS record can also fix the issue, because connection pool is stored with hostname key (IP address in this case).

Note that disabling connection pooling may slow down your load test performance.

Upvotes: 0

Cyberwiz
Cyberwiz

Reputation: 11426

That is kind of hard, as your OS will most likely cache the dns resolution.

You could use dnspython to resolve the address yourself (https://blog.devgenius.io/pyops-dnspython-toolkit-590a368b5c2)

A = dns.resolver.resolve(domain, 'A')
for answer in A.response.answer:
    for item in answer.items:
        ip = item.address

And then do the request against that ip, manually adding the appropriate Host header.

self.client.get(f"http://{ip}", headers={"Host": domain})

Edit: if using OS dns resolution is ok or even desirable, then maybe just creating a new Session is enough. That creates a new tcp connection and should trigger a new DNS query once any OS level caching times out:

from locust.clients import HttpSession

...
    
@task
def t(self):
    self.client.close()
    self.client = HttpSession(
        base_url=self.host,
        request_event=self.environment.events.request,
        user=self
    )
    # your actual requests

Upvotes: 1

Related Questions