CODEWITHSUNDEEP
azurejmeterdns
evolvah
evolvah

Reputation: 645

Intermittent JMeter DNS failure in Azure

I am running a low-rate performance test of an API server from an Ubuntu VM. API server runs on an AKS cluster in Azure and the test VM runs in Azure in the same region. I'm using JMeter to exercise an HTTPS endpoint at a rate of about 16 requests per second.

The endpoint is associated with a Public IP Address resource, which in turn has an "alias" in one of DNS zones I have hosted in Azure.

Intermittently, JMeter begins to fail while throwing java.net.UnknownHostException. This can last for about 30 seconds out of a 48-hour run. It recovers automatically after this short period of failures, but this behavior taints my test results because failures are cause by the SNUT (JMeter) rather than the SUT (my API server).

The PCAP trace do not show anything incriminating:

Nothing on the wire seems to suggest an explicit DNS error. There is one implicit glitch, and that is the AAAA-record lookup. One time, the AAAA-record lookup does not receive a response and systemd-resolved retries the query, which returns in 8ms.

I am pretty certain that JMeter and systemd-resolved are well-tested software components that should not be struggling with 16 TPS that my test generates.

Can it be Azure DNS at 168.63.129.16? But for all DNS queries from this machine, I generate somewhere around 0.2 TPS (one DNS transaction per 5 seconds towards 168.63.129.16).

I also suspect some internal DNS cache inside of JMeter at play because at 16 TPS for my HTTPS requests, I see much less frequent queries from 127.0.0.1 to 127.0.0.53.

Having said all this, I begin to suspect that it is some internal mechanics and configuration of JMeter that plays a trick on me. And I am not a JMeter expert at all. It is a blackbox to me.

I would really appreciate any comments or suggestions that could point me in the right direction in the search for explanation.

I could blindly add dnsmasq or something else, but I don't want to cover up the problem. I want to understand it.

Thank you for reading this far!

Upvotes: 0

Views: 150

Answers (1)

Dmitri T
Dmitri T

Reputation: 168122

Try adding DNS Cache Manager to your Test Plan and configure it to use explicit custom DNS resolver of your choice.

Alternatively you can set sun.net.inetaddr.ttl or even better networkaddress.cache.ttl JMeter system property to 0 and this way each thread will resolve the underlying IP address on its own each time.

More information:

Upvotes: 1

Related Questions