CODEWITHSUNDEEP
pythonpython-requestspython-multithreading
Tim D
Tim D

Reputation: 69

Why do some identical POST requests take two or three times longer than others?

I want to send dozens of POST requests, each one separated by 50ms. All requests are identical. They take about 315 ms (send time + latency) but for one reason I ignore one request out of four (on average) takes two to three times longer (see Charles Proxy timeline chart below).

Can someone explain to me why? Is it because of a bad implementation of multi-threading (see code below)?

import thread
import time
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning

url = XXX
req_data = XXX

session = requests.Session()
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
session.cookies.clear()

def send_request(url, data):
    r = session.post(url, verify=False, data=data)

for i in range(20):
    time.sleep(0.050)
    thread.start_new_thread( send_request, (url, req_data,) )

Charles Request Timeline

Legend from Charles documentation :

Each bar on the chart is divided into three segments:

  • Request - the time spent sending (uploading) the request (dark blue)
  • Latency - the time spent waiting for network latency or processing time on the server (mid blue)
  • Response - the time spent receiving (downloading) the response (light blue)

EDIT :

I analysed connections through Wireshark and below is an example of a ~300ms request and a ~800ms request.

It seems that the 800ms has two more ACK packets than the 300ms. Any one understands what is going on and what is different between these two requests ?

300 ms request 300ms

800 ms request 800ms

Upvotes: 2

Views: 432

Answers (1)

Tim D
Tim D

Reputation: 69

The packet analysis with Wireshark (see edit in the main post) shows that the 500ms gap between the two queries (a short and a long one) comes from the ACK packet at the time of SSL key exchange.

As a reminder, all my requests went through the proxy Charles Web Proxy.

After some tests, it seems that the problem lies in the SSL verification with Charles (which sometimes adds 500ms to the request uploading time).

The queries have equivalent and regular sending times in the following cases:

  • When I do not use the Charles Proxy
  • When I use the proxy Charles but I go through "http" instead of "https" (No SSL) (this is my retained solution here because there is no sensitive data).

Thanks for your help guys :)

This answer shows link between SSL certification by Charles Web Proxy and the issue but doesn't explain it plently

Upvotes: 1

Related Questions