Fast http queries in Python

Question

I need to send many http post requests (several thousands) and read the responses in parallel. I use httplib in python, it takes me on average 0.5 ms to send a request, and it is quite slow to read the response. I use coroutines (gevent) instead of threads to make it faster. I thought about the following to make it faster:

• write C code (I would need to find a good http library in C), and write a c-extension
• use Cython

Which approach would yield the fastest running code? Any sharing of experience with one or the other approach would be much appreciated as well. Any other idea very welcome.

Answer 1

I would suggest to take a look a PyCURL. Before yo do that you should check if you get higher throughput when you use curl from the command line. If that's not the case, you probably won't get better results with PyCURL.