CODEWITHSUNDEEP
pythonpython-requests
AXO
AXO

Reputation: 9086

Is it possible for requests.Response.iter_content to return empty chunks?

In many StackOverflow questions and answers like 1, 2, or ..., and many other places all around the web, people are using the following pattern to retrieve results using iter_content:

for chunk in r.iter_content(chunk_size=512 * 1024): 
    if chunk: # filter out keep-alive new chunks
        f.write(chunk)

But is the if chunk: condition really necessary, especially when the chunk size is a positive integer like here?

Doesn't the underlying code already handle this case? To me, it seems that it does, see 3, 4...

Is it still a good practice to check for empty chunks?

P.S. I know that the requests docs suggest to do this for iter_lines:

for line in r.iter_lines():

    # filter out keep-alive new lines
    if line:

This question is only about iter_content.

Update

It's also interesting to look at this the other way:

Thus people should probably not worry about these things, even when writing to a file using iter_lines.

Upvotes: 1

Views: 2606

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1121834

You are entirely correct, testing if chunk is non-empty is redundant. The entire codepath in requests and urllib3 already ensures that you are only served non-empty chunks.

The underlying code does need to check for empty chunks; those can be caused by network delays and / or decompression (where not enough data has yet arrived to decompress another block).

Upvotes: 4

Related Questions