CODEWITHSUNDEEP
pythonpython-3.xpython-requests
nanounanue
nanounanue

Reputation: 8352

Missing new line when downloading a text file using python's request package

I am using the following code for downloading a csv file from a remote http server:

with contextlib.closing(requests.get("http://some_url/file.csv", stream= True)) as response, open('/tmp/archivo.csv', 'wb') as output_file:               
        for chunk in response.iter_lines(chunk_size=1024*8):
            if chunk:       
                print(chunk)
                output_file.write(chunk)

Everything seems to work, the output on the screen is:

b'Code,Description'
b'"1","Some"'
b'"2","Few"'
b'"3","human"'
....

But the file is:

$ cat /tmp/archivo.csv                                                                                                                          
Code,Description"1","Some""2","Few""3","human"...

I think that the issue is related to the new line encoding between Windows, GNU/Linux and Mac, but I don't know how to specify that in the code just showed.

The problem is not related to opening the file in 'wb' mode, is I made the changes needed

with contextlib.closing(requests.get("http://some_url/file.csv", stream= True)) as response, open('/tmp/archivo.csv', 'w') as output_file:               
        for chunk in response.iter_lines(chunk_size=1024*8):
            if chunk:       
                print(chunk.decode('utf-8'))
                output_file.write(chunk.decode('utf-8'))

The output on the screen changes,

'Code,Description'
'"1","Some"'
'"2","Few"'
'"3","human"'
...

but the file doesn't

Any ideas?

Upvotes: 0

Views: 1564

Answers (1)

Nils Werner
Nils Werner

Reputation: 36775

You are seeing newlines because print() automatically terminates every output with a newline. Compare your output to

print(chunk.decode('utf-8'), end='')

which specifically does not terminate with newlines.

To correctly insert newlines into output_file you can use

output_file.write(chunk.decode('utf-8') + '\n')

or iterate over chunks instead of lines using

for chunk in response.iter_content(chunk_size=1024*8):
    if chunk:
        output_file.write(chunk)

Upvotes: 3

Related Questions