CODEWITHSUNDEEP
python
Charlie
Charlie

Reputation: 316

how to make urllib.request append to an existing file?

I'm trying to download a load of text in Python and want it all to save to a single file.

The code I'm currently using creates a separate file for each url. It loops through an archive of urls, requests the data and then saves it to its own file.

filename = archive[i]
urllib.request.urlretrieve(url, path + filename + ".pgn")

I've tried using the same filename for each url but it just overwrites the file.

Is there a way to loop through the archive and, rather than saving the data in its own separate file, add each block of text to a single file? Or do I need to just loop through all the files afterwards and concatenate them together?

Upvotes: 0

Views: 1533

Answers (2)

vaku
vaku

Reputation: 712

Python's urlretrive docs says that

If you wish to retrieve a resource via URL and store it in a temporary location, you can do so via the urlretrieve() function

so if you wish to append the retrieved data in one file you have use urlopen for that

Like this :

import urllib.request

filename = "MY_FILE_PATH"
#-----------inside your i loop-------------
with urllib.request.urlopen(url) as response:
    data = response.read()
    # change your file type according e.g. "ab" for binary file
    with open(filename + ".pgn", "a+") as fp: fp.write(str(data))

Upvotes: 1

Nuhman
Nuhman

Reputation: 1270

Note that urlretrieve might become deprecated at some point in the future. So use urlopen instead. 

import urllib.request
import shutil

...

filename = archive[i]
with urllib.request.urlopen(url) as response, open(filename, 'ab') as out_file:  
    shutil.copyfileobj(response, out_file)

Upvotes: 1

Related Questions