Reputation: 307
I want to copy the contents of a file from a webpage to a local file by opening the online file using requests, then copying all of my requests content to my local file.
The problem is that all I am getting in my local file is a complaint about the file not being UTF-8 encoded.
I need to create the file before writing to it, so what I have tried so far is opening and closing the file for writing text and encoding it like this: f = open(path, 'w', encoding='UTF-8') f.close() This did not work.
import requests
from bs4 import BeautifulSoup as soup
for i in range(3586, 9003, 95):
print(i)
#Get the link
gal_links = requests.get(url + str(i))
if not (gal_links.status_code == 200):
print('DNE: ' + str(gal_links.status_code))
else:
scraper = soup(gal_links.text)
href = scraper.find_all('a')
#choose a random link starting after the 7th, and ending at nth - 2
rand = randint(7, len(href)-2)
star_link = href[rand]['href']
file_url = url + str(i) + '/' + star_link
print(file_url)
#get the .fits file
req = requests.get(file_url)
path = '/Users/TheBestKid/Desktop/Hubble/fits/' + str(i) + '_' + str(rand) + '.fits'
#write binary to local file
#This is where I tried opening and closing the file for reading
with open(path, 'wb+') as file:
file.write(req.content)
I would expect the file to be a bunch of jibberish, just like any other file opened in binary; however, instead it just contains this message:
Error! 'File_Name' is not UTF-8 encoded
Saving disabled
See console for more details
The console output is like this:
[W 01:10:00.905 NotebookApp]
/Users/TheBestKid/Desktop/Hubble/fits/3586_539.fits is not UTF-8
encoded
[W 01:10:00.905 NotebookApp] 400 GET
/api/contents/fits/3586_539.fits?
type=file&format=text&_=1562310600681 (::1) 1.45ms
referer=http://localhost:8889/edit/fits/3586_539.fits
Upvotes: 1
Views: 1378
Reputation: 1901
You have to do this on save file
file.write(req.content.encode("utf-8"))
Upvotes: 1