CODEWITHSUNDEEP
pythonfile
Liam Keeley
Liam Keeley

Reputation: 307

Fix issue with writing binary data to file: 'file is not UTF-8 encoded'

I want to copy the contents of a file from a webpage to a local file by opening the online file using requests, then copying all of my requests content to my local file.

The problem is that all I am getting in my local file is a complaint about the file not being UTF-8 encoded.

I need to create the file before writing to it, so what I have tried so far is opening and closing the file for writing text and encoding it like this: f = open(path, 'w', encoding='UTF-8') f.close() This did not work.

import requests
from bs4 import BeautifulSoup as soup
for i in range(3586, 9003, 95):
    print(i)
    #Get the link
    gal_links = requests.get(url + str(i))

    if not (gal_links.status_code == 200):
        print('DNE: ' + str(gal_links.status_code))

    else:
        scraper = soup(gal_links.text)
        href = scraper.find_all('a')
        #choose a random link starting after the 7th, and ending at nth - 2
        rand = randint(7, len(href)-2)
        star_link = href[rand]['href']
        file_url = url + str(i) + '/' + star_link
        print(file_url)

        #get the .fits file
        req = requests.get(file_url)


        path = '/Users/TheBestKid/Desktop/Hubble/fits/' + str(i) + '_' + str(rand) + '.fits'

        #write binary to local file
        #This is where I tried opening and closing the file for reading
        with open(path, 'wb+') as file:
            file.write(req.content)

I would expect the file to be a bunch of jibberish, just like any other file opened in binary; however, instead it just contains this message:

Error! 'File_Name' is not UTF-8 encoded
Saving disabled
See console for more details

The console output is like this:

[W 01:10:00.905 NotebookApp] 
/Users/TheBestKid/Desktop/Hubble/fits/3586_539.fits is not UTF-8 
encoded
[W 01:10:00.905 NotebookApp] 400 GET 
/api/contents/fits/3586_539.fits? 
type=file&format=text&_=1562310600681 (::1) 1.45ms 
referer=http://localhost:8889/edit/fits/3586_539.fits

Upvotes: 1

Views: 1386

Answers (1)

Wonka
Wonka

Reputation: 1901

You have to do this on save file

file.write(req.content.encode("utf-8"))

Upvotes: 1

Related Questions