CODEWITHSUNDEEP
pythonjson
Marcus Scipio
Marcus Scipio

Reputation: 31

Python Rejects Valid JSON

I'm trying to process this JSON using python3: http://www.bom.gov.au/fwo/IDV60701/IDV60701.94857.json

But I'm getting the following error:

Traceback (most recent call last):
  File "./weath.py", line 41, in <module>
    data1                   = response.json()
  File "/home/dz/anaconda3/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/dz/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/dz/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/dz/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

https://jsonlint.com/

Validates the JSON as valid, so I'm not sure why this is failing.

Here is the python code:

url                     = "http://www.bom.gov.au/fwo/IDV60701/IDV60701.94857.json"
response                = requests.get(url)
data1                   = response.json()

This was working 2 week ago. How can I fix this?

Upvotes: 0

Views: 147

Answers (2)

user13945859
user13945859

Reputation:

When I ran this code:

import requests

url = "http://www.bom.gov.au/fwo/IDV60701/IDV60701.94857.json"
response = requests.get(url).text

print(response)

The print returned

Potential automated request detected! We are making changes to our website therefore web scraping is no longer supported. Please contact us by filling in the details at http://reg.bom.gov.au/screenscraper/screenscraper_enquiry_form/ and we will get in touch with you.

Seems like that website has disabled web-scraping or something

Upvotes: 1

ricekab
ricekab

Reputation: 650

This is an issue related to the specific service endpoint you're using. They have disabled web scraping through some mechanism.

If you look at your response object, you'll see it's a 403 (forbidden) with the following message:

Potential automated request detected! We are making changes to our website therefore web scraping is no longer supported. Please contact us by filling in the details at http://reg.bom.gov.au/screenscraper/screenscraper_enquiry_form/ and we will get in touch with you.

You can verify this for yourself:

response = requests.get("http://www.bom.gov.au/fwo/IDV60701/IDV60701.94857.json")
print(response.status_code)  # 403
print(response.text)  # above quote

Upvotes: 2

Related Questions