CODEWITHSUNDEEP
pythonjsonpython-requests
user12928589
user12928589

Reputation:

How to turn text/html requests response into JSON format in python?

I have a problem just like I mentioned in topic. Trying to get the name of all CVE's on website and their description. I want to iterate JSON response using for loop and send everything to DB. Code:

r = requests.get('https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=microsoft')
r.headers['content-type']

After printing out the headers it shows me that the response is text/html and I cannot convert it into JSON. I was trying to do it like I usually do, following the documentation using r.json(). But after running code im getting error like this:

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I also tried json.dumps() , unfortunatelly didn't solve the problem. I was looking for the solution for hours and couldn't find anything that would help me convert this response into JSON. Is this even possible with this website and this response? Can I do it differently? Is there any other good solution to achieve my goal? If I wrote something not understendable, please correct me or ask me a question.

Thank you a lot in advance for any kind of a help. :-)

Upvotes: 1

Views: 1898

Answers (1)

sushanth
sushanth

Reputation: 8302

Response for the API is not a JSON, instead its a HTML you can either use Pandas read_html or Beautifulsoup to parse the details required, below is the example using pandas read_html

import pandas as pd

df = pd.read_html("https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=Microsoft")

df[2].to_dict(orient='records')

Upvotes: 1

Related Questions