How to convert the URL response to a dataframe

Question

I have been working on a requirement where i need to download the file from a website which outputs the data in csv format and then write it to a SQL table. I am using the below logic to download the data from a website.

import urllib
from bs4 import BeautifulSoup

url = "https://abcdef.oz/login"
response = opener.open(url)
if response.status == 200:
    #webUrl = urllib.request.urlopen(url)
    #print("Result code: " + str(webUrl.getcode()))
    data = response.read().decode("utf-8")
    soup = BeautifulSoup(data)

    #Got a token
    token = soup.find_all("meta", {"name":"token"}, limit=1)[0]['content']
    print("token",token)
    sdata = {"email": "abc@def.com", "password": "password"}

    data = urllib.parse.urlencode(sdata).encode()

    print("data",data)
    url = "https://abcdef.oz.co/reports"

    response = opener.open(url)
    r = response.read().decode('utf-8')

    print(r)
 else:
    print("No Response")

How can the response now be converted to a format where i can skip the header and write the data to a SQL table.

The output of the response is as below

"Col1","Col2","Col3"\n"abc","def","efg"\n"hij","klm","mno"

Thanks in advance

Answer 1

this is not mindblowing, but did you try:

df = pandas.read_html(url)