CODEWITHSUNDEEP
pythonpython-3.xcsvweb-scraping
SIM
SIM

Reputation: 22440

Unable to write data across columns in a csv file

I've written a script in python to scrape different names and their values out of a table from a webpage and write the same in a csv file. My below script can parse them flawlessly but I can't write them to a csv file in a customized manner.

What I wish to do is write the names and values across columns which you may see in image 2.

This is my try:

import csv
from bs4 import BeautifulSoup
import requests

res = requests.get("https://www.bloomberg.com/markets/stocks",headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text, "lxml")
with open("outputfile.csv","w",newline="") as infile:
    writer = csv.writer(infile)
    for table in soup.select(".data-table-body tr"):
        name = table.select_one("[data-type='full']").text
        value = table.select_one("[data-type='value']").text
        print(f'{name} {value}')
        writer.writerow([name,value])

Output I'm getting like below:

enter image description here

How I wish to get the output is like the following:

enter image description here

Any help to solve this will be vastly appreciated.

Upvotes: 0

Views: 71

Answers (3)

ThePoetCoder
ThePoetCoder

Reputation: 182

If I understand you correctly, try making just one call to writerow instead of one per loop

import csv
from bs4 import BeautifulSoup
import requests

res = requests.get("https://www.bloomberg.com/markets/stocks",headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text, "lxml")
with open("outputfile.csv","w",newline="") as infile:
    writer = csv.writer(infile)
    data = []
    for table in soup.select(".data-table-body tr"):
        name = table.select_one("[data-type='full']").text
        value = table.select_one("[data-type='value']").text
        print(f'{name} {value}')
        data.extend([name, value])
    writer.writerow(data)

Upvotes: 1

Andersson
Andersson

Reputation: 52665

Try to define empty list, append all the values in a loop and then write them all at once:

with open("outputfile.csv","w",newline="") as infile:
    writer = csv.writer(infile)
    names_and_values = []
    for table in soup.select(".data-table-body tr"):
        name = table.select_one("[data-type='full']").text
        value = table.select_one("[data-type='value']").text
        print(f'{name} {value}')
        names_and_values.extend([name,value])
    writer.writerow(names_and_values)

Upvotes: 2

Denziloe
Denziloe

Reputation: 8131

That seems like an ugly thing to want to do, are you sure?

Use pandas for getting csvs and manipulating tables. You'll want to do something like:

import pandas as pd

df = pd.read_csv(path)
df.values.ravel()

Upvotes: 0

Related Questions