CODEWITHSUNDEEP
pythoncsvbeautifulsoup
littlejiver
littlejiver

Reputation: 255

how to write to csv from beautifulsoup data

Looking to take data I have extracted with beautifulsoup to .csv file

this the code to extract:

from requests import get

url = 'https://howlongtobeat.com/game.php?id=38050'

    response = get(url)

    from bs4 import BeautifulSoup

    html_soup = BeautifulSoup(response.text, 'html.parser')

    game_name = html_soup.select('div.profile_header')[0].text
    game_length = html_soup.select('div.game_times li div')[-1].text
    game_developer = html_soup.find_all('strong', string='\nDeveloper:\n')[0].next_sibling
    game_publisher = html_soup.find_all('strong', string='\nPublisher:\n')[0].next_sibling
    game_console = html_soup.find_all('strong', string='\nPlayable On:\n')[0].next_sibling
    game_genres = html_soup.find_all('strong', string='\nGenres:\n')[0].next_sibling

I would like to write the results of these to csv (it's pulling the info I want but I think it needs to be cleaned up)

not sure how to write to csv or to clean up data

please help

Upvotes: 2

Views: 2043

Answers (3)

Ajax1234
Ajax1234

Reputation: 71451

You can use csv.writer:

import csv, re
from bs4 import BeautifulSoup as soup
import requests
flag = False
with open('filename.csv', 'w') as f:
  write = csv.writer(f)
  for i in range(1, 30871):
    s = soup(requests.get(f'https://howlongtobeat.com/game.php?id={i}').text, 'html.parser')
    if not flag: #write header to file once
      write.writerow(['Name', 'Length']+[re.sub('[:\n]+', '', i.find('strong').text) for i in s.find_all('div', {'class':'profile_info'})])
      flag = True
    name = s.find('div', {"class":'profile_header shadow_text'}).text
    length = [[i.find('h5').text, i.find("div").text] for i in s.find_all('li', {'class':'time_100'})]
    stats = [re.sub('\n+[\w\s]+:\n+', '', i.text) for i in s.find_all('div', {'class':'profile_info'})]
    write.writerows([[name, length[0][-1]]+stats[:4]])

Upvotes: 1

yadavankit
yadavankit

Reputation: 353

For writing this data to a CSV File,

game_info = [game_name, game_publisher, game_console, game_genre, game_length, game_developer]
with open("game.csv", 'w') as outfile:
    csv.register_dialect('custom', delimiter='\n', quoting=csv.QUOTE_NONE, escapechar='\\')
    writer = csv.writer(outfile,'custom')
    row = game_info
    writer.writerow(row)

Upvotes: 0

Jānis Š.
Jānis Š.

Reputation: 552

You can use the Python's csv module: https://docs.python.org/3/library/csv.html or https://docs.python.org/2.7/library/csv.html.

Upvotes: 0

Related Questions