CODEWITHSUNDEEP
pythoncsvbeautifulsoupexport-to-csv
kittykat14
kittykat14

Reputation: 35

How to fix writing to csv file

I want my program to write article dates, titles, and body text to a csv file. When I print the body text in the console, it prints everything, however on the csv file, it only prints the last line of the article.

CSV Result:

enter image description here

Console Print:

enter image description here

I've tried writing the date, title, and body text into rows in separate lines of code as apposed to as a list and it had the same result.

from bs4 import BeautifulSoup
from urllib.request import urlopen
import csv

csvfile = "C:/Users/katew/Dropbox/granularitygrowth/Politico/pol.csv"
with open(csvfile, mode='w', newline='') as pol:
    csvwriter = csv.writer(pol, delimiter='|', quoting=csv.QUOTE_MINIMAL)
    csvwriter.writerow(["Date", "Title", "Article"])

    #for each page on Politico archive
    for p in range(0,1):
        url = urlopen("https://www.politico.com/newsletters/playbook/archive/%d" % p)
        content = url.read()

        #Parse article links from page
        soup = BeautifulSoup(content,"lxml")
        articleLinks = soup.findAll('article', attrs={'class':'story-frag format-l'})

        #Each article link on page
        for article in articleLinks:
            link = article.find('a', attrs={'target':'_top'}).get('href')

            #Open and read each article link
            articleURL = urlopen(link)
            articleContent = articleURL.read()

            #Parse body text from article page
            soupArticle = BeautifulSoup(articleContent, "lxml")

            #Limits to div class = story-text tag (where article text is)
            articleText = soupArticle.findAll('div', attrs={'class':'story-text'})
            for div in articleText:

                #Find date
                footer = div.find('footer', attrs={'class':'meta'})
                date = footer.find('time').get('datetime')
                print(date)

                #Find title
                headerSection = div.find('header')
                title = headerSection.find('h1').text
                print(title)

                bodyText = div.findAll('p')
                for p in bodyText:
                    p_string = str(p.text)
                    textContent = "" + p_string
                    print(textContent)

                #Adds data to csv file
                csvwriter.writerow([date, title, textContent])

I expect the csv file to include the date, title and full body text.

Upvotes: 0

Views: 71

Answers (1)

user5386938
user5386938

Reputation:

The problem's in your for p in bodyText: loop. You're assigning the text of the last p to your textContent variable. Try something like:

textContent = ""
bodyText = div.findAll('p')
for p in bodyText:
    p_string = str(p.text)
    textContent += p_string + ' '

print(textContent)
csvwriter.writerow([date, title, textContent])

Upvotes: 2

Related Questions