CODEWITHSUNDEEP
python-2.7dataframeweb-scrapingpython-requestsexport-to-csv
Jodmoreira
Jodmoreira

Reputation: 105

Pandas writes only the last line in a CSV File

i'm scraping urls from a txt file and export it to a csv file. But after all the process my code writes only the information from the last url. My guess is that i'm forgetting a loop. But where? Here's my code:

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib import urlopen

file = open('urls.txt', 'r')
filelines = (line.strip() for line in file)
for code in filelines:
    site = urlopen(code)
    soup = BeautifulSoup(site, "html.parser")
    final = soup.find_all("span", {"class": "bd js-title-main-info"})
    print final

records = []
for pagetxt in final:
    print pagetxt.text
    records.append((pagetxt.text))
df = pd.DataFrame(records, columns=['product name'])  
df.to_csv('test.csv', index=False, encoding='utf-8')

Thanks

Upvotes: 0

Views: 591

Answers (1)

Eugene Lisitsky
Eugene Lisitsky

Reputation: 12895

When you get data from file you keep only last value in variable final. Try to append data earlier (I've marked changes with #####):

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib import urlopen

file = open('urls.txt', 'r')
filelines = (line.strip() for line in file)
records = []                         ######
for code in filelines:
    site = urlopen(code)
    soup = BeautifulSoup(site, "html.parser")
    final = soup.find_all("span", {"class": "bd js-title-main-info"})
    print final

    for pagetxt in final:               ######
       print pagetxt.text               ######
       records.append((pagetxt.text))   ######   

df = pd.DataFrame(records, columns=['product name'])  
df.to_csv('test.csv', index=False, encoding='utf-8')

Upvotes: 1

Related Questions