CODEWITHSUNDEEP
pythonpython-3.xcsvdataframe
ceciestunepipe
ceciestunepipe

Reputation: 13

.writerow() csv in Python not writing all data

I'm new to Python and I'm trying to scrape some data and save them in a csv. I'm trying to loop a csv with a list of URLs, read the data from each URL and write that information in another csv file

The following code is writing roughly half of the data in the cvs but is printing everything fine while it's writing

df_link = pd.read_csv('url_list')

with open('url_list.csv', newline='') as urls, open('output.csv', 'w', newline='') as output:
    csv_urls = csv.reader(urls)
    csv_output = csv.writer(output)
    csv_output.writerow(['details','date'])

    for link in df_link.iterrows():
        url = link[1]['url']
        browser.get(url)
        soup = BeautifulSoup(browser.page_source)

        csv_file = open('output.csv', 'w')
        csv_writer = csv.writer(csv_file)
        csv_writer.writerow(['details'])


        details=[i.text for i in soup.find_all(class_='product-info-content- 
        block product-info')]
        print('details :', details)

        dt = date.today()
        print('date :', dt)

        csv_output.writerow([str(details).strip('[]'), dt])
        csv_file.close()

Everything is being printed fine when the code is running, but not all the rows of data are being written in the output csv.

I hope someone can help. Thank you!

Upvotes: 1

Views: 1744

Answers (1)

Fredrik
Fredrik

Reputation: 451

It looks like you are opening output.csv twice, once in the beginning and then in the for loop. Since you are opening with the option w like csv_file = open('output.csv', 'w') it will overwrite the file every loop.

So if you move the below part out of the loop it might work better

    csv_file = open('output.csv', 'w')
    csv_writer = csv.writer(csv_file)
    csv_writer.writerow(['details'])

Upvotes: 1

Related Questions