CODEWITHSUNDEEP
pythonpython-3.xselenium-webdriver
Saara Ligamena
Saara Ligamena

Reputation: 45

How to create a pandas data frame from web scraping using python

I'm trying to create a pandas data frame using data that I'm collecting from the web.So every time there is a login ,the data should be stored in a data frame that i will save as an excel file. I have a code that creates the data frame and writes the data, but it does not append the data, so whenever there's a new entry it overwrites the existing information that was in the excel file.

Python Code:

count = 0
data = []
df = pd.DataFrame(columns=['Date','Country','Name','Age'])
df.loc[count, 'Date'] = datetime.datetime.now().strftime('%d %B %Y')
df.loc[count, 'Country'] = driver.find_element_by_name('country').get_attribute('value')
df.loc[count, 'Name'] = driver.find_element_by_name('customername').get_attribute('value')
df.loc[count, 'Age'] = driver.find_element_by_name('acli.age').get_attribute('value')
df.to_excel("C:\Web Data\MIS REPORT.xlsx")

Instead of the code overwriting the existing rows in the excel file it should append it.

Upvotes: 0

Views: 43

Answers (1)

user12758604
user12758604

Reputation: 375

You did not increment your count every time there is a new entry, hence it's always overwriting the first entry of the dataframe!

Also, with each entry, the following lines:

count = 0
data = []
df = pd.DataFrame(columns=['Date','Country','Name','Age'])

should not be executed again so as to not recreate the dataframe/reset the count back to 0.

Hope that helps! :)

Upvotes: 1

Related Questions