CODEWITHSUNDEEP
pythoncsvweb-scrapingbeautifulsoup
benazir
benazir

Reputation: 1

Python Web Scraping - Save data in CSV

I am trying to save the data scraped from URLs such as "https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-3502.html". When saving the data in a csv file only the data from last url from the range gets saved in the csv file. I need that data from all of the URLs gets saved in the csv file.

pages = []
for i in range(1, 10, 1):
    url = "https://www.holidify.com/places/shimla/mall-road-shimla-sightseeing-350" + str(i) + '.html'
    pages.append(url)
    for item in pages:
        page = requests.get(item)
        soup = BeautifulSoup(page.text, 'html.parser')
        Place = list(soup.find(class_="col-md-10 col-xs-10 nopadding"))[1].get_text()
        City = list(soup.find_all(class_="smallerText"))[1].get_text()
        State = list(soup.find_all(class_="smallerText"))[2].get_text()
        Country = list(soup.find_all(class_="smallerText"))[3].get_text()
        About = list(soup.find_all(class_="biggerTextOverview"))[0].get_text()
        more_About = list(soup.find_all(class_="objHeading smallerText"))[0].get_text()
        Weather = soup.find(class_="currentWeather").get_text()
        demo = pd.DataFrame({ "Place": Place, "City": City, "State": State, "Country": Country, "About": About,"More About Places": more_About}, index=[0])
        demo.to_csv('demo.csv', index=False, encoding='utf-8')

Upvotes: 0

Views: 506

Answers (2)

Yash Kanojia
Yash Kanojia

Reputation: 154

As suggested by @Umair append the data into the data frame and place the command demo.to_csv('demo.csv', index=False, encoding='utf-8') outside the loop.

Upvotes: 0

Umair Ayub
Umair Ayub

Reputation: 21341

You need to append data into that file

demo.to_csv('demo.csv', index=False, encoding='utf-8', mode = 'a')

Upvotes: 1

Related Questions