CODEWITHSUNDEEP
pythonpython-3.xcsvweb-crawler
SIM
SIM

Reputation: 22440

Scraped data getting printed to csv in a single line

Running my crawler I could see that it fetches data as it should but when it comes to print the data to a csv file, it prints that in a single line. I'm very new to work with class in python so there may be numerous mistakes in my code which I have written hypothetically seeing different examples. So, at this point I wish to fix the single line printing and make it go down creating new lines. Any suggestion will be highly appreciated.

import csv
import requests
from lxml import html

class wiseowl:
    def __init__(self,start_url):
        self.start_url=start_url
        self.storage=[]

    def crawl(self):
        self.get_link(self.start_url)


    def get_link(self,link):
        response=requests.get(link)
        tree=html.fromstring(response.text)
        titles=tree.xpath("//p[@class='woVideoListDefaultSeriesTitle']")
        for title in titles:
            name=title.xpath(".//a/text()")[0]
            urls=title.xpath(".//a/@href")[0]
            Docs=(name,urls)
            self.storage.append(Docs)


    def writing_csv(self):
        with open("Wiseowl.csv","w",newline="") as f:
            writer=csv.writer(f)
            writer.writerow(["Title","Link"])
            writer.writerow(self.storage)


    def __str__(self):  
        return "{}".format(self.storage)


crawler=wiseowl("http://www.wiseowl.co.uk/videos/")
crawler.crawl()
crawler.writing_csv()
for item in crawler.storage:
    print(item)

Upvotes: 0

Views: 44

Answers (1)

Nurjan
Nurjan

Reputation: 6073

If I have understood you correctly you already have the answer in your code where you print out the crawler.storage line by line.

Just change the method writing_csv to this:

def writing_csv(self):
    with open("Wiseowl.csv","w",newline="") as f:
        writer=csv.writer(f)
        writer.writerow(["Title","Link"])

        for item in self.storage:
            writer.writerow(item)

When you use writer.writerow(self.storage) the method writerow considers self.storage as one line string. This is why it stores it as one line in the file.

Note I run the code using python3.

Upvotes: 1

Related Questions