CODEWITHSUNDEEP
pythonpython-3.xclassweb-scraping
robots.txt
robots.txt

Reputation: 137

Trouble making my script print the result

I've created a scipt in python using class to collect the question links from a webpage. I used __str__() method within my script to print the results. However, when I try to print it, I get None as an output.

The main purpose here is to make __str__() this method work in my following script.

Where I'm going wrong and how can I fix it?

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

URL = "https://stackoverflow.com/questions/tagged/web-scraping"

class DataSourcer:

    def __init__(self,link):
        self.link = link
        self.datalist = []

    def fetch(self):
        res = requests.get(self.link)
        soup = BeautifulSoup(res.text,"lxml")
        for ilink in soup.select(".summary .question-hyperlink"):
            self.datalist.append(urljoin(self.link,ilink.get("href")))

    def __str__(self):
        return self.datalist

if __name__ == '__main__':    
    crawler = DataSourcer(URL)
    print(crawler.fetch())

Upvotes: 1

Views: 67

Answers (3)

alexs
alexs

Reputation: 41

Your method is fine, it seems like you are printing it incorrectly. You are printing crawler.fetch() which returns None.

import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup

URL = "https://stackoverflow.com/questions/tagged/web-scraping"

class DataSourcer:

    def __init__(self,link):
        self.link = link
        self.datalist = []

    def fetch(self):
        res = requests.get(self.link)
        soup = BeautifulSoup(res.text,"lxml")
        for ilink in soup.select(".summary .question-hyperlink"):
            self.datalist.append(urljoin(self.link,ilink.get("href")))

    def __str__(self):
        return self.datalist

if __name__ == '__main__':    
    crawler = DataSourcer(URL)
    crawler.fetch()
    print(crawler.__str__())

Upvotes: 1

Koral
Koral

Reputation: 80

You are not printing right thing. Please check documentation how to use such methods.

crawler = DataSourcer(URL)
crawler.fetch() 
print(crawler)

Upvotes: 1

Racerdude
Racerdude

Reputation: 74

print(crawler.fetch()) will print the return value of the fetch() method (it has no return value) What you should do is:

crawler.fetch()
print(crawler)

Upvotes: 0

Related Questions