CODEWITHSUNDEEP
pythonpython-3.xsearch-enginetextblobpython-newspaper
Jack Pan
Jack Pan

Reputation: 1411

Python 3: How can I get news articles that contain a certain keyword

I'm trying to write a little web app that returns the sentiment of a news article involving a keyword.

I used the TextBlob and Newspaper3K python 3 packages. I tried to make the url string for Newspaper3K the result of a search query on Google News but the newspaper package just seems to redirect to the "main page" of Google News.

Is there any way to get a list of newspaper articles that contain a certain keyword? In addition, is it possible for newspaper to iterate through pages?

The following is my code:

from textblob import TextBlob
import newspaper

#keyword = input("Please enter the keyword: ")
keyword = "Apple" #for testing only
keyword_lowercase = keyword.lower()

search_string = "" # only for google news
split_keyword = keyword.split()
for i in range(len(split_keyword)):
    search_string += split_keyword[i]
    if i != len(split_keyword)-1:
        search_string += '+'

def google_news_site(search_query):
    prefix = 'http://news.google.com/news?q='
    return prefix+search_string

#Currently for news.google.com only
url_string = google_news_site(search_string)
paper = newspaper.build(url_string, memoize_articles=False)

def sentiment(text):
    return TextBlob(text).sentiment.polarity

current_sum = 0.0
relevant_article_count = 0
for article in paper.articles:
    print(article.url)
    article_text = article.text
    article_text_lowercase = article_text.lower()
    if keyword_lowercase in article_text_lowercase:
        current_sum += sentiment(article_text)

print("Article count is", str(relevant_article_count)+".")

rating = current_sum/max(relevant_article_count, 1)
print("The rating for", keyword, "is", str(rating)+".")

Upvotes: 5

Views: 3274

Answers (1)

amirouche
amirouche

Reputation: 7873

The easiest way to go would be to setup an instance of the software called searx or use an existing instance like framabee.org.

searx is a metasearch engine it will query real search engines, merge results and possibly return a json file. Here is an example query:

$ curl "https://framabee.org/?q=Apple&categories=news&time_range=week&language=en&format=json" | jq . | head -n 100


{
  "number_of_results": 0,
  "corrections": [],
  "query": "Apple",
  "infoboxes": [],
  "suggestions": [],
  "results": [
    {
      "engine": "bing news",
      "category": "news",
      "parsed_url": [
        "https",
        "www.apfelnews.de",
        "/2019/09/22/apple-iphone-11-falltests-mit-unterschiedlichen-ergebnissen/",
        "",
        "",
        ""
      ],
      "pubdate": "2019-09-22 08:28:00+0000",
      "engines": [
        "bing news"
      ],
      "publishedDate": "il y a 9 heure(s), 5 minute(s)",
      "url": "https://www.apfelnews.de/2019/09/22/apple-iphone-11-falltests-mit-unterschiedlichen-ergebnissen/",
      "positions": [
        1
      ],
      "title": "Apple iPhone 11 Falltests mit unterschiedlichen Ergebnissen",
      "content": "Auf der Keynote 2019 am 10. September 2019 wurde das Apple iPhone 11 mit dem härtesten Glas in einem Smartphone beworben.",
      "pretty_url": "https://www.apfelnews.de/2019/09/22/ap[...]sts-mit-unterschiedlichen-ergebnissen/",
      "score": 1,
      "img_src": "http://www.bing.com/th?id=ON.EA4492580B994DBA90318950CC35E5A6&pid=News"
    },
...

Since searx is python code, you could directly call the appropriate python function.

Upvotes: 1

Related Questions