CODEWITHSUNDEEP
pythonscrapyweb-crawler
a0142204
a0142204

Reputation: 21

Detecting the language of a text from a crawled website in Python

I wrote several different spiders for different websites that output the text of articles and the URLs. Example:

import scrapy
import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from bs4 import BeautifulSoup

stop_words = set(stopwords.words("german"))


class FruehstueckSpider(scrapy.Spider):
    name = "fruestueckerinnen"

    def start_requests(self):
        urls = [
            'https://www.diefruehstueckerinnen.at/stadt/wien/',
        ]
        urls += [urls[0] + 'page/' + str(i) + '/' for i in range(1,17)]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        hrefs = response.css('div.text > a')
        yield from response.follow_all(hrefs, callback = self.parse_attr)

    def parse_attr(self, response):

        yield {
                'text': ' '.join([i for i  in word_tokenize(re.sub(pattern='[^a-zA-Z_\-ÖöÜüÄäßèé]',string=  BeautifulSoup(response.css('.content-inner.single-content').get(),"html.parser").find(class_="content-inner single-content").text , repl=' ')) if i not in stop_words and not re.match('[0-9]', i) and len(i) >1]),
                'url': response.request.url,
        }

I want to detect the language the whole text is written into. Does it make sense to write it under 'text' and 'url' as another property? I know that there is a function called detect (and the input is a string) from langdetect, but how do I use it in this case?

Upvotes: 1

Views: 636

Answers (2)

Sagun Shrestha
Sagun Shrestha

Reputation: 1198

You can add another field inside yield like this

from langdetect import detect  # add this to your import


# change the parse_attr function like this
def parse_attr(self, response):
    text = ' '.join([i for i  in word_tokenize(re.sub(pattern='[^a-zA-Z_\-ÖöÜüÄäßèé]',string=  BeautifulSoup(response.css('.content-inner.single-content').get(),"html.parser").find(class_="content-inner single-content").text , repl=' ')) if i not in stop_words and not re.match('[0-9]', i) and len(i) >1])
    language = detect(text)

    yield {
            'text': text,
            'language': language,
            'url': response.request.url,
    }

Upvotes: 1

Gerardsson
Gerardsson

Reputation: 382

The "lang"-attribute a html attribute that should define the language of a page. I suggest that you use this as a reference for the site because it is the most direct way to recognize this attribute. This attribute was defined to help for example voice software to choose the right language for the pronunciation. 

 <html lang="en">
...
</html>

Adding the language to the output is a matter of taste. It doesn't hurt but do you really need it? You can always include but not use the value everywhere.

Upvotes: 0

Related Questions