CODEWITHSUNDEEP
pythonweb-scrapingscrapyscreen-scrapinggoogle-scholar
0xwaleed
0xwaleed

Reputation: 124

Google Author Profile Scraping

I am scraping the google scholar author profile page. I am facing a problem when i tried to scrape the titles of each author, every author has more than 500 titles and they are displayed using load more button, i have got the link for loadmore pagination.

Problem is i want to calculate the total number of titles one author has but i am not getting right total value. When i try to scrape only 2 authors it returns correct but when i try to scrape all the authors in a page (10 authors in one page) then i got wrong total value.

My code is below. where my logic is wrong?

   def parse(self, response):

       for author_sel in response.xpath('.//div[@class="gsc_1usr"]'):  // loop to get all the author in a page

           link = author_sel.xpath(".//h3[@class='gs_ai_name']/a/@href").extract_first()
           url = response.urljoin(link)
           yield scrapy.Request(url,callback=self.parse_url_to_crawl)

  def parse_url_to_crawl(self,response):
      url = response.url
      yield scrapy.Request(url+'&cstart=0&pagesize=100',callback=self.parse_profile_content)

  def parse_profile_content(self,response):

    url = response.url
    idx = url.find("user")
    _id = url[idx+5:idx+17]
    name = response.xpath("//div[@id='gsc_prf_in']/text()").extract()[0]
    tmp = response.xpath('//tbody[@id="gsc_a_b"]/tr[@class="gsc_a_tr"]/td[@class="gsc_a_t"]/a/text()').extract() //it extracts the title 

    item = GooglescholarItem()  
    n = len(tmp)
    titles=[]
    if tmp:

        offset = 0; d = 0
        idx = url.find('cstart=')
        idx += 7
        while url[idx].isdigit():
            offset = offset*10 + int(url[idx])
            idx += 1
            d += 1
        self.n += len(tmp)
        titles.append(self.n)
        self.totaltitle = titles[-1]
        logging.info('inside if URL is: %s',url[:idx-d] + str(offset+100) + '&pagesize=100')  
        yield scrapy.Request(url[:idx-d] + str(offset+100) + '&pagesize=100', self.parse_profile_content)

    else:

        item = GooglescholarItem()
        item['name'] = name
        item['totaltitle'] = self.totaltitle
        self.n=0
        self.totaltitle=0
        yield item

This is the result but i am getting wrong total title value. Klaus-Robert Müller has total 837 title and Tom Mitchell has 264 titles. For Log please see attached image. i know there is a problem in my logic

 [
 {"name": "Carl Edward Rasmussen", "totaltitle": 1684},
 {"name": "Carlos Guestrin", "totaltitle": 365},
 {"name": "Chris Williams", "totaltitle": 1072},
 {"name": "Ruslan Salakhutdinov", "totaltitle": 208},
 {"name": "Sepp Hochreiter", "totaltitle": 399},
 {"name": "Tom Mitchell", "totaltitle": 282},
 {"name": "Johannes Brandstetter", "totaltitle": 1821},
 {"name": "Klaus-Robert Müller", "totaltitle": 549},
 {"name": "Ajith Abraham", "totaltitle": 1259},
 {"name": "Amit kumar", "totaltitle": 1127}
 ]

enter image description here

Upvotes: 0

Views: 555

Answers (1)

gangabass
gangabass

Reputation: 10666

I think you're overcomplicating it. I recommend using request.meta to save your offset and counted articles:

  def parse_url_to_crawl(self,response):
      url = response.url
      user = find_user_here()
      yield scrapy.Request(url+'&cstart=0&pagesize=100',callback=self.parse_profile_content, meta={'offset': 0, 'user': user})

  def parse_profile_content(self,response):
    offset = response.meta['offset']
    total_articles = response.meta.get("total_articles", 0)
    user = response.meta['user']
    # parse and count all articles
    total_articles += current_page_articles
    if load_more:
       offset += 100
       yield scrapy.Request("https://scholar.google.com/citations?hl=en&user={user}&cstart={offset}&pagesize=100".format(offset=offset, user=user),callback=self.parse_profile_content, meta={'offset': 0, 'user': user, 'total_articles': total_articles})
    else:
        yield {'user': user, 'total_articles': total_articles}

Upvotes: 1

Related Questions