CODEWITHSUNDEEP
pythongoogle-apiweb-crawlergoogle-crawlersgoogle-trends
EJ Kang
EJ Kang

Reputation: 495

Google Trend Crawler code 429 error

I am new to python and using unofficial pytrends API to crawl Google Trend. I have 2000+ keywords as DNA list and try to crawl data. When I run this code, it appears with "Google returned a response with code 429" even though I added time.sleep(1). Can anyone help me with this problem?

below is my code

#DNA has 2000+ lists
from pytrends.request import TrendReq
import pandas as pd
import xlsxwriter
import time

pytrends = TrendReq(hl='en-US,tz=360')
Data = pd.DataFrame()

#Google Trend Crawler
for i in range(DNA[i]):
    time.sleep(1)
    kw_list = [DNA[i]]
    pytrends.build_payload(kw_list, cat=0, timeframe='today 5-y', geo='', gprop='')
    df = pd.DataFrame(pytrends.interest_over_time())

    #Setting a Google Trend Dates
    if(i==0):
        Googledate = pd.DataFrame(pytrends.interest_over_time())
        Data['Date'] = Googledate.index
        Data.set_index('Date', inplace=True)

    #results
    if(df.empty == True):
        Data[DNA[i]] = ""  
    else:
        df.index.name = 'Date'
        df.reset_index(inplace=True)
        Data[DNA[i]] = df.loc[:, DNA[i]]
Data

Upvotes: 6

Views: 13005

Answers (1)

Linda Lawton - DaImTo
Linda Lawton - DaImTo

Reputation: 117176

HTTP/1.1 429 Too Many Requests Content-Type: text/html Retry-After: 3600

Too Many Requests

Too Many Requests

There is no official API for Google Trends. Google has probably placed a limit on the number of requests coming from the same IP.

  1. slow down until you figure out the limit.
  2. run it on several servers allowing you to appear to come from different IP addresses.
  3. stop trying to crawl Google for data they don't want to share.

Upvotes: 6

Related Questions