CODEWITHSUNDEEP
pythonpython-requests
SirBoboHobo
SirBoboHobo

Reputation: 3

Python Navigation Timeout Exceeded: 8000 ms exceeded

I keep getting this error on multiple scripts, I'm doing a lot of scraping, I have a loop that scrapes through hundreds of pages, and at some point the scripts just stops due to this error. Here's an example of a script

Example 2: 

def scrape(urls):
    for url in urls:
        session = HTMLSession()
        resp = session.get(url)
        resp.html.render()
        try:
            phone = resp.html.find('span.phone')[0].text
        except IndexError:
            phone = None
        biz_name = resp.html.find('h1')[0].text
        try:
            biz_desc = resp.html.find('p.biz-description-text')[0].text
        except IndexError:
            biz_desc = None
        biz_location = resp.html.find('span.title-address-text')[0].text
        city = biz_location.split(',')[-1]

        print(
            f'phone is: {phone}\nthe business name is: {biz_name}\nthe description is: {biz_desc}\nthe city is: {city}')
        import_data(biz_name, phone, biz_desc, city)

def import_data(name, phone, desc, city):
    global keyword
    wp_title_box = driver.find_element_by_xpath('//*[@id="title"]')
    wp_title_box.send_keys(name)
    time.sleep(1)

    wp_desc_box = driver.find_element_by_xpath('//*[@id="content_ifr"]')
    wp_desc_box.send_keys(desc)
    time.sleep(1)

    new_field_button = driver.find_element_by_xpath('//*[@id="newmeta-submit"]')
    select_box = Select(driver.find_element_by_xpath('//*[@id="metakeyselect"]'))

    select_box.select_by_value("ad_city")
    wp_city_fill = driver.find_element_by_xpath('//*[@id="metavalue"]')
    wp_city_fill.send_keys(city)
    new_field_button.click()
    time.sleep(2)

    select_box.select_by_value("ad_phone")
    wp_city_fill = driver.find_element_by_xpath('//*[@id="metavalue"]')
    wp_city_fill.send_keys(phone)
    new_field_button.click()
    time.sleep(2)

    select_box.select_by_value("ad_promote")
    wp_city_fill = driver.find_element_by_xpath('//*[@id="metavalue"]')
    wp_city_fill.send_keys('1')
    new_field_button.click()
    time.sleep(2)

    save_btn = driver.find_element_by_xpath('//*[@id="save-post"]')
    driver.execute_script("window.scrollTo(0,0);")
    time.sleep(1)
    save_btn.click()
    time.sleep(2)
    driver.find_element_by_xpath('//*[@id="menu-posts"]/ul/li[3]/a').click()
    time.sleep(2)

I've added example 2, as example 1 was solved by a loop provided below. In the second example the script should end since I'm using a for loop, once it has finished going through all of the urls and importing them, it should be done, am I missing something?

Upvotes: 0

Views: 1633

Answers (1)

malanb5
malanb5

Reputation: 314

Your program never terminates. Number calls scrape, which calls number, which calls scrape, which calls number etc. If you are going to use recursion you need to have a terminating or base case.

One suggestion is using a counter to track the depth of your recursion and then increment the counter at each step until it reaches the specified depth.

I do think for what you are doing you do not need recursion at all which is expensive due to the overhead of function calls. A simple loop would be fine:

import random
import urllib3
from requests_html import HTMLSession

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


def scrape(rand_num):

    session = HTMLSession()
    resp = session.get("https://www.example.com/prize/?d=" + '92' + str(rand_num))
    resp.html.render()
    print(f'trying coupon code 92{rand_num}')
    prize = resp.html.find(containing="You've won a prize")
    print(prize)
    if prize:
        print("https://www.example.com/prize/?d=" + '92' + str(rand_num))


def number():

    for i in range(99999999):
        x = random.randint(00000000, 99999999)
        scrape(x)


number()

Upvotes: 1

Related Questions