CODEWITHSUNDEEP
web-scrapingweb-crawler
gtomer
gtomer

Reputation: 6574

How to web-scrape and find form element

Trying to scrape the following site: https://israeldrugs.health.gov.il/#!/byDrug

You need to enter a search term in the form and press the blue button on the left.

However, failed with bs4 because it cannot find the form element.

Thanks for your help.

The previous answer to the same question does not work anymore.

Upvotes: 1

Views: 956

Answers (1)

Efeu
Efeu

Reputation: 11

If you go to view-source:https://israeldrugs.health.gov.il/#!/byDrug in your browser you can see the initial HTML which you receive when you make a request to the page. This is most likely the HTML you are working on with bs4. It seems like the form element is only inserted after the pageload with javascript.

What you can do is using a tool like Selenium. It is designed to interact with a webpage like a user would. So you literally tell it to open a page, find the form element & input something and then press that button.

When you have installed Selenium your code will look something like:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get(f"https://israeldrugs.health.gov.il/#!/byDrug")

time.sleep(1) # maybe needed to let the form load

form = driver.find_elements_by_css_selector("form")[0]

inp = "Your search term"
form.send_keys(inp)
form.send_keys(Keys.ENTER) # It seems like you don't have to press the button, if you input ENTER into the form

time.sleep(1) # time to load the results

# get the new html of the page after if was updated with the search results
# from here you can work with bs4 again (though you can do all the things you do in bs4 also with Selenium)
html = driver.page_source

Upvotes: 1

Related Questions