Soup.find_all returns None even the element exists

Question

It returns none for the 5+ pages even the class exists there.

The URL Which works fine:

https://www.ebay.com/sch/i.html?_from=R40&_nkw=Apple&_sacat=0&rt=nc&LH_Sold=1&LH_Complete=1&_ipg=200&_pgn=1

But it doesn't work for 5-6 pages

https://www.ebay.com/sch/i.html?_from=R40&_nkw=Apple&_sacat=0&rt=nc&LH_Sold=1&LH_Complete=1&_ipg=200&_pgn=5

My Code So far:

response = requests.get(url)
soup = BeautifulSoup(response.content,'html.parser')
app = soup.find_all('li',class_ = 's-item')
for x in app:
    print(x)

Printing app > Prints empty LIST: []

I have checked it manually, The class exists on all the pages.

Answer 1

Your bot may be detected and the 5th page often occurs to be a captcha or pop-up window.

Try to use another library like Selenium to witness your bit behavior in browser, or screenshot the window at every page query

Answer 2

The content is probably dynamically generated with JavaScript. You should use Selenium to run the javascript components, then extract the information you want from the resulting webpage