CODEWITHSUNDEEP
pythonseleniumweb-scrapingscrapy
user12512567
user12512567

Reputation: 51

Selenium is not loading TikTok pages

I'm implementing a TikTok crawler using selenium and scrapy

start_urls = ['https://www.tiktok.com/trending']
....
def parse(self, response):
    options = webdriver.ChromeOptions()
    from fake_useragent import UserAgent
    ua = UserAgent()
    user_agent = ua.random
    options.add_argument(f'user-agent={user_agent}')
    options.add_argument('window-size=800x841')
    driver = webdriver.Chrome(chrome_options=options)
    driver.get(response.url)

The crawler open Chrome but it does not load videos. Image loading

The same problem happens also using Firefox No loading page using Firefox

The same problem using a simple script using Selenium

from selenium import webdriver
import time


driver = webdriver.Firefox()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

driver = webdriver.Chrome()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

Upvotes: 5

Views: 5743

Answers (2)

zebo
zebo

Reputation: 141

Did u try to navigate further within the selenium browser window? If an error 404 appears on following sites, I have a solution that worked for me:

I simply changed my User-Agent to "Naverbot" which is "allowed" by the robots.txt file from Tik Tok

(Robots.txt)

After changing that all sites and videos loaded properly.

Other user-agents that are listed under the "allow" segment should work too, if you want to add a rotation.

Upvotes: 3

Zeeshan Ahmad
Zeeshan Ahmad

Reputation: 39

You can use Windows IE. Instead of chrome or firefox

Videos will load in IE but IE's Layout of showing feed is somehow different from chrome and firefox.

Reasons, why your page, is not loading.

Few advance web apps check your browser history, profile data and cached to check the authentication of the user. One other thing you can do is run your default profile within your selenium It would be helpfull.

Upvotes: 0

Related Questions