Reputation: 199
Question: How do I recreate the effect of manually opening a new tab in scrapy-playwright?
Explanation: I have an issue in scrapy-playwright where a site does not recognize login credentials unless a new tab is manually opened in the headful chrome browser. Then, when navigating to the url from the manually created new tab page, the login credentials are recognized.
If you use the tab that was opened by playwright, it does not work. Either the username/password is wrong, or the page reloads without any changes. Navigating to a different page and then back to the login page yields the same result. Creating a new window in the same session does not work either. Adding additional delay does not work. Manually clicking submit does not work. In Firefox I get the "Failed to create secure connection" message. Accepting cookies doesn't do anything. --disable-web-security
, --ignore-certificate-errors
does nothing. Enabling cookies in the scrapy settings does not seem to do anything either. Everything seems to work fine on other sites.
I am 99% sure this is not an intentional security measure.
This is the site: https://shop.allmendinger.eu/account/login
Login code example: Not doing anything weird
def parse(self, response):
yield Request(
callback=self.login,
url="https://shop.allmendinger.eu/",
meta=dict(
playwright=True,
playwright_include_page=True,
errback=self.errback_close_page
)
)
async def login(self, response):
page = response.meta["playwright_page"]
print("Accepting cookies")
await page.locator('"Alle Cookies akzeptieren"').click() # locate the element by the button text
print("Filling login username")
await page.get_by_placeholder("E-Mail-Adresse eingeben ...").fill("[USERNAME]")
print("Filling login password")
await page.get_by_placeholder("Passwort eingeben ...").fill("[PASSWORD]")
print("Pressing enter")
await page.get_by_placeholder('Passwort eingeben ...').focus()
await page.keyboard.press("Enter")
page.wait_for_url("https://shop.allmendinger.eu/account")
# page.locator('"Anmelden"').click()
# await page.wait_for_url('https://shop.allmendinger.eu/account')
# await page.wait_for_selector('.icon-avatar')
screenshot = await page.screenshot(path="example.png", full_page=True)
Upvotes: 0
Views: 253