cforcomputer
cforcomputer

Reputation: 199

Login auth not working on pages opened by scrapy-playwright

Question: How do I recreate the effect of manually opening a new tab in scrapy-playwright?

Explanation: I have an issue in scrapy-playwright where a site does not recognize login credentials unless a new tab is manually opened in the headful chrome browser. Then, when navigating to the url from the manually created new tab page, the login credentials are recognized.

If you use the tab that was opened by playwright, it does not work. Either the username/password is wrong, or the page reloads without any changes. Navigating to a different page and then back to the login page yields the same result. Creating a new window in the same session does not work either. Adding additional delay does not work. Manually clicking submit does not work. In Firefox I get the "Failed to create secure connection" message. Accepting cookies doesn't do anything. --disable-web-security, --ignore-certificate-errors does nothing. Enabling cookies in the scrapy settings does not seem to do anything either. Everything seems to work fine on other sites.

I am 99% sure this is not an intentional security measure.

This is the site: https://shop.allmendinger.eu/account/login

Login code example: Not doing anything weird

     def parse(self, response):
            yield Request(
                callback=self.login,
                url="https://shop.allmendinger.eu/",
                meta=dict(
                playwright=True,
                playwright_include_page=True,
                errback=self.errback_close_page
                )
            )
    async def login(self, response):
        page = response.meta["playwright_page"]
        print("Accepting cookies")
        await page.locator('"Alle Cookies akzeptieren"').click() # locate the element by the button text
        print("Filling login username")
        await page.get_by_placeholder("E-Mail-Adresse eingeben ...").fill("[USERNAME]")
        print("Filling login password")
        await page.get_by_placeholder("Passwort eingeben ...").fill("[PASSWORD]")
        print("Pressing enter")
        await page.get_by_placeholder('Passwort eingeben ...').focus()
        await page.keyboard.press("Enter")
        page.wait_for_url("https://shop.allmendinger.eu/account")
        # page.locator('"Anmelden"').click()

        # await page.wait_for_url('https://shop.allmendinger.eu/account')
        # await page.wait_for_selector('.icon-avatar')
        screenshot = await page.screenshot(path="example.png", full_page=True)

Upvotes: 0

Views: 253

Answers (0)

Related Questions