CODEWITHSUNDEEP
pythonscrapyrequestheaderplaywright
Curious miss Pythonist'a
Curious miss Pythonist'a

Reputation: 1

The set_extra_http_headers method doesn't work

I am new to Playwright and trying to create a Scrapy Middleware that uses Playwright to make a request and returns a response.body.

The problem is that I am trying to send headers with my request, but both these methods don't work for me:

browser_context.set_extra_http_headers(headers)

or

page.set_extra_http_headers(headers)

My code looks like this:

async def navigate_page(self, url, headers):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        # At first I tried this method, but it didn't work, so I decided to set headers with page method
        # context.set_extra_http_headers(headers)
        page = await context.new_page()
        await page.set_extra_http_headers(headers)
        await page.goto(url)
        print(page.request.headers) # showed nothing, debug also showed empty headers
        page_content = await page.content()
        await browser.close()
        return page_content

My headers variable looks like this:

{'referer': 'some url'}

I am pretty sure that I missed something but I searched every article that Google could find and came here in desperation :)

Upvotes: 0

Views: 1347

Answers (1)

Joe
Joe

Reputation: 11

I am not sure if you have solved this yet. I am having a similar problem and I don't understand all the syntax yet. The docs are definitely better for Javascript.

I am able to set extra HTTP headers with this code though:

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True).new_context(ignore_https_errors=True, extra_http_headers={"namex": "romeo"}, user_agent="joe", locale="de-DE", )
                        
    page = browser.new_page()
    page.set_viewport_size({"width": 1600, "height": 1200})
            
    page.goto("http://example.com")
                       
    browser.close()

Upvotes: 1

Related Questions