CODEWITHSUNDEEP
pythonhttphttp-headerspython-requestsresponse
Mr Yahli
Mr Yahli

Reputation: 321

Requests HTTP headers

I've a trouble with the HTTP headers that the module Requests returns.

I'm using the following code :

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests 

response = requests.get("http://www.google.co.il",proxies={'http': '','https':''})

data = response.text 
# response.text returns the appropriate html code 
# (<!doctype html><html dir="rtl" itemscope=""....)

if response.status_code == requests.codes.ok:
    # How do I send those headers to the conn (browser)   
    print "HEADERS: " + str(response.headers) 
    conn.send(data)

I'm trying to send a GET request to www.google.co.il, and send the response to the browser (on the example I called it "conn"). The problem is that the browser won't show the received HTML code and instead I'm receiving ERR_EMPTY_RESPONSE. The headers in the response are :

HEADERS: {'Content-Length': '5451', 'X-XSS-Protection': '1; mode=block', 'Content-Encoding': 'gzip', 'Set-Cookie': 'NID=103=RJzu4RTCNxkh-75dvKBHx-_jen9M8iPes_AdOIQqzBVZ0VPTz1PlQaAVLpwYOmxZlTKmcogiDb1VoY__Es0HqSNwlkmHl3SuBZC8_8XUfqh1PzdWTjrXRnB4S738M1lm; expires=Wed, 08-Nov-2017 10:05:46 GMT; path=/; domain=.google.co.il; HttpOnly', 'Expires': '-1', 'Server': 'gws', 'Cache-Control': 'private, max-age=0', 'Date': 'Tue, 09 May 2017 10:05:46 GMT', 'P3P': 'CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."', 'Content-Type': 'text/html; charset=windows-1255', 'X-Frame-Options': 'SAMEORIGIN'}

Someone told me that the problem is that I'm not sending any header to the browser. Is this really the problem ? Any other suggestions ? and if it is the problem, how do I send the appropriate headers to the browser ?

Edit: I forgot to mention that the connection is through a Proxy server.

Any help would be great!

Thanks alot, Yahli.

Upvotes: 1

Views: 3400

Answers (1)

t.m.adam
t.m.adam

Reputation: 15376

I couldn't find anything about geting the raw http response ( not response.raw ) in requests documentation so i wrote a function : 

def http_response(response):
    return 'HTTP/1.1 {} {}\r\n{}\r\n\r\n{}'.format(
        response.status_code, response.reason , 
        '\r\n'.join(k + ': ' + v for k, v in response.headers.items()), 
        response.content
    )

I tested it by setting Firefox HTTP proxy to localhost:port ( with a listening socket on port ) , and it works fine .

Alternatively you can get the host from conn.recv , open a new socket to that host, and send the data . Example : 

data = conn.recv(1024)
host = [ l.split(':')[1].strip() for l in data.splitlines() if l.startswith('Host:') ]
if len(host)  : 
    cli = socket.socket()
    cli.connect((host[0], 80))
    cli.send(data)
    response = ''
    while True : 
        data = cli.recv(1024)
        if not data.strip() : 
            break
        response += data
    conn.send(response)
    cli.close()

Where conn is the connection to the web browser . This is just a quick example , assuming you have only HTTP requests ( port 80 ) . There is room for much optimization

Upvotes: 2

Related Questions