CODEWITHSUNDEEP
pythonpython-2.7web-crawlerpython-requests
Manas Chaturvedi
Manas Chaturvedi

Reputation: 5540

Requests: Explanation of the .text format

I'm using the requests module along with Python 2.7 to build a basic web crawler. 

source_code = requests.get(url)
plain_text = source_code.text

Now, in the above lines of code, I'm storing the source code of the specified URL and other metadata inside the source_code variable. Now, in source_code.text, what exactly is the .text attribute? It is not a function. I couldn't find anything in the documentation which explains the origin or feature of .text either.

Upvotes: 2

Views: 3738

Answers (2)

Martijn Pieters
Martijn Pieters

Reputation: 1123520

requests.get() returns a Response object; it is that object that has the .text attribute; it is not the 'source code' of the URL, it is an object that lets you access the source code (the body) of the response, as well as other information. The Response.text attribute gives you the body of the response, decoded to unicode.

See the Response Content section of the Quickstart documentation:

When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers. The text encoding guessed by Requests is used when you access r.text.

Further information can be found in the API documentation, see the Response.text entry:

Content of the response, in unicode.

If Response.encoding is None, encoding will be guessed using chardet.

The encoding of the response content is determined based solely on HTTP headers, following RFC 2616 to the letter. If you can take advantage of non-HTTP knowledge to make a better guess at the encoding, you should set r.encoding appropriately before accessing this property.

You can also use Response.content to access the response body undecoded, as raw bytes.

Upvotes: 5

rbp
rbp

Reputation: 1898

in this line 

 source_code = requests.get(url)

source_code has a response object, not the source code.

it should be 

response = requests.get(url)
source_code = response.text

Upvotes: 1

Related Questions