CODEWITHSUNDEEP
pythonpython-requestslxml
David542
David542

Reputation: 110382

Should I use .text or .content when parsing a Requests response?

I occasionally use res.content or res.text to parse a response from Requests. In the use cases I have had, it didn't seem to matter which option I used.

What is the main difference in parsing HTML with .content or .text? For example:

import requests 
from lxml import html
res = requests.get(...)
node = html.fromstring(res.content)

In the above situation, should I be using res.content or res.text? What is a good rule of thumb for when to use each?

Upvotes: 13

Views: 14852

Answers (2)

Akshay Katiha
Akshay Katiha

Reputation: 470

I think res.content and res.text are not comparable and which one you should use depends on your use case. Use res.text for textual responses and use res.content for binary files like images or PDF.

All internet content is received in the form of bytes. To prevent converting it to Unicode format, Requests provides text attribute. Assume text as a sugar-coated version of content.

import requests
res = requests.get("https://httpbin.org/get")
res.text == res.content.decode(encoding == res.encoding)
True

Upvotes: 0

Francisco
Francisco

Reputation: 11496

From the documentation:

When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers. The text encoding guessed by Requests is used when you access r.text. You can find out what encoding Requests is using, and change it, using the r.encoding property:

>>> r.encoding
'utf-8'
>>> r.encoding = 'ISO-8859-1'

If you change the encoding, Requests will use the new value of r.encoding whenever you call r.text. You might want to do this in any situation where you can apply special logic to work out what the encoding of the content will be. For example, HTTP and XML have the ability to specify their encoding in their body. In situations like this, you should use r.content to find the encoding, and then set r.encoding. This will let you use r.text with the correct encoding.

So r.content is used when the server returns binary data, or bogus encoding headers, to try to find the correct encoding inside a meta tag.

Upvotes: 15

Related Questions