CODEWITHSUNDEEP
pythonencode
Guilherme Duarte
Guilherme Duarte

Reputation: 33

Problems reading special characters in requests python. All kinds of unsuccessful attempts

I've tried to find similar problems here on the stack, but I haven't found solutions to my problem. I have the following code in python

        headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
            'Accept': '*/*',
            'Accept-Language': 'pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6',
            'Accept-Encoding': 'gzip, deflate, br',
            'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
            'X-Requested-With': 'XMLHttpRequest',
            'Connection': 'keep-alive',
        }


        response = requests.request("POST", URL_ENDPOINT, headers=headers, data=payload_novo)
        print(response.text)

The output is:

{"output": "\n                        <h3 class=\"titulo-servico\">Local de vota\u00e7\u00e3o</h3>\n                        <p></p>\n                        <p>Os dados informados (nome, data de nascimento ou filia\u00e7\u00e3o) n\u00e3o conferem com aqueles constantes do Cadastro Eleitoral.</p>\n                        <p></p>\n                        <button type=\"button\" \n                            class=\"btn btn-amarelo\" \n
                  onclick=\"exibirConsultaLocalVotacao()\">\n                            Nova consulta\n                        </button>\n                        </p>\n                        ", "type": "success"}

See that he came back with characters like "\\ u00e7 \ u00e3o". I've tried in every way to decode through decode, but the result is always the same

However, if I create a new python file and put this return in a variable and print it out, the result will be displayed without problems, that is, with the special characters working

myString= '''{"output": "\n                        <h3 class=\"titulo-servico\">Local de vota\u00e7\u00e3o</h3>\n                        <p></p>\n                        <p>Os dados informados (nome, data de nascimento ou filia\u00e7\u00e3o) n\u00e3o conferem com aqueles constantes do Cadastro Eleitoral.</p>\n                        <p></p>\n                        <button type=\"button\" \n                            class=\"btn btn-amarelo\" \n
                      onclick=\"exibirConsultaLocalVotacao()\">\n                            Nova consulta\n                        </button>\n                        </p>\n                        ", "type": "success"}'''    


print(myString)

The output, which is correct, is:

{"output": "
                        <h3 class="titulo-servico">Local de votação</h3>
                        <p></p>
                        <p>Os dados informados (nome, data de nascimento ou filiação) não conferem com aqueles constantes do Cadastro Eleitoral.</p>
                        <p></p>
                        <button type="button"
                            class="btn btn-amarelo"

                      onclick="exibirConsultaLocalVotacao()">
                            Nova consulta
                        </button>
                        </p>
                        ", "type": "success"}

Note that: Exit 1 segment:

<h3 class=\"titulo-servico\">Local de vota\u00e7\u00e3o</h3>

Exit 2 segment:

<h3 class="titulo-servico">Local de votação</h3>

Upvotes: 2

Views: 1066

Answers (1)

Hugo G
Hugo G

Reputation: 16496

This could be one of two problems:

  1. The response you are printing is actually a JSON payload. In this case the correct way of decoding that is not through text but json():

    print(response.json())

    or more specifically

    print(response.json()["output"])

  2. The second thing that could have gone wrong is requests' automatic charset encoding detection.

    You can verify it's correctly detecting utf-8 by printing the encoding field of the response:

    print(response.encoding)

    If it is something unexpected, you can explicitly set it before using the text or json fields:

    response.encoding = 'utf-8'

Upvotes: 1

Related Questions