Trying to log in to a website with requests

Question

Here is the site: https://nacionalidade.justica.gov.pt/

As you can see there's an input class. how can I send a request to the site with a paramater that will be sent to the input class?

After I type in my password, the page changes, how can I get the content of the new page?

This is the class I need to send the password to.

import requests
from lxml import html

payload = {'password' : 'mypassword... not gonna write it here',
       'ABE4A1723D5F2906F222936AD0E9BE0E' : 
       'ABE4A1723D5F2906F222936AD0E9BE0E'}
url = "https://nacionalidade.justica.gov.pt/"

session_requests = requests.session()
result = session_requests.get(url)

tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//input[@name='ABE4A1723D5F2906F222936AD0E9BE0E']/@value")))[0]

result = session_requests.post(
    url, 
    data = payload, 
    headers = dict(referer=url)
)

r =requests.get(url,headers= dict(referer = url))
tree = html.fromstring(result.content)
bucket_names = tree.xpath("//div[@class='repo-list--repo']/a/text()")

print(bucket_names)

I'm getting an index error and I don't know why

Traceback (most recent call last):
File "c:/Users/?/Desktop/?/?/new.py", line 12, in <module>

authenticity_token = list(set(tree.xpath("//input[@name='ABE4A1723D5F2906F222936AD0E9BE0E']/@value")))[0]
IndexError: list index out of range

Answer 1

Here is an example "How to do it correct way", but without your passwords & etc. So feel free to adopt it for your solution.

from lxml.html import fromstring
import requests

session = requests.Session()

html = session.get('https://nacionalidade.justica.gov.pt/').content
form = fromstring(html).xpath('//form')[0]

post = {}
for input in form.inputs:
    if not input.name:
        continue

    post[input.name] = input.value

print('Initial form data')
print(post)

response = session.post('https://nacionalidade.justica.gov.pt/Home/GetEstadoProcessoAjax', data=post)
print(response.text)

Outputs

Initial form data
{'__RequestVerificationToken': 'p3AmUxqx-6Ipv9EbujoUid5TAKTm76oVmYIaB2UZMv_QzcH7LXaYhGwcQcqXqjWlJjJyvcKgcX48brUkWNvrmz-q3MPlg8mZAm56EGUooYw1', 'SenhaAcesso': '', 'DD94BAEBEF20FBD64FF8CA12170D623D': None, 'Email': None}
<link href="/layout/assets/css/grafico_nacionalidade.css" rel="stylesheet" />
<style>
    .titulo {
        color: #3399cc;
    }

    /*Bloco para o estado do processo*/
    #block_container {
        font-size: 1.5rem;
        font-weight: bold;
    }
    #bloc1
    {
        display:inline;
    }
    #bloc2
    {
        display:inline;
        font-size:1rem;
        font-weight:normal;
    }

    /*-------------------------------------*/

</style>
<div>


        <div id="txtErro" style="background-color:#FBEDED; width:100%; ">
            <div style="padding:35px;">
                A senha <b></b> n&#227;o corresponde a nenhum processo de nacionalidade ativo.
                <p></p>
                <br />
                Verifique que digitou a sua senha corretamente.
                <p></p>
                <br />
                Caso n&#227;o encontre o seu processo dirija-se &#224; <a href="http://www.irn.mj.pt/IRN/sections/irn/contactos">conservat&#243;ria</a> onde entregou o seu pedido.

                Na <a href="http://www.irn.mj.pt/IRN/sections/irn/contactos">Conservatória dos Registos Centrais</a>, em Lisboa, sem marcação prévia é possível obter a sua senha de acesso.
            </div>
        </div>


</div>
<hr />
<p>
    <a href="/Home/PesquisaProcesso">Voltar</a>
</p>

Important:

1. You have to use one requests session object for all requests because there is cookies session with CSRF token feature activated.

2. You have to send all initial form data + your password or security number field

3. Maybe you have to send some extra HTTP headers for getting the response since it is an AJAX request (I believe).

Answer 2

Looks like there’s no input with that name. See if you can use a more stable selector.