Python HTML fill and submit form

Question

I am trying to write a Python script that goes to a form on a internal website (with the name "DefaultForm") and fills in the input name="username" field in the form with 'user001', the input name="password" field with 'pass001' and click on submit

Tried doing this with Selenium and it works. But want to accomplish the same task with Requests(and BeautifulSoup for some html scraping later on)

Code I wrote which does NOT work!

url = 'http://SERVER:PORT/dashboard/portal'
payload = {'username':'user001','password':'pass001'}
r = requests.get(url, params=payload)
print(r.text)

I check the content in r.text before and after the requests.get(..) and both are the same.

Can anyone help me on how to do this ?

Edit/Update: Also tried this to submit my form using lxml but there seems to be an error which i can't seem to get my head around

page = parse(url).getroot()
page.forms[0].fields['username'] = 'user001'
page.forms[0].fields['password'] = 'pass001'
result = parse(submit_form(page.forms[0]).encode('utf-8')).getroot()
print(result.text)

This is the console result i get

runfile('C:/Users/mgreza/Downloads/WinPython-64bit-3.5.1.3/notebooks/temp.py', wdir='C:/Users/mgreza/Downloads/WinPython-64bit-3.5.1.3/notebooks') Traceback (most recent call last):

File "", line 1, in runfile('C:/Users/mgreza/Downloads/WinPython-64bit-3.5.1.3/notebooks/temp.py', wdir='C:/Users/mgreza/Downloads/WinPython-64bit-3.5.1.3/notebooks')

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 841, in runfile execfile(filename, namespace)

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 103, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/mgreza/Downloads/WinPython-64bit-3.5.1.3/notebooks/temp.py", line 13, in result = parse(submit_form(page.forms[0]).encode('utf-8')).getroot()

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\lxml\html__init__.py", line 1110, in submit_form return open_http(form.method, url, values)

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\lxml\html__init__.py", line 1131, in open_http_urllib return urlopen(url, data)

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\urllib\request.py", line 162, in urlopen return opener.open(url, data, timeout)

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\urllib\request.py", line 463, in open req = meth(req)

File "C:\Users\mgreza\Downloads\WinPython-64bit-3.5.1.3\python-3.5.1.amd64\lib\urllib\request.py", line 1170, in do_request_ raise TypeError(msg)

TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.

Please help!!!!!!!

This is the page I am trying to submit







  


  


  XYZ









   




  
    
      
        

          

            
            
            
            

            

              
                
                  
                    
                      
                        
                          
                            
                          
                          
                            
                          
                          
                            
                          
                        
                        
                          
                            
                          
                          
                            
                              
                                
                                    
                                    
                                      
                                        
                                           
                                        
                                        
                                           
                                          
                                            
                                          
                                           
                                          
                                        
                                        
                                           
                                        
                                      
                                    
                                  
                                  
                                    
                                  
                                  
                                    
                                      
                                        
                                           
                                        
                                        
                                          
                                            
                                              
                                            
                                          
                                        
                                        
                                          
                                            

                                          
                                        

                                        
                                          
                                            
                                              
                                                
                                                  
                                                    
                                                      
                                                      

                                                        
                                                          
                                                            
                                                              
                                                              
                                                                
                                                              
                                                              
                                                                
                                                              
                                                              
                                                            
                                                            
                                                               
                                                              Please sign in

                                                               
                                                            
                                                            
                                                               
                                                               
                                                              
                                                                
                                                              
                                                               
                                                            
                                                            
                                                               
                                                              User ID
                                                              
                                                                
                                                              
                                                               
                                                            
                                                            
                                                               
                                                              
                                                                
                                                              

                                                               
                                                            
                                                            
                                                               
                                                               
                                                              
                                                                
                                                              
                                                               
                                                            
                                                            
                                                               
                                                              Password

EDIT/UPDATE: Logs when POST-ing form

Request Header:- Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Accept-Encoding:gzip, deflate Accept-Language:en-US,en;q=0.8,en-GB;q=0.6 Cache-Control:max-age=0 Connection:keep-alive Content-Length:140 Content-Type:application/x-www-form-urlencoded Cookie:JSESSIONID=1ax740u3chasqa4rmen8ifq5b; SCI_DLSSO=U2Vzc2lvbklEaGl1YWFseGJqeGx1MTN1OGFtaXpva3Yybw== Host:10.1.28.189:5010 Origin:http://SERVER:PORT Referer:http://SERVER:PORT/dashboard/ Upgrade-Insecure-Requests:1 User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 Response Header:- Cache-Control:no-cache Content-Type:text/html; charset=utf-8 Expires:Thu, 01 Jan 1970 00:00:00 GMT Last-Modified:Thu, 30 Jun 2016 01:47:48 GMT Pragma:no-cache Set-Cookie:JSESSIONID=17zql1r4hdylrfg54lardx14p;Path=/dashboard/;HttpOnly Set-Cookie:SCI_DLSSO=U2Vzc2lvbklEMTd6cWwxcjRoZHlscmZnNTRsYXJkeDE0cA==;Path=/;HttpOnly Transfer-Encoding:chunked X-FRAME-OPTIONS:SAMEORIGIN Form Data:- action=JLoginUser&serverTimeStamp=1467251264480&clientTimeStamp=2146&clientIP=10.220.12.101&username=user001&password=pass001&submit=Sign+In

Python HTML fill and submit form

Answers (1)

Related Questions