Python Beautiful Soup Error : list index out of range

Question

I am getting the following error with the script, i want to do is,

if i get a correct URL, i want it to check with BeautifulSoup if there is a form with value button "Send"

Traceback (most recent call last): File "tester.py", line 27, in if viewstate[0]['value'] == "Send": IndexError: list index out of range

#!/usr/bin/env python

import urllib2
import sys
import os

url = sys.argv[1]
open_dir_list = open("dirlist.txt",'r')
dirs = open_dir_list.read().split("\n")
open_dir_list.close()

for dir in dirs:

        uri = url+"/"+dir

        try:
                from BeautifulSoup import BeautifulStoneSoup
                response = urllib2.urlopen(uri)
                if response.getcode() == 200:
                        s = uri
                        soup = BeautifulStoneSoup(uri)
                        viewstate = soup.findAll("input", {"type": "submit"})
                        if viewstate[0]['value'] == "Send":
                                print("Yay!!!")
                        else:
                                print("There's nothing here")
        except urllib2.HTTPError, e:
                if e.code == 401:
                        print "[!] Authorization Required %s " % (uri)
                elif e.code == 403:
                        print "[!] Forbidden %s " % (uri)
                elif e.code == 404:
                        print "[-] Not Found %s " % (uri)
                elif e.code == 503:
                        print "[!] Service Unavailable %s " % (uri)
                else:
                        print "[?] Unknwon"



print "\n:. FINISH :.\n"

It is working fine with this script, but it only checks only a given path

import urllib
f = urllib.urlopen("http://xxx.xxx.xxx.xxx/button.jsp")
s = f.read()
f.close()

from BeautifulSoup import BeautifulStoneSoup
soup = BeautifulStoneSoup(s)

viewstate = soup.findAll("input", {"type": "submit"})
if viewstate[0]['value'] == "Send":
        print(" Yay!!!")
else:
        print("No Submit Button")

Answer 1

Apart from what I mentioned in a comment you are not passing the returned html you are passing uri = url+"/"+dir instead of response.read() to BeautifulSoup so you are searching for the tag in uri which I imagine certainly does not contain any tags. You need to pass read as below:

     response = urllib2.urlopen(uri)
            if response.getcode() == 200:                       
                    soup = BeautifulStoneSoup(response.read())

If you want the first match use .find using if viewstate to make sure it matched something, you can also iterate over the file object getting a line at a time:

from BeautifulSoup import BeautifulStoneSoup
import urllib2
import sys

url = sys.argv[1]
with  open("dirlist.txt",'r') as f:
    for dir in f:
        uri = url + "/" + dir.rstrip()
        try:
            response = urllib2.urlopen(uri)
            if response.getcode() == 200:
                    soup = BeautifulStoneSoup(response.read())
                    viewstate = soup.find("input", {"type": "submit"})
                    if viewstate and viewstate.get("value") == "Send":
                        print("Shell is found!! Yay!!!")
                    else:
                        print("There's nothing here")
        except urllib2.HTTPError, e:
                if e.code == 401:
                    print "[!] Authorization Required %s " % (uri)
                elif e.code == 403:
                    print "[!] Forbidden %s " % (uri)
                elif e.code == 404:
                    print "[-] Not Found %s " % (uri)
                elif e.code == 503:
                    print "[!] Service Unavailable %s " % (uri)
                else:
                    print "[?] Unknwon"



    print "\n:. FINISH :.\n"