CODEWITHSUNDEEP
pythonfor-loopweb-scrapingbeautifulsoup
reuben
reuben

Reputation: 91

webscraping code not working-python

So i'm brand new to python wrote abit of code for webscraping average prices for houses throughout the UK but it hardly every works for lists > 5. This is normally due to the website because when i hit it again it works...but only sometimes.

the price variable used to be one long statement but it kept messing up, so i split it into a for statement and this keeps popping up:

Traceback (most recent call last):
  File "airbnb.py", line 18, in <module>
    town = item [i]
  File "//anaconda/lib/python3.5/site-packages/bs4/element.py", line 997, in __getitem__
    return self.attrs[key]
KeyError: 1

this is my entire code:

df = pd.read_csv('towns.csv')
item=df.test
name=[]
l1=[]
date=[]

for i in range(len(item)):
    town = item [i]
    url='https://www.airbnb.co.uk/s/?page=1&source=filters&airbnb_plus_only=false&room_types%5B%5D=Entire%20home%2Fapt&ss_id=6eaibfgb&allow_override%5B%5D=&s_tag=QmOEOer6'
    index=url.find('?page=')
    url=url[:index] + str(town) + url[index:]
    r=requests.get(url)
    soup=BeautifulSoup(r.content,'lxml')
    price=(soup.find('div',{"class":"avg-price"}))
    for item in price:
        j=(soup.find('span',{"class":"price"}).text)
    j=j [1:][:3]
    l1.append(j)
    name.append(town)
    time.sleep(1)


    stamp=time.strftime('%x')

    date.append(stamp)

    print(town)
    print(l1)
    print(date)

Upvotes: 2

Views: 136

Answers (1)

IgnacioF
IgnacioF

Reputation: 51

I think the error is in j=j [1:][:3] I recommend that before doing that you check the length of the j variable because it may be too short.

Upvotes: 1

Related Questions