CODEWITHSUNDEEP
pythonbeautifulsoup
Hellosiroverthere
Hellosiroverthere

Reputation: 315

Python - How to scrape a class one by one

So I have been trying to scrape a value one by one, meaning that I used a for loop to scrape a selected class and then try to print out item by item. The HTML is looking:

HTML

What I tried to do is following code:

select_tags = bs4.find_all('select', {'autocomplete': 'off'})
    test = []
    for select_tag in select_tags:

        if select_tag.select("option.disabled.ReminderRevealButton"):
            continue
        else:
            print(select_tag)
            test.append(select_tag.text)

Output that I am getting is:

['\nPlease select number\n\n                            (Number 1)                        \n\n                            (Number 2)                        \n\n                            (Number 3)                        \n\n                            (Number 4)                        \n\n                            (Number 5)                        \n\n                            (Number 6)                        \n\n                            (Number 7)                        \n\n                            (Number 7)                        \n\n                            (Number 8)                        \n\n                            (Number 9)                        \n\n                            (Number 10)                        \n\n                            (Number 11)                        \n']}

which indeed prints out the names. However the problem is that it prints it all out as one line and not (the numbers from HTML code) one by one.

What do I need to do to be able to solve so it prints Number 1,2,3,4,5.. one by one?

Upvotes: 0

Views: 81

Answers (1)

ewwink
ewwink

Reputation: 19154

it print one in line because you're looping the select tag and not option tag. The :not is not supported by BeautifulSoup.

select_tags = bs4.find_all('select', {'autocomplete': 'off'})
test = []

for select_tag in select_tags:
    for option in select_tag.select('option'):
    # if set(['ReminderRevealButton', 'disabled']) == set(option.get('class')):
    if "disabled ReminderRevealButton" in str(option):
        # print(str)
        # <option class="disabled ReminderRevealButton"> (Number 1) </option>
        continue
    else:
        print(option.get('class'))
        test.append(option.text.strip())

I'm not using option['class'] or option.get('class') because it return array of class, if the class has same length you can use set() to compare else need loop.

Upvotes: 1

Related Questions