CODEWITHSUNDEEP
pythonbeautifulsoup
Life is complex
Life is complex

Reputation: 15629

Obtain <h2> tag related to <li> tags

I'm using BeautifulSoup to scrape a website. I can obtain all the data in the <li class="level-item"> tag, but I need to obtain the date in the <h2> tag related to the corresponding <li> tags.

Desired output: 

05182018,/somedirectoryname/anothername/009,sometext,another value,long description 
05182018,/somedirectoryname/anothername/008,sometext,another value,long description 
03092018,/somedirectoryname/anothername/007,sometext,another value,long description 
03092018,/somedirectoryname/anothername/006,sometext,another value,long description 
03092018,/somedirectoryname/anothername/005,sometext,another value,long description 
03092018,/somedirectoryname/anothername/004,sometext,another value,long description

Web page structure:

<h2>May 18, 2018<h2>
<ul>

 <li class="level-item"><a href=“/somedirectoryname/anothername/009”><span class=“some text”>another value</span> long description </a></li>

 <li class="level-item"><a href=“/somedirectoryname/anothername/008”><span class=“some text”>another value</span> long description </a></li>

</ul>

<h2>March 9, 2018<h2>
<ul>
<li class="level-item"><a href=“/somedirectoryname/anothername/007”><span class=“some text”>another value</span> long description </a></li>

<li class="level-item"><a href=“/somedirectoryname/anothername/006”><span class=“some text”>another value</span> long description </a></li>

<li class="level-item"><a href=“/somedirectoryname/anothername/005”><span class=“some text”>another value</span> long description </a></li>

<li class="level-item"><a href=“/somedirectoryname/anothername/004”><span class=“some text”>another value</span> long description </a></li>

</ul>

<h2>December 1, 2017<h2>
<ul>

<li class="level-item"><a href=“/somedirectoryname/anothername/003”><span class=“some text”>another value</span> long description </a></li>

<li class="level-item"><a href=“/somedirectoryname/anothername/002”><span class=“some text”>another value</span> long description </a></li>

<li class="level-item"><a href=“/somedirectoryname/anothername/001”><span class=“some text”>another value</span> long description </a></li>

Snippet of my code: I only need to obtain the date(s) directly above the <ul> tag related to the <li> tags. 

date = results_table.find_all('h2', string=re.compile('January|February|March|April|May|June|July|August|September|October|November|December'))
    locale.setlocale(locale.LC_ALL, 'en_US')
    changeDateFormat = date.text.strip()
    datePublished = datetime.datetime.strptime(changeDateFormat, '%B %d, %Y').strftime('%m%d%Y')
    ul = results_table.find('ul')

    for item in results_table.find_all('li', {'class': 'level-item'}):
        # try to obtain the correct date
        print(ul.previous_element)
        for nextLink in item.find_all('a'):
            for ad_id in nextLink.find_all('span'):
                print(ad_id.text.strip())

Upvotes: 1

Views: 294

Answers (1)

Keyur Potdar
Keyur Potdar

Reputation: 7238

After find all the <h2> tags using what you've done, you can get the corresponding <ul> tag using find_next() or .next_sibling. Then simply iterate over all the <li> tags.

Code:

for date_tag in results_table.find_all('h2'):
    date = date_tag.text
    for item in date_tag.find_next('ul').find_all('li'):
        print(date, item.a['href'], item.span['class'][0], item.get_text(',', strip=True), sep=',')

Output:

May 18, 2018,/somedirectoryname/anothername/009,some,another value,long description
May 18, 2018,/somedirectoryname/anothername/008,some,another value,long description
March 9, 2018,/somedirectoryname/anothername/007,some,another value,long description
March 9, 2018,/somedirectoryname/anothername/006,some,another value,long description
March 9, 2018,/somedirectoryname/anothername/005,some,another value,long description
March 9, 2018,/somedirectoryname/anothername/004,some,another value,long description
December 1, 2017,/somedirectoryname/anothername/003,some,another value,long description
December 1, 2017,/somedirectoryname/anothername/002,some,another value,long description
December 1, 2017,/somedirectoryname/anothername/001,some,another value,long description

Upvotes: 2

Related Questions