CODEWITHSUNDEEP
pythonbeautifulsoup
jFreezy C
jFreezy C

Reputation: 1

How can I extract the value inside the curly brackets in python beautiful soup?

HTML excerpt

This is my code so far:

s = BS(r.content, 'lxml')
findDiv = s.find('div', {'id':'BodyContentPlaceholder_T00DF23F8005_Col00'})
findTable = findDiv.findAll('table', {'style':'width: 100%; border-collapse: collapse;'})

for table in findTable:
        rows = table.findAll('tr')
        rows = rows[1:]
        for row in rows:
            cells = row.findAll('td')
            extension = cells[3].button.input.text
            print('docs page ' + str(extension))
            #extention = re.compile(pattern, text)
            docpage_url_ending = cells[3].find('value')
            print('url ' + str(docpage_url_ending))

I'm trying to get navigateUrl text from this

<input id="ctl00_BodyContentPlaceholder_C013_RadListView1_ctrl1_rlb1_ClientState" name="ctl00_BodyContentPlaceholder_C013_RadListView1_ctrl1_rlb1_ClientState" type="hidden" autocomplete="off" value="{&quot;text&quot;:&quot;Documents&quot;,&quot;value&quot;:&quot;&quot;,&quot;target&quot;:&quot;&quot;,&quot;navigateUrl&quot;:&quot;/procurement/procurement-bids/constructionprocurementdetail?Title=Lakeland Adult Daycare Center Roof Replacement 18-785&quot;,&quot;primary&quot;:false}">

Upvotes: 0

Views: 1285

Answers (2)

asklc
asklc

Reputation: 495

That's how you get to your navigateUrl:

import json
from bs4 import BeautifulSoup as BS

s = BS(r.content, 'lxml')
findDiv = s.find('div', {'id':'BodyContentPlaceholder_T00DF23F8005_Col00'})
findTable = findDiv.findAll('table', {'style':'width: 100%; border-collapse: collapse;'})

for table in findTable:
    rows = table.findAll('tr')
    rows = rows[1:]
    for row in rows:
        cells = row.findAll('td')
        extension = cells[3].button.input.text
        print('docs page ' + str(extension))
        #extention = re.compile(pattern, text)
        docpage_url_ending = json.loads(cells[3].button.input.attr['value'])['navigateUrl']
        print('url ' + docpage_url_ending)

Upvotes: 0

JosefAssad
JosefAssad

Reputation: 4128

import json
import bs4
import urllib

data = '<input id="ctl00_BodyContentPlaceholder_C013_RadListView1_ctrl1_rlb1_ClientState" name="ctl00_BodyContentPlaceholder_C013_RadListView1_ctrl1_rlb1_ClientState" type="hidden" autocomplete="off" value="{&quot;text&quot;:&quot;Documents&quot;,&quot;value&quot;:&quot;&quot;,&quot;target&quot;:&quot;&quot;,&quot;navigateUrl&quot;:&quot;/procurement/procurement-bids/constructionprocurementdetail?Title=Lakeland Adult Daycare Center Roof Replacement 18-785&quot;,&quot;primary&quot;:false}">'

# cook up some soup
soup = bs4.BeautifulSoup(data)
# extract the relevant attribute
vals_as_string = soup.html.body.input.attrs['value']
# it's urlencoded, so decode it
unquoted_vals_as_string = urllib.parse.unquote(vals_as_string)
# turns out, it's json
vals_as_json = json.loads(unquoted_vals_as_string)
# well, json converts to dict, so there's our target
navigateUrl = vals_as_json['navigateUrl']

Upvotes: 1

Related Questions