Beautifulsoup problem of scraping text in array

Question

Data=

<div class="dojoxGridView" id="dojox_grid__View_1" role="presentation" style="width: 1900px; height: 721px; left: 1px; top: 0px;" widgetid="dojox_grid__View_1">
       <input class="dojoxGridHiddenFocus" dojoattachpoint="hiddenFocusNode" role="presentation" type="checkbox"/>
       <input class="dojoxGridHiddenFocus" role="presentation" type="checkbox"/>
       <div class="dojoxGridScrollbox" dojoattachpoint="scrollboxNode" role="presentation" style="height: 721px;">
        <div class="dojoxGridContent" dojoattachpoint="contentNode" hidefocus="hidefocus" role="presentation" style="height: 504px; width: 1900px;">
         <div role="presentation" style="position: absolute; left: 0px; top: 0px;">
          <div aria-selected="false" class="dojoxGridRow" role="row" style="">
           <table border="0" cellpadding="0" cellspacing="0" class="dojoxGridRowTable" role="presentation" style="width: 1900px;">
            <tbody>
             <tr>
              <td class="dojoxGridCell" idx="0" role="gridcell" style="display:none;width:100px;" tabindex="-1">
               78126
              </td>
              <td class="dojoxGridCell" idx="1" role="gridcell" style="width:10%;" tabindex="-1">
               Approved Plan
              </td>
              <td class="dojoxGridCell" idx="2" role="gridcell" style="width:10%;" tabindex="-1">
               G-10
              </td>
              <td class="dojoxGridCell" idx="3" role="gridcell" style="width:40%;" tabindex="-1">
               ROOF PLAN
              </td>
             </tr>
            </tbody>
           </table>
          </div>

Input=

    source = driver.page_source
soup = BeautifulSoup(source, "lxml")
print(soup. prettify())
for article in soup.find_all('div', class_='dojoxGridContent'):
drawing_no = article.find_all('td', class_='dojoxGridCell', idx='3')
# ->need one more line to extract text
print(""drawing_no")

Output=

<td class="dojoxGridCell" idx="3" role="gridcell" style="width:40%;" tabindex="-1">ROOF PLAN</td> ...

I just want to extract "ROOF PLAN" how should I edit my code? I tried drawing_no.text and drawing_no.value but it said "no attribute". Thanks for your help!

Answer 1

You can use the idx attribute and select by its value

print(soup.select_one("[idx='3']").text.strip())

Answer 2

try followig code

source="""<div class="dojoxGridView" id="dojox_grid__View_1" role="presentation" style="width: 1900px; height: 721px; left: 1px; top: 0px;" widgetid="dojox_grid__View_1">
       <input class="dojoxGridHiddenFocus" dojoattachpoint="hiddenFocusNode" role="presentation" type="checkbox"/>
       <input class="dojoxGridHiddenFocus" role="presentation" type="checkbox"/>
       <div class="dojoxGridScrollbox" dojoattachpoint="scrollboxNode" role="presentation" style="height: 721px;">
        <div class="dojoxGridContent" dojoattachpoint="contentNode" hidefocus="hidefocus" role="presentation" style="height: 504px; width: 1900px;">
         <div role="presentation" style="position: absolute; left: 0px; top: 0px;">
          <div aria-selected="false" class="dojoxGridRow" role="row" style="">
           <table border="0" cellpadding="0" cellspacing="0" class="dojoxGridRowTable" role="presentation" style="width: 1900px;">
            <tbody>
             <tr>
              <td class="dojoxGridCell" idx="0" role="gridcell" style="display:none;width:100px;" tabindex="-1">
               78126
              </td>
              <td class="dojoxGridCell" idx="1" role="gridcell" style="width:10%;" tabindex="-1">
               Approved Plan
              </td>
              <td class="dojoxGridCell" idx="2" role="gridcell" style="width:10%;" tabindex="-1">
               G-10
              </td>
              <td class="dojoxGridCell" idx="3" role="gridcell" style="width:40%;" tabindex="-1">
               ROOF PLAN
              </td>
             </tr>
            </tbody>
           </table>
          </div>"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(source,"html.parser")
for article in soup.find_all('div', class_='dojoxGridContent'):
  drawing_no = article.find('td', class_='dojoxGridCell', idx='3')
  if drawing_no:
    print(drawing_no.get_text())

Answer 3

please try below code. But in general if you pass in idx=3 it will only return one single element. If you want to extract text from multiple element you might want to use a more general identifier.

import lxml
from lxml import html

html_string = """
<div class="dojoxGridView" id="dojox_grid__View_1" role="presentation" style="width: 1900px; height: 721px; left: 1px; top: 0px;" widgetid="dojox_grid__View_1">
  <input class="dojoxGridHiddenFocus" dojoattachpoint="hiddenFocusNode" role="presentation" type="checkbox"/>
  <input class="dojoxGridHiddenFocus" role="presentation" type="checkbox"/>
  <div class="dojoxGridScrollbox" dojoattachpoint="scrollboxNode" role="presentation" style="height: 721px;">
    <div class="dojoxGridContent" dojoattachpoint="contentNode" hidefocus="hidefocus" role="presentation" style="height: 504px; width: 1900px;">
      <div role="presentation" style="position: absolute; left: 0px; top: 0px;">
        <div aria-selected="false" class="dojoxGridRow" role="row" style="">
          <table border="0" cellpadding="0" cellspacing="0" class="dojoxGridRowTable" role="presentation" style="width: 1900px;">
            <tbody>
              <tr>
                <td class="dojoxGridCell" idx="0" role="gridcell" style="display:none;width:100px;" tabindex="-1">
                78126
                </td>
                <td class="dojoxGridCell" idx="1" role="gridcell" style="width:10%;" tabindex="-1">
                Approved Plan
                </td>
                <td class="dojoxGridCell" idx="2" role="gridcell" style="width:10%;" tabindex="-1">
                G-10
                </td>
                <td class="dojoxGridCell" idx="3" role="gridcell" style="width:40%;" tabindex="-1">
                ROOF PLAN
                </td>
              </tr>
            </tbody>
          </table>
        </div>
      </div>
    </div>
  </div>
</div>
"""

tree = html.fromstring(html_string)
ROOFPLAN = tree.xpath('//tbody/tr//td[@idx="3"]/text()')
print(''.join(ROOFPLAN).strip())