Can't fetch the content of a table from a website through web scraping using BeautifulSoup

Question

I'm trying to pull out data off the table from the section COVID-19 Statewise Status from https://www.mohfw.gov.in/ website.

I'm using BeautifulSoup for this, but when I try to read the contents of tbody, it outputs it as None. Here is my code:

import requests
from bs4 import BeautifulSoup
url = "https://www.mohfw.gov.in/"
soup = BeautifulSoup(requests.get(url).content, "lxml")

t1=soup.find(id="state-data")
t2=t1.find('div',class_='data-table table-responsive')
t3=t2.find('table')
tab=t3.find('tbody')
data=tab.find_all('tr')
for d in data:
   t=d.find_all('td')
   for t1 in t:
       val=t1.text
       print(val)

All the data of the table clearly present in the tbody tag. But when I tried to scrape them the output shows None. How to fetch them, someone please help. Thank You in advance!

Answer 1

The response from requests.get(url).content contains a comment in the html code, <!--<tbody>, which is causing issues down the line. Your line tab=t3.find('tbody') returns None

One way to get past this might be to remove the comment. See below.

import requests
from bs4 import BeautifulSoup
url = "https://www.mohfw.gov.in/"
soup = BeautifulSoup(requests.get(url).content, "lxml")

t1=soup.find(id="state-data")
t2=t1.find('div',class_='data-table table-responsive')

t3=t2.find('table')
# tab=t3.find('tbody')
t3 = str(t3).replace("<!--<tbody>", "")
t3 = BeautifulSoup(t3, "lxml")
data=t3.find_all('tr')
for d in data:
    t=d.find_all('td')
    for t1 in t:
        val=t1.text
        print(val)

Answer 2

The data is loaded from external source via Javascript. You can use requests to load this json data or use pandas:

import pandas as pd

df = pd.read_json("https://www.mohfw.gov.in/data/datanew.json")
print(df)

Prints:

      sno                                state_name  active  positive     cured   death  new_active  new_positive  new_cured  new_death state_code
0       2               Andaman and Nicobar Islands      97      7415      7191     127         103          7425       7195        127         35
1       1                            Andhra Pradesh   58140   1853183   1782680   12363       53880       1857352    1791056      12416         28
2       3                         Arunachal Pradesh    2539     33375     30677     159        2548         33664      30956        160         12
3       4                                     Assam   32625    485310    448442    4243       32975        488179     450924       4280         18
4       5                                     Bihar    3017    719939    707365    9557        2811        720207     707833       9563         10
5       6                                Chandigarh     311     61444     60327     806         278         61467      60383        806         04
6       7                              Chhattisgarh    8564    991171    969212   13395        8007        991653     970244      13402         22
7       8  Dadra and Nagar Haveli and Daman and Diu      60     10516     10452       4          61         10520      10455          4         26
8      10                                     Delhi    1996   1432381   1405460   24925        1918       1432778    1405927      24933         07
9      11                                       Goa    3066    164654    158591    2997        2920        164957     159029       3008         30
10     12                                   Gujarat    5639    822485    806812   10034        5159        822620     807424      10037         24
11     13                                   Haryana    2337    767580    755968    9275        2200        767726     756231       9295         06
12     14                          Himachal Pradesh    2408    200603    194747    3448        2276        200791     195062       3453         02
13     15                         Jammu and Kashmir    7759    312156    300135    4262        7181        312584     301134       4269         01
14     16                                 Jharkhand    1489    344665    338076    5100        1417        344775     338256       5102         20
15     17                                 Karnataka  123156   2811320   2654139   34025      118615       2815029    2662250      34164         29
16     18                                    Kerala  100135   2816843   2704554   12154      100881       2829460    2716284      12295         32
17     19                                    Ladakh     365     19838     19271     202         360         19871      19309        202         37
18     20                               Lakshadweep     319      9471      9106      46         315          9504       9142         47         31
19     21                            Madhya Pradesh    1980    789350    778584    8786        1707        789415     778902       8806         23
20     22                               Maharashtra  127523   5979051   5733215  118313      126468       5987521    5742258     118795         27
21     23                                   Manipur    9298     64418     54065    1055        9214         64993      54714       1065         14
22     24                                 Meghalaya    4196     45555     40574     785        4273         45976      40915        788         17
23     25                                   Mizoram    4227     17979     13667      85        4424         18409      13900         85         15
24     26                                  Nagaland    1844     24374     22055     475        1757         24438      22204        477         13
25     27                                    Odisha   32099    880533    844801    3633       30859        883490     848960       3671         21
26     28                                Puducherry    3364    115080    109990    1726        3214        115364     110423       1727         34
27     29                                    Punjab    6477    592658    570327   15854        5968        593063     571207      15888         03
28     30                                 Rajasthan    2691    951256    939664    8901        2388        951393     940101       8904         08
29     31                                    Sikkim    2448     19321     16580     293        2430         19458      16732        296         11
30     32                                Tamil Nadu   61329   2429924   2337209   31386       56886       2436819    2348353      31580         33
31     34                                 Telangana   17246    614399    593577    3576       16640        615574     595348       3586         36
32     33                                   Tripura    3910     62745     58181     654        3747         63140      58735        658         16
33     35                               Uttarakhand    2964    338807    328799    7044        2896        338978     329030       7052         05
34     36                             Uttar Pradesh    4163   1704476   1678089   22224        3910       1704678    1678486      22282         09
35     37                               West Bengal   22740   1483586   1443456   17390       22508       1485438    1445493      17437         19
36  11111                                            662521  29977861  28926038  389302      643194      30028709   28994855     390660