Web scrape of forbes website using requests-html

Question

I'm trying to scrape the list from https://www.forbes.com/best-states-for-business/list/#tab:overall

import requests_html
session= requests_html.HTMLSession()
r = session.get("https://www.forbes.com/best-states-for-business/list/#tab:overall")
r.html.render()
body=r.text.find('#list-table-body')
print(body)

This returns -1, not the table content. How can I get the actual table content?

Answer 1

Data is loaded dynamically from external source via API and You can grab the required data using requests module only then store via pandas DataFrame.

import pandas as pd
import requests

headers = {'user-agent':'Mozilla/5.0'}
url = 'https://www.forbes.com/ajax/list/data?year=2019&uri=best-states-for-business&type=place'
r= requests.get(url,headers=headers)

df = pd.DataFrame(r.json())
print(df)

Output:

 position  rank            name uri  ... regulatoryEnvironment  economicClimate  growthProspects  lifeQuality
0          1     1  North Carolina  nc  ...                     1               13               13           16    
1          2     2           Texas  tx  ...                    21                4                1           15    
2          3     3            Utah  ut  ...                     6                8                7            9    
3          4     4        Virginia  va  ...                     3               20               24            1    
4          5     5         Florida  fl  ...                     7                3                5           18    
5          6     6         Georgia  ga  ...                     9                7               11           23    
6          7     7       Tennessee  tn  ...                     4               11               14           29    
7          8     8      Washington  wa  ...                    29                6                8           30    
8          9     9        Colorado  co  ...                    19                2                4           21    
9         10    10           Idaho  id  ...                     8               10                2           24    
10        11    11        Nebraska  ne  ...                     2               28               36           19    
11        12    12         Indiana  in  ...                     5               25               25            7    
12        13    13          Nevada  nv  ...                    14               14                6           48    
13        14    14    South Dakota  sd  ...                    13               39               20           28    
14        15    15       Minnesota  mn  ...                    16               16               27            3    
15        16    16  South Carolina  sc  ...                    17               15               12           39    
16        17    17            Iowa  ia  ...                    11               36               35           10    
17        18    18         Arizona  az  ...                    18               12                3           35    
18        19    19   Massachusetts  ma  ...                    37                5               15            4    
19        20    20          Oregon  or  ...                    36                9                9           38    
20        21    21       Wisconsin  wi  ...                    10               19               37            8    
21        22    22        Missouri  mo  ...                    25               26               18           17    
22        23    23        Delaware  de  ...                    42               37               19           43    
23        24    24        Oklahoma  ok  ...                    15               31               33           31    
24        25    25   New Hampshire  nh  ...                    32               21               22           22    
25        26    26    North Dakota  nd  ...                    22               45               26           42    
26        27    27    Pennsylvania  pa  ...                    35               23               40           12    
27        28    28        New York  ny  ...                    34               18               21           14    
28        29    29            Ohio  oh  ...                    26               22               44            2    
29        30    30         Montana  mt  ...                    28               35               17           45    
30        31    31      California  ca  ...                    40                1               10           27    
31        32    32         Wyoming  wy  ...                    12               49               23           36    
32        33    33        Arkansas  ar  ...                    20               33               39           41    
33        34    34        Maryland  md  ...                    41               27               29           26    
34        35    35        Michigan  mi  ...                    22               17               41           13    
35        36    36          Kansas  ks  ...                    24               32               42           32    
36        37    37        Illinois  il  ...                    39               30               45           11    
37        38    38        Kentucky  ky  ...                    33               41               34           25    
38        39    39      New Jersey  nj  ...                    49               29               30            5    
39        40    40         Alabama  al  ...                    27               38               31           44    
40        41    41    Rhode Island  ri  ...                    44               40               32           20    
41        42    42     Mississippi  ms  ...                    30               46               47           37    
42        43    43     Connecticut  ct  ...                    43               42               48            6    
43        44    44           Maine  me  ...                    48               34               28           34    
44        45    45         Vermont  vt  ...                    45               43               38           33    
45        46    46       Louisiana  la  ...                    47               47               46           47    
46        47    47          Hawaii  hi  ...                    38               24               49           40    
47        48    48      New Mexico  nm  ...                    46               44               15           49    
48        49    49   West Virginia  wv  ...                    50               48               50           46    
49        50    50          Alaska  ak  ...                    31               50               43           50    

[50 rows x 15 columns]