Newskooler
Reputation: 4255
Why this html table is not seen with BeautifulSoup?
I am trying to scrape the following table
url = https://www.bitmex.com/app/settlementHistory?start=100
but after trying in multiple ways - no success.
Here is what I tried:
Attempt 1
import pandas as pd
pd.read_html(url)
This returns a list of 1 which is empty (does not contain the table in question).
Attempt 2
I tried iterating over table rows (tr) but when I do this:
from bs4 import BeautifulSoup
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
soup.find_all('tr')
I get no table rows of the actual table...
Attempt 3
I tried finding all tables with soup.find_all('table') but again a similar result as above.
Can anyone please help me load this into a pd.DataFrame and more importantly explain why the previous attempts did not work at all?
Upvotes: 1
Views: 27
Answers (1)
Andrej Kesely
Reputation: 195458
The data is loaded from external URL in Json format. The easiest way how to load it to dataframe is to use pd.read_json():
df = pd.read_json(
"https://www.bitmex.com/api/v1/settlement?count=100&start=100&reverse=true&filter="
)
print(df)
Prints:
timestamp symbol settlementType settledPrice optionStrikePrice optionUnderlyingPrice bankrupt taxBase taxRate
0 2019-10-04 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 8500.0 8144.71 NaN NaN NaN
1 2019-09-27 12:00:00+00:00 TRXU19 Settlement 0.000002 NaN NaN NaN NaN NaN
2 2019-09-27 12:00:00+00:00 EOSU19 Settlement 0.000346 NaN NaN NaN NaN NaN
3 2019-09-27 12:00:00+00:00 ADAU19 Settlement 0.000005 NaN NaN NaN NaN NaN
4 2019-09-27 12:00:00+00:00 BCHU19 Settlement 0.026730 NaN NaN NaN NaN NaN
5 2019-09-27 12:00:00+00:00 XRPU19 Settlement 0.000030 NaN NaN NaN NaN NaN
6 2019-09-27 12:00:00+00:00 LTCU19 Settlement 0.006842 NaN NaN NaN NaN NaN
7 2019-09-27 12:00:00+00:00 ETHU19 Settlement 0.020660 NaN NaN NaN NaN NaN
8 2019-09-27 12:00:00+00:00 XBTU19 Settlement 7979.550000 NaN NaN NaN NaN NaN
9 2019-09-27 12:00:00+00:00 XBT7D_D95 Settlement 0.022190 9750.0 7979.55 NaN NaN NaN
10 2019-09-27 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 10750.0 7979.55 NaN NaN NaN
11 2019-09-20 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 9750.0 10151.10 NaN NaN NaN
12 2019-09-20 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 10750.0 10151.10 NaN NaN NaN
13 2019-09-13 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 10250.0 10301.01 NaN NaN NaN
14 2019-09-13 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 11500.0 10301.01 NaN NaN NaN
15 2019-09-06 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 9000.0 10874.27 NaN NaN NaN
16 2019-09-06 12:00:00+00:00 XBT7D_U105 Settlement 0.008040 10000.0 10874.27 NaN NaN NaN
17 2019-08-30 12:00:00+00:00 XBT7D_D95 Settlement 0.001780 9750.0 9579.34 NaN NaN NaN
18 2019-08-30 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 10750.0 9579.34 NaN NaN NaN
19 2019-08-23 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 9500.0 10153.85 NaN NaN NaN
20 2019-08-23 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 10500.0 10153.85 NaN NaN NaN
21 2019-08-16 12:00:00+00:00 XBT7D_D95 Settlement 0.012340 11250.0 10014.39 NaN NaN NaN
22 2019-08-16 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 12250.0 10014.39 NaN NaN NaN
23 2019-08-09 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 10000.0 11722.96 NaN NaN NaN
24 2019-08-09 12:00:00+00:00 XBT7D_U105 Settlement 0.006170 11000.0 11722.96 NaN NaN NaN
25 2019-08-02 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 9250.0 10505.43 NaN NaN NaN
26 2019-08-02 12:00:00+00:00 XBT7D_U105 Settlement 0.002430 10250.0 10505.43 NaN NaN NaN
27 2019-07-26 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 9750.0 9809.83 NaN NaN NaN
28 2019-07-26 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 10750.0 9809.83 NaN NaN NaN
29 2019-07-19 12:00:00+00:00 XBT7D_D95 Settlement 0.006720 11000.0 10307.18 NaN NaN NaN
30 2019-07-19 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 12250.0 10307.18 NaN NaN NaN
31 2019-07-12 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 10750.0 11685.10 NaN NaN NaN
32 2019-07-12 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 11750.0 11685.10 NaN NaN NaN
33 2019-07-05 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 11250.0 11256.95 NaN NaN NaN
34 2019-07-05 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 12500.0 11256.95 NaN NaN NaN
35 2019-06-28 12:00:00+00:00 TRXM19 Settlement 0.000003 NaN NaN NaN NaN NaN
36 2019-06-28 12:00:00+00:00 EOSM19 Settlement 0.000499 NaN NaN NaN NaN NaN
37 2019-06-28 12:00:00+00:00 ADAM19 Settlement 0.000007 NaN NaN NaN NaN NaN
38 2019-06-28 12:00:00+00:00 BCHM19 Settlement 0.035550 NaN NaN NaN NaN NaN
39 2019-06-28 12:00:00+00:00 XRPM19 Settlement 0.000034 NaN NaN NaN NaN NaN
40 2019-06-28 12:00:00+00:00 LTCM19 Settlement 0.009907 NaN NaN NaN NaN NaN
41 2019-06-28 12:00:00+00:00 ETHM19 Settlement 0.025620 NaN NaN NaN NaN NaN
42 2019-06-28 12:00:00+00:00 XBTM19 Settlement 11918.300000 NaN NaN NaN NaN NaN
43 2019-06-28 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 9250.0 11918.30 NaN NaN NaN
44 2019-06-28 12:00:00+00:00 XBT7D_U105 Settlement 0.014000 10250.0 11918.30 NaN NaN NaN
45 2019-06-21 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 8000.0 9821.26 NaN NaN NaN
46 2019-06-21 12:00:00+00:00 XBT7D_U105 Settlement 0.010910 8750.0 9821.26 NaN NaN NaN
47 2019-06-14 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 7500.0 8290.34 NaN NaN NaN
48 2019-06-14 12:00:00+00:00 XBT7D_U105 Settlement 0.000490 8250.0 8290.34 NaN NaN NaN
49 2019-06-07 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 7750.0 7931.67 NaN NaN NaN
50 2019-06-07 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 8750.0 7931.67 NaN NaN NaN
51 2019-05-31 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 7750.0 8229.91 NaN NaN NaN
52 2019-05-31 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 8500.0 8229.91 NaN NaN NaN
53 2019-05-24 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 6750.0 8038.43 NaN NaN NaN
54 2019-05-24 12:00:00+00:00 XBT7D_U105 Settlement 0.006700 7500.0 8038.43 NaN NaN NaN
55 2019-05-17 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 6000.0 7139.66 NaN NaN NaN
56 2019-05-17 12:00:00+00:00 XBT7D_U105 Settlement 0.008960 6500.0 7139.66 NaN NaN NaN
57 2019-05-10 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 5500.0 6281.85 NaN NaN NaN
58 2019-05-10 12:00:00+00:00 XBT7D_U105 Settlement 0.004490 6000.0 6281.85 NaN NaN NaN
59 2019-05-03 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 5000.0 5707.68 NaN NaN NaN
60 2019-05-03 12:00:00+00:00 XBT7D_U105 Settlement 0.003640 5500.0 5707.68 NaN NaN NaN
61 2019-04-26 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 5000.0 5160.06 NaN NaN NaN
62 2019-04-26 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 5500.0 5160.06 NaN NaN NaN
63 2019-04-19 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 4750.0 5261.75 NaN NaN NaN
64 2019-04-19 12:00:00+00:00 XBT7D_U105 Settlement 0.000220 5250.0 5261.75 NaN NaN NaN
65 2019-04-12 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 4750.0 5074.01 NaN NaN NaN
66 2019-04-12 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 5250.0 5074.01 NaN NaN NaN
67 2019-04-05 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 4000.0 5003.48 NaN NaN NaN
68 2019-04-05 12:00:00+00:00 XBT7D_U105 Settlement 0.015060 4250.0 5003.48 NaN NaN NaN
69 2019-03-29 12:00:00+00:00 TRXH19 Settlement 0.000006 NaN NaN NaN NaN NaN
70 2019-03-29 12:00:00+00:00 BCHH19 Settlement 0.041210 NaN NaN NaN NaN NaN
71 2019-03-29 12:00:00+00:00 XRPH19 Settlement 0.000074 NaN NaN NaN NaN NaN
72 2019-03-29 12:00:00+00:00 EOSH19 Settlement 0.001055 NaN NaN NaN NaN NaN
73 2019-03-29 12:00:00+00:00 ADAH19 Settlement 0.000017 NaN NaN NaN NaN NaN
74 2019-03-29 12:00:00+00:00 LTCH19 Settlement 0.014920 NaN NaN NaN NaN NaN
75 2019-03-29 12:00:00+00:00 ETHH19 Settlement 0.034360 NaN NaN NaN NaN NaN
76 2019-03-29 12:00:00+00:00 XBTH19 Settlement 4095.130000 NaN NaN NaN NaN NaN
77 2019-03-29 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3750.0 4095.13 NaN NaN NaN
78 2019-03-29 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 4250.0 4095.13 NaN NaN NaN
79 2019-03-22 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3750.0 3982.17 NaN NaN NaN
80 2019-03-22 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 4000.0 3982.17 NaN NaN NaN
81 2019-03-15 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3750.0 3881.87 NaN NaN NaN
82 2019-03-15 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 4000.0 3881.87 NaN NaN NaN
83 2019-03-08 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3500.0 3876.24 NaN NaN NaN
84 2019-03-08 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 4000.0 3876.24 NaN NaN NaN
85 2019-03-01 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3750.0 3810.54 NaN NaN NaN
86 2019-03-01 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 4000.0 3810.54 NaN NaN NaN
87 2019-02-22 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3500.0 3923.44 NaN NaN NaN
88 2019-02-22 12:00:00+00:00 XBT7D_U105 Settlement 0.004420 3750.0 3923.44 NaN NaN NaN
89 2019-02-15 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3250.0 3574.36 NaN NaN NaN
90 2019-02-15 12:00:00+00:00 XBT7D_U105 Settlement 0.002080 3500.0 3574.36 NaN NaN NaN
91 2019-02-08 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3250.0 3390.14 NaN NaN NaN
92 2019-02-08 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 3500.0 3390.14 NaN NaN NaN
93 2019-02-01 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3250.0 3423.25 NaN NaN NaN
94 2019-02-01 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 3750.0 3423.25 NaN NaN NaN
95 2019-01-25 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3500.0 3535.21 NaN NaN NaN
96 2019-01-25 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 3750.0 3535.21 NaN NaN NaN
97 2019-01-18 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3500.0 3611.65 NaN NaN NaN
98 2019-01-18 12:00:00+00:00 XBT7D_U105 Settlement 0.000000 3750.0 3611.65 NaN NaN NaN
99 2019-01-11 12:00:00+00:00 XBT7D_D95 Settlement 0.000000 3500.0 3633.44 NaN NaN NaN
Upvotes: 1
Related Questions
- Python Beautiful Soup can't find specific table
- Getting html table with BeautifulSoup
- Pandas read_html unable to read tables
- pandas read_html - no tables found
- unable to parse html table with Beautiful Soup
- Can't find an HTML Table using BeautifulSoup in Python
- Unable to webscrape HTML table with BeautifulSoup and load it into a Pandas dataframe with Python
- BeautifulSoup4 cannot locate table no matter what I try
- Table element not showing in BeautifulSoup
- Scraping an html table with beautiful soup into pandas