CODEWITHSUNDEEP
pythonhtmljsonbeautifulsoup
Kin. Z
Kin. Z

Reputation: 43

Extract JSON from HTML Script tag with BeautifulSoup in Python

I have the following HTML, and what should I do to extract the JSON from the variable: window.__INITIAL_STATE__

<!DOCTYPE doctype html>

<html lang="en">
<script>
                  window.sessConf = "-2912474957111138742";
                  /* <sl:translate_json> */
                  window.__INITIAL_STATE__ = { /* Target JSON here with 12 million characters */};
                  /* </sl:translate_json> */
                </script>
</html>

Upvotes: 4

Views: 4258

Answers (2)

MUHAMMED OKUYUCU
MUHAMMED OKUYUCU

Reputation: 41

gdlmx's code is correct and very helpfull.

from subprocess import check_output
soup = BeautifulSoup(html)
s=soup.find('script')
js = 'window = {};\n'+s.text.strip()+';\nprocess.stdout.write(JSON.stringify(window.__INITIAL_STATE__));'
window_init_state = check_output(['node','temp.js'])

type(window_init_state) will be . So then you shuld use following code.

jsonData= window_init_state.decode("utf-8")

Upvotes: 0

gdlmx
gdlmx

Reputation: 6789

You can use the following Python code to extract the JavaScript code.

soup = BeautifulSoup(html)
s=soup.find('script')
js = 'window = {};\n'+s.text.strip()+';\nprocess.stdout.write(JSON.stringify(window.__INITIAL_STATE__));'
with open('temp.js','w') as f:
    f.write(js)

The JS code will be written to a file "temp.js". Then you can call node to execute the JS file.

from subprocess import check_output
window_init_state = check_output(['node','temp.js'])

The python variable window_init_state contains the JSON string of the JS object window.__INITIAL_STATE__, which you can parse in python with JSONDecoder.

Example

from subprocess import check_output
import json, bs4
html='''<!DOCTYPE doctype html>

<html lang="en">
<script> window.sessConf = "-2912474957111138742";
                  /* <sl:translate_json> */
                  window.__INITIAL_STATE__ = { 'Hello':'World'};
                  /* </sl:translate_json> */
                </script>
</html>'''
soup = bs4.BeautifulSoup(html)
with open('temp.js','w') as f:
    f.write('window = {};\n'+
            soup.find('script').text.strip()+
            ';\nprocess.stdout.write(JSON.stringify(window.__INITIAL_STATE__));')
window_init_state = check_output(['node','temp.js'])
print(json.loads(window_init_state))

Output:

{'Hello': 'World'}

Upvotes: 5

Related Questions