CODEWITHSUNDEEP
pythonjsonbeautifulsoup
Ervan
Ervan

Reputation: 91

Extract json content in <script using BeautifulSoup

I'm trying to extract a json part of the script using beautiful soup but it prints Nothing. What's wrong?

import requests,json
from bs4 import BeautifulSoup

headers = {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
        'referrer': 'https://google.com',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'en-US,en;q=0.9',
        'Pragma': 'no-cache',
    }
r = requests.get('https://www.joox.com/id/id/single/xtMtD9ZdeLEdHp1w1fip8w==',headers=headers)
soup = BeautifulSoup(r.content, 'html5lib')
data = json.loads(soup.find('script').text)
print(data)

The soup result is like:

<script>

      __NEXT_DATA__ = {"props":{"trackData":{"album_url":"https://imgcache.qq.com/music/photo/mid_album_300/O/F/002tkoBZ2zJPOF.jpg","code":0,"country":"hk","encodeSongId":"xtMtD9ZdeLEdHp1w1fip8w==","express_domain":"http://stream.music.joox.com/","flag":0,"gososo":0,"has_hifi":false,"has_hq":false,"imgSrc":"https://imgcache.qq.com/music/photo/mid_album_300/O/F/002tkoBZ2zJPOF.jpg","kbps_map":"{\"128\":4082449,\"192\":6522038,\"24\":825973,\"320\":10216761,\"48\":1708095,\"96\":3701266,\"ape\":0,\"flac\":0}\n","ktrack_id":0,"m4aUrl":"https://hk.stream.music.joox.com/C4000035Jew421IKj2.m4a?

Any ideas for this? thanks

Upvotes: 1

Views: 5390

Answers (2)

ewwink
ewwink

Reputation: 19154

even BeautifulSoup still need Regex, why not use it directly

import re

....
r = requests.get('https://www.joox.com/id/id/single/xtMtD9ZdeLEdHp1w1fip8w==',headers=headers)

jsData = re.search(r'__NEXT_DATA__\s+=\s+(.*)', r.text)
data = json.loads(jsData.group(1))
print(data)

Upvotes: 1

SanthoshSolomon
SanthoshSolomon

Reputation: 1402

You can use substring and regular expression for obtaining what you are seeking for. 

    import requests,json, re
    from bs4 import BeautifulSoup

    headers = {
            'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
            'referrer': 'https://google.com',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Pragma': 'no-cache',
        }
    r = requests.get('https://www.joox.com/id/id/single/xtMtD9ZdeLEdHp1w1fip8w==',headers=headers)
    soup = BeautifulSoup(r.content, 'html5lib')
    # finding the script element whihc contains the required data
    json_data = soup.find('script', text=re.compile("__NEXT_DATA__") )
    # identifying the required elements and obtaining it by sub string
    data = str(json_data)[str(json_data).find('__NEXT_DATA__ = '):str(json_data).find('module={}')]
    data = json.loads(data.replace('__NEXT_DATA__ = ', ''))
    print(data)

Using slicing technique we are identifying the string pattern and extracting the json string alone. After that you are good to go!

Hope this helps! Cheers!

Upvotes: 3

Related Questions