CODEWITHSUNDEEP
pythonbeautifulsouppython-requestslazy-loading
euh
euh

Reputation: 451

Web scraping lazy list (lazy loading) using python request (without selenium/scarpy)

I have written a simple script for myself as practice to find who had bought same tracks as me on bandcamp to ideally find accounts with similar tastes and so more same music on their accounts.

The problem is that fan list on a album/track page is lazy loading. Using python's requests and bs4 I am only getting 60 results out of potential 700.

I am trying to figure out how to send request i.e. here https://pitp.bandcamp.com/album/fragments-distancing to open more of the list. After finding what request is send when I click in finder, I used that json request to send it using requests, although without any result

res = requests.get(track_link)
    open_more = {"tralbum_type":"a","tralbum_id":3542956135,"token":"1:1609185066:1714678:0:1:0","count":100}
    for i in range(0,3):
        requests.post(track_link, json=open_more)

Will appreciate any help!

Upvotes: 1

Views: 648

Answers (1)

folen gateis
folen gateis

Reputation: 2012

i think that just typing a ridiculous number for count will do. i did some automation on your script too if you want to get data on other albums

from urllib.parse import urlsplit
import json

import requests
from bs4 import BeautifulSoup

# build the post link
get_link="https://pitp.bandcamp.com/album/fragments-distancing"
link=urlsplit(get_link)
base_link=f'{link.scheme}://{link.netloc}'
post_link=f"{base_link}/api/tralbumcollectors/2/thumbs"

with requests.session() as s:
    res = s.get(get_link)
    soup = BeautifulSoup(res.text, 'lxml')

    # the data for tralbum_type and tralbum_id
    # are stored in a script attribute
    key="data-band-follow-info"
    data=soup.select_one(f'script[{key}]')[key]
    data=json.loads(data)
    open_more = {
        "tralbum_type":data["tralbum_type"],
        "tralbum_id":data["tralbum_id"],
        "count":1000}
        
    r=s.post(post_link, json=open_more).json()
    print(r['more_available']) # if not false put a bigger count

Upvotes: 1

Related Questions