CODEWITHSUNDEEP
javascriptpythonweb-scrapingscrapyscreen-scraping
Nandril
Nandril

Reputation: 107

Web Scraping: Next page is rendered in Javascript, how can I get it using Scrapy

I have been trying to scrape this website using Scrapy https://www.remax.com/homes-for-sale/ny/new-york/city/3651000. I am able to get the contents on the page but i can't go to the next page since it seems to be rendered in javascript. How can do this?

Upvotes: 0

Views: 483

Answers (1)

sarartur
sarartur

Reputation: 1228

I am not sure how to achieve what you are doing with scrapy, but it looks like the javascript is pulling all the results from a backend API. You can find the backend URL and the details of the AJAX request through browser development tools. It all looks like the code below, give it a try. You might be able to extract the info you are looking for directly from their API.

import requests

payload = {
    "count":24,
    "offset":0,
    "sorts":{
        "0":{
            "listingContractDate":"desc"
        }
    },
    "terms":{
        "place":{
            "lat":40.70668199998021,
            "lon":-73.97795499996471,
            "city":"New York",
            "state":"NY",
            "placename":"New York, NY",
            "placeType":"city",
            "placeId":"3651000",
            "areaSquareMiles":308.12
        },"locationRect":{
            "minLat":40.3400891972592,
            "maxLat":41.07193158068027,
            "minLon":-74.24986662109377,
            "maxLon":-73.70604337890627
        },
        "bPropertyType":[
            "Single Family",
            "Condo/Townhome",
            "Mobile Home",
            "Multi-Family",
            "Rental","Farm",
            "Land"],
        "bStatus":[
            "For Sale",
            "Under Contract"
        ],
        "city":[
            "New York"
        ],
        "State":[
            "NY"
        ]
    },
    "listingLoadLevel":"Search"
}


r = requests.post("https://public-api-gateway-prod.kube.remax.booj.io/listings/search/run/", json=payload)
print(r.json())

Upvotes: 1

Related Questions