CODEWITHSUNDEEP
splash-screenscrapy-splashsplash-js-render
Maciek
Maciek

Reputation: 1972

Splash won't render all contents of page

I am using Splash v2.3.2 and I am trying to render a page but it is not rendering everything. It won't render images or dynamically loaded content.

I am using my http://localhost:8050/ with script:

function main(splash)
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait(10))
  return {
    html = splash:html(),
    png = splash:png(),
    har = splash:har(),
  }
end

Here is a browser rendering: browser rendering

Here is a screenshot of the Splash rendering: Splash rendering

I have tried to change the wait time and also tried to allow plugins. None of this will work. I am assuming that the dynamically loaded content is being restricted but I am unsure. Any help is appreciated.

Upvotes: 2

Views: 3002

Answers (2)

Mikhail Korobov
Mikhail Korobov

Reputation: 22238

The problem is with localStorage - site uses it, but Splash uses Private Mode by default, and this disabls localStorage. To fix it, disable private mode (see here). This script works for me (Splash 3.0):

function main(splash)
  splash.private_mode_enabled = false
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait(10))
  return {
    html = splash:html(),
    png = splash:png(),
    har = splash:har(),
  }
end

See also: http://splash.readthedocs.io/en/stable/faq.html#website-is-not-rendered-correctly

Upvotes: 6

Eswar
Eswar

Reputation: 309

I'm assuming that you are trying to scrape property description text. In your code you have just added splash:wait(10), My suggestion is you should try and implement wait for a specific css element. In you case, span#listingpropertydescription. You can write a function to wait for this particular element and then return the html page.

Note you can find a sample wait-for-element code in http://localhost:8050/

Hope this will help you

Upvotes: 0

Related Questions