CODEWITHSUNDEEP
javaperformanceseleniumselenium-webdriverbrowsermob
Tom
Tom

Reputation: 1385

Progammatic browsing performance issues

I'm using Selenium (in Java) to do programmatic browsing (Firefox) and, to speed up page loading, I'm going through a programmatic Java proxy running in the same application (Browsermob) to prevent the loading of external content (ads, etc) as I only need website's hosted content.

The application is browsing through hundreds of thousands of pages but unfortunately, after a few hours, speed is reducing significantly (from ~5s/page to ~30s/page).

Can it be related to browser cache size? Proxy cache? How to check? Any pointer or hint would be more than welcome!

Many thanks, Tom

Upvotes: 2

Views: 111

Answers (1)

Lee Savory
Lee Savory

Reputation: 19

Hi I would change your User Agent to something like Chrome to check wether or not Firefox is the problem or if its down to memory management in your app.

Are you closing down the Browser after a number of views? remember the browser will cache and start to take up a lot of memory. It may be better for you to use a text based browser (Lynx) in your app that removes a lot of overheads if your are simply going for text content.

Id kill the process every so often as memory leaks will cause the process to slow over time.

Upvotes: 2

Related Questions