RAILS // Selenium/Watir in a Sidekiq job on Heroku

Question

I basically have a big memory bloat when I run a rake task to scrape data. I have been using it for years and it was fine but I've had to upgrade loads of librairies (as well as the stack of Heroku to 24 and moved to the buildpack Chrome for testing from the individual buildpacks Chrome and Chromedriver) and the memory now very quickly explodes.

In local I have used Memory Profiler to check the RAM used and it looks OK however when I run the task on Heroku it goes way over the limit quickly and the process gets killed.

In order to debug I have Scout APM (which doesn't look at rake tasks) and I tried to add New Relic but I didn't manage to set it up properly to dive deep into the rake task.

The workaround I've tried to do is to call my rake task through a Sidekiq Job and so that I could see more details of it in Scout in the Background processes. However I've run into another problem.

When I run my rake task that uses Selenium/Watir it runs on a one-off dyno and it runs without any problem (without specifying either chromedriver or chrome route - they should be in the PATH). However through Sidekiq I have the following issue:

Selenium::WebDriver::Error::WebDriverError (unable to connect to /app/.chrome-for-testing/chromedriver-linux64/chromedriver 127.0.0.1:9515)

The current version of chrome-for-testing is 129.x.x. I have tried specifying explicity the route to both Chrome and Chromedriver but to no avail:

service = Selenium::WebDriver::Service.chrome(path: '/app/.chrome-for-testing/chromedriver-linux64/chromedriver')
options = Selenium::WebDriver::Chrome::Options.new
options.binary = '/app/.chrome-for-testing/chrome-linux64/chrome'
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')

        browser = Watir::Browser.new :chrome, :options => options, :service => service

Any idea how to solve that ?