CODEWITHSUNDEEP
rubyselenium-webdriverweb-scrapingmechanizewatir-webdriver
Ethan Barron
Ethan Barron

Reputation: 1304

Cookies driving me crazy, or am I crazy?

Working on a project for a client, and an aspect of this job is to utilize sessions for individual users on a series of separate websites.

Unfortunately, all of the sudden, I'm running into innumerable issues with this approach. Mechanize doesn't like cookies (obviously, because there is JS involved).

The next choice would be to utilize something like Watir or Capybara to work around this problem. This is what I originally had done, but my client was not satisfied with the speed.

So, questions:

  1. Is there a good way to get this done w/o using a browser-driver and without deeply analyzing the JS of each website, reverse-engineering their cookie setting process, and re-implementing that in our codebase? (my guess: absolutely not)
  2. If not, what would be the best way around this problem? (my guess: browser-driver, like Watir or Capybara)
  3. If my guesses to questions #1 and #2 sound correct to you, then how can I convince this client that I know what I'm saying?

It's quite irritating to be hired to develop something on the magnitude of difficulty that I was hired to develop, and then to have your recommendations and insight be ignored. How can I better handle this problem?

This probably isn't a great question for Stack Overflow, and I apologize for that. I find myself coming here for help quite a bit. You guys usually have good answers. Thank you in advance for answering, if you do.

EDIT: To be a bit clearer, the issue is one of speed vs. reliability. He wants the utmost speed. Obviously, a browser-driver is not going to be the best speed, but it does sort of "guarantee", in a way, that you can "persist" a given session. Mechanize is much speedier, but requires much more finnicky diddling about to get things working correctly. Considering we're working with dozens of websites, I'm thinking the best route is to sacrifice speed and gain reliability/accuracy. What do you all think?

Ultimately, I'm looking for your help, because I'm at a loss for any more good arguments. I had plenty, but have exhausted them all, and he seems convinced that there is a way to do this without sacrificing either speed or accuracy (at least with the amount of human resources invested [a.k.a me]). I've tried explaining this, that we can have one or the other. Basically, as far as I know, we can only have one. The one we should choose is accuracy. How might I be able to argue this in a way that this person might listen more acutely?

Upvotes: 1

Views: 105

Answers (1)

pguardiario
pguardiario

Reputation: 54984

Well you can always login with Watir (I'm assuming that's where you can't solve the cookies issue) and then load the browser cookies into Mechanize. I know there's some sample code in other mechanize questions for this.

BTW, switching to Watir doesn't gain you reliability, it just makes it easier to solve your cookies problem. In my experience, Mechanize is generally more reliable.

Upvotes: 1

Related Questions