Mechanize not Submitting Form

Question

I'm trying to login to a site via mechanize, and I am filling out all of the forms, and then submitting, but each time I try it, I end up on the same page, so either I'm being redirected back (with no error message), or the form is not submitting. Why is that?

The code

require 'mechanize'
class Scraper

  def initialize
    @a = Mechanize.new { |agent|
      agent.follow_meta_refresh = true
    }
  end

  def login


    @a.get("https://login.salesforce.com/") do |page|
      red = page.forms.first do |form|
        form.field_with(:type => "email").value = username
        form.field_with(:type => "password").value  =  password
      end.click_button
      pp red
      #puts main.title
    end
  end
end
s = Scraper.new
s.login

The initial page:

   #<Mechanize::Page
   {url #<URI::HTTPS:0x007f8f39d6fb30 URL:https://login.salesforce.com/>}
   {meta_refresh}
   {title "salesforce.com - Customer Secure Login Page"}
   {iframes
    #<Mechanize::Page::Frame
     "marketing"
     "https://www.salesforce.com/login-messages/messages.html?noroundedcorner">}
   {frames}
   {links
    #<Mechanize::Page::Link "Salesforce" "http://www.salesforce.com">
    #<Mechanize::Page::Link
     "Forgot your password?"
     "/secur/forgotpassword.jsp?locale=us">
    #<Mechanize::Page::Link
     "Sign up for free."
     "https://www.salesforce.com/form/trial/freetrial.jsp?d=70130000000Enus">}
   {forms
    #<Mechanize::Form
     {name "login"}
     {method "POST"}
     {action "https://login.salesforce.com/"}
     {fields
      [hidden:0x3fc79cec63ac type: hidden name: un value: ]
      [hidden:0x3fc79cec6244 type: hidden name: width value: ]
      [hidden:0x3fc79cec60c8 type: hidden name: height value: ]
      [hidden:0x3fc79cec5efc type: hidden name: hasRememberUn value: true]
      [hidden:0x3fc79cec5d58 type: hidden name: startURL value: ]
      [hidden:0x3fc79cec5bc8 type: hidden name: loginURL value: ]
      [hidden:0x3fc79cec5a38 type: hidden name: loginType value: ]
      [hidden:0x3fc79cec987c type: hidden name: useSecure value: true]
      [hidden:0x3fc79cec969c type: hidden name: local value: ]
      [hidden:0x3fc79cec9520 type: hidden name: lt value: standard]
      [hidden:0x3fc79cec9340 type: hidden name: qs value: ]
      [hidden:0x3fc79cec9174 type: hidden name: locale value: ]
      [hidden:0x3fc79cec8f80 type: hidden name: oauth_token value: ]
      [hidden:0x3fc79cec8db4 type: hidden name: oauth_callback value: ]
      [hidden:0x3fc79cec8be8 type: hidden name: login value: ]
      [hidden:0x3fc79cec89cc type: hidden name: serverid value: ]
      [hidden:0x3fc79cec8814 type: hidden name: display value: page]
      [field:0x3fc79cec8670 type: email name: username value: ]
      [field:0x3fc79cec84e0 type: password name: pw value: ]}
     {radiobuttons}
     {checkboxes
      [checkbox:0x3fc79cec833c type: checkbox name: rememberUn value: ]}
     {file_uploads}
     {buttons [button:0x3fc79cecac2c type:  name: Login value: ]}>}>

The final page:

#<Mechanize::Page
{url #<URI::HTTPS:0x007f9d1d250960 URL:https://login.salesforce.com/>}


{meta_refresh}
   {title "salesforce.com - Customer Secure Login Page"}
   {iframes
    #<Mechanize::Page::Frame
     "marketing"
     "https://www.salesforce.com/login-messages/messages.html?    r=https%3A%2F%2Flogin.salesforce.com%2F&noroundedcorner">}
   {frames}
   {links
    #<Mechanize::Page::Link "Salesforce" "http://www.salesforce.com">
    #<Mechanize::Page::Link
     "Forgot your password?"
     "/secur/forgotpassword.jsp?locale=us">
    #<Mechanize::Page::Link
     "Sign up for free."
     "https://www.salesforce.com/form/trial/freetrial.jsp?d=70130000000Enus">}
   {forms
    #<Mechanize::Form
     {name "login"}
     {method "POST"}
     {action "https://login.salesforce.com/"}
     {fields
      [hidden:0x3fce8e93aaac type: hidden name: un value: ]
      [hidden:0x3fce8e93a8a4 type: hidden name: width value: ]
      [hidden:0x3fce8e93a638 type: hidden name: height value: ]
      [hidden:0x3fce8e93a390 type: hidden name: hasRememberUn value: true]
      [hidden:0x3fce8e93a19c type: hidden name: startURL value: null]
      [hidden:0x3fce8e939f58 type: hidden name: loginURL value: null]
      [hidden:0x3fce8e939cc4 type: hidden name: loginType value: ]
      [hidden:0x3fce8e9399a4 type: hidden name: useSecure value: true]
      [hidden:0x3fce8e93979c type: hidden name: local value: ]
      [hidden:0x3fce8e939648 type: hidden name: lt value: standard]
      [hidden:0x3fce8e93d414 type: hidden name: qs value:r=https%3A%2F%2Flogin.salesforce.com%2F]
      [hidden:0x3fce8e93d284 type: hidden name: locale value: ]
      [hidden:0x3fce8e93d0cc type: hidden name: oauth_token value: ]
      [hidden:0x3fce8e93cf50 type: hidden name: oauth_callback value: ]
      [hidden:0x3fce8e93cd98 type: hidden name: login value: ]
      [hidden:0x3fce8e93cc44 type: hidden name: serverid value: ]
      [hidden:0x3fce8e93cab4 type: hidden name: display value: page]
      [field:0x3fce8e93c780 type: email name: username value: ]
      [field:0x3fce8e93c4c4 type: password name: pw value: ]}
     {radiobuttons}
     {checkboxes
      [checkbox:0x3fce8e93c334 type: checkbox name: rememberUn value: ]}
     {file_uploads}
     {buttons [button:0x3fce8e93b9d4 type:  name: Login value: ]}>}>

What is wrong with my code?

Answer 1

On the form there are hidden fields un, width, height that also need to contain the username and some numbers, in addition to the username field containing the username.

  [hidden:0x3fce8e93aaac type: hidden name: un value: ]
  [hidden:0x3fce8e93a8a4 type: hidden name: width value: ]
  [hidden:0x3fce8e93a638 type: hidden name: height value: ]

You can use Chrome's Inspector under the Network tab to monitor the contents of the request that actually gets sent to the server (enable the "preserve log" option), after it gets modified by javascript.

Answer 2

The website uses Javascript to process the login, which mechanize cannot handle. You could use something like Selenium to access the site.