CODEWITHSUNDEEP
rweb-crawlerrvest
hfisch
hfisch

Reputation: 1332

Submit form with no submit button in rvest

I'm trying write a crawler to download some information, similar to this Stack Overflow post. The answer is useful for creating the filled-in form, but I'm struggling to find a way to submit the form when a submit button is not part of the form. Here is an example:

session <- html_session("www.chase.com")
form <- html_form(session)[[3]]

filledform <- set_values(form, `user_name` = user_name, `usr_password` = usr_password)
session <- submit_form(session, filledform)

At this point, I receive this error:

Error in names(submits)[[1]] : subscript out of bounds

How can I make this form submit?

Upvotes: 8

Views: 3577

Answers (1)

Tripartio
Tripartio

Reputation: 2189

Here's a dirty hack that works for me: After studying the submit_form source code, I figured that I could work around the problem by injecting a fake submit button into my code version of the form, and then the submit_form function would call that. It works, except that it gives a warning that often lists an inappropriate input object (not in the example below, though). However, despite the warning, the code works for me:

session <- html_session("www.chase.com")
form <- html_form(session)[[3]]

# Form on home page has no submit button,
# so inject a fake submit button or else rvest cannot submit it.
# When I do this, rvest gives a warning "Submitting with '___'", where "___" is
# often an irrelevant field item.
# This warning might be an rvest (version 0.3.2) bug, but the code works.
fake_submit_button <- list(name = NULL,
                           type = "submit",
                           value = NULL,
                           checked = NULL,
                           disabled = NULL,
                           readonly = NULL,
                           required = FALSE)
attr(fake_submit_button, "class") <- "input"
form[["fields"]][["submit"]] <- fake_submit_button

user_name <- "user"
usr_password <- "password"

filledform <- set_values(form, `user_name` = user_name, `usr_password` = usr_password)
session <- submit_form(session, filledform)

The successful result displays the following warning, which I simply ignore:

> Submitting with 'submit'

Upvotes: 12

Related Questions