CODEWITHSUNDEEP
rformspost
abhisarihan
abhisarihan

Reputation: 261

Using RCurl to post form

I am using RCurl for the first time to enter data into a form on the website. I need to be able to select a particular option from a drop down list and submit the form. The tricky part is the website is directed from another page before that. To be more specific, below is the url of the page I am trying to post the form in:

http://energyalmanac.ca.gov/petroleum/fuels_watch/page2.php

As you can see there are no options available in the drop down list as this page is directed from another page before this which populates the options. By manually inputting the data in the previous page and inspecting the values populated, I have figured out that the values in the drop down list increment by 1 every week. Currently, for example, that value is 1091. This value needs to be assigned to the input field 'PkID'. However, despite looking at other StackOverflow questions and reading up on the postForm function, I am unable to populate the form and get to the next page.

My question is am I missing something in my code that is not posting the value correctly? Or is the code not working because entering that url, there are no available options in the drop down list.

Below is the code that I have. I apologize if my explanation isn't clear.

postForm("http://energyalmanac.ca.gov/petroleum/fuels_watch/page2.php",
    'PkID' = '1091', style = "post")

Any help is greatly appreciated!

Upvotes: 1

Views: 1432

Answers (1)

Jake Burkhead
Jake Burkhead

Reputation: 6545

You have to post the Month and Year parameters to http://energyalmanac.ca.gov/petroleum/fuels_watch/page2.php so it knows which PkIDs to fill in. You can then grab those and post them to http://energyalmanac.ca.gov/petroleum/fuels_watch/output.php to get to the page with the actual data.

 > library(RCurl)
 > library(XML)
 > form <- postForm("http://energyalmanac.ca.gov/petroleum/fuels_watch/page2.php", Year = 1995, Month = 2)                                                                                                                                   
 > doc <- htmlParse(form)                                                                                                                                                                                                                    
 >                                                                                                                                                                                                                                           
 > pkids <- xpathSApply(doc, "//select[@name = 'PkID']/option", xmlAttrs)                                                                                                                                                                    
 > pkids                                                                                                                                                                                                                                     
 value value value value
 "162" "163" "164" "165"
 >                                                                                                                                                                                                                                           
 > data <- lapply(pkids, function(x) {                                                                                                                                                                                                       
    form <- postForm(uri = "http://energyalmanac.ca.gov/petroleum/fuels_watch/output.php", PkID = x)                                                                                                                                         
    htmlParse(form)                                                                                                                                                                                                                          
    })                                                                                                                                                                                                                                       
 >                                                                                                                                                                                                                                           
 > tab <- readHTMLTable(data[[1]], which = 1)                                                                                                                                                                                                
 R> head(tab)                                                                                                                                                                                                                                 
                                   02/03/1995 01/27/1995 Percent\n        Change
 1                  Refinery Input       <NA>       <NA>                    <NA>
 2                       Crude Oil      10965      11271                   -2.7%
 3                                       <NA>       <NA>                    <NA>
 4             Refinery Production       <NA>       <NA>                    <NA>
 5                 Motor Gasoline:       <NA>       <NA>                    <NA>
 6 CARB RFG (incl. Non-Oxygenated)       3049       2482                   22.8% ...

Upvotes: 2

Related Questions