CODEWITHSUNDEEP
rweb-scrapinghttp-redirectespn
chris
chris

Reputation: 101

Using R as my browser how can I log into http://games.espn.go.com/ffl/signin and scrape my FFL Team HTML tables?

I have been trying everything I can find online to log in and set cookies and certificates.... can't seem to get past the redirect to a login screen.

Here is what I am trying to do:

##################################################
library("RCurl")
library("XML")

loginURL <- "http://games.espn.go.com/ffl/signin"
dataURL <- "http://games.espn.go.com/ffl/clubhouse?leagueId=123456&teamId=8&seasonId=2014"


# ESPN Fantasy Football Login Screen
userID <- dQuote("myUsername")
pword <-dQuote("myPassword")
pushbutton <- dQuote("OK")

# concatenate the url and log in options
FFLsigninURL <- paste(loginURL ,
    "&username=",userID,
    "&password=",pword,
    "&submit=",pushbutton)

page <- getURL(loginURL , verbose = TRUE)

and this seems to be leading me to a redirect for logging in - so Problem 1 - login not working

Part 2- one logged in - How can I proceed to the dataURL to scrape the tables? I tried login parameters on the data page as well but still get redirected to a login screen.

I'm sure I am missing something simple - just not seeing it...

Upvotes: 2

Views: 1013

Answers (1)

jdharrison
jdharrison

Reputation: 30455

It should be possible to follow location etc using RCurl alternatively you could use selenium and drive a browser:

library(RSelenium)
loginURL <- "http://games.espn.go.com/ffl/signin"
user <- 'myPass'
pass <- 'myUser'
RSelenium::checkForServer()
RSelenium::startServer()
remDr <- remoteDriver()
remDr$open()
remDr$navigate(loginURL)
webElem <- remDr$findElement('name', 'username')
webElem$sendKeysToElement(list(user))
webElem <- remDr$findElement('name', 'password')
webElem$sendKeysToElement(list(pass))
remDr$findElement('name', 'submit')$clickElement()
dataURL <- "http://games.espn.go.com/ffl/clubhouse?leagueId=123456&teamId=8&seasonId=2014"
remDr$navigate(dataURL)
# YOU can get the page source for example 
pageSrc <- remDr$getPageSource()[[1]]
# now operate on pageSrc using for example library(XML) etc
# readHTMLTable(pageSrc) # for example
remDr$close()
remDr$closeServer()

Upvotes: 1

Related Questions