CODEWITHSUNDEEP
gohtmlgo-http
J Mead
J Mead

Reputation: 73

Why can't Golang download certain webpages?

I want to download Fantasy Football Data to analyse in Go, but when I try to download from this api page then I get back an empty response, even though the code works for other websites, e.g. this api page

Minimal reproduction, outputs an empty array.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

const AllPlayerData = "https://fantasy.premierleague.com/drf/bootstrap-static"

func main() {
    downloadAllData()
}

func downloadAllData() {
    client := &http.Client{
        Timeout: 20 * time.Second,
    }

    response, err := client.Get(AllPlayerData)
    if err != nil {
        fmt.Println("Unable to download player data.")
        return
    }

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Failed to read response")
        return
    }

    defer response.Body.Close()

    fmt.Println(body)
}

The same webpage downloads fine in Python:

import requests
url = "https://fantasy.premierleague.com/drf/bootstrap-static"
r = requests.get(url)
print(r.content)

I don't think it's related to e.g. Ajax calls as viewing the network requests in Chrome doesn't show up any beyond the page load itself

Upvotes: 2

Views: 127

Answers (1)

dave
dave

Reputation: 64687

They are doing some sort of validation on the user agent, the following code works:

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

const AllPlayerData = "https://fantasy.premierleague.com/drf/bootstrap-static"

func main() {
    downloadAllData()
}

func downloadAllData() {
    client := &http.Client{
        Timeout: 20 * time.Second,
    }

    request, err := http.NewRequest(http.MethodGet, AllPlayerData, nil)
    if err != nil {
        fmt.Println("Unable to create request.")
        return
    }
    request.Header.Add("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36")
    response, err := client.Do(request)
    if err != nil {
        fmt.Println("Unable to download player data.")
        return
    }

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Failed to read response")
        return
    }

    defer response.Body.Close()

    fmt.Println(string(body))
}

Upvotes: 6

Related Questions