CODEWITHSUNDEEP
go
Bruno
Bruno

Reputation: 709

Selectively Follow Redirects in Go

I'm trying to write a twitter reader that resolves the final URLs of link shorteners etc, but gives me a URL along the way for a list of manually defined host patterns. The reason to do this is that i don't want to end up with the paywall URL but the one before.

As far as i can tell the way to do this is write my own client based on the default RoundTripper because returning an error from a custom CheckRedirect function aborts the client without yielding a response.

Is there a way to use the default client and record a list of URLs/specific URL from a custom checkRedirect function?

Upvotes: 2

Views: 3182

Answers (1)

Flo Lauber
Flo Lauber

Reputation: 46

The client request will actually still return the last valid Response in cases where your custom CheckResponse yields an error (As mentioned in the comments).

http://golang.org/pkg/net/http/#Client

If CheckRedirect returns an error, the Client's Get method returns both the previous Response and CheckRedirect's error (wrapped in a url.Error) instead of issuing the Request req.

If you maintain a list of "known" paywall-urls, you can abort the paywall-redirect in your CheckResponse with a custom error type (Paywalled in the example below). Your error handling code later has to consider that error type as a special (non-erroneous) case.

Example:

package main

import (
    "errors"
    "fmt"
    "net/http"
    "net/url"
)

var Paywalled = errors.New("next redirect would hit a paywall")

var badHosts = map[string]error{
    "registration.ft.com": Paywalled,
}

var client = &http.Client{
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
        // N.B.: when used in production, also check for redirect loops
        return badHosts[req.URL.Host]
    },
}

func main() {
    resp, err := client.Get("http://on.ft.com/14pQBYE")
    // ignore non-nil err if it's a `Paywalled` wrapped in url.Error
    if e, ok := err.(*url.Error); (ok && e.Err != Paywalled) || (!ok && err != nil) {
        fmt.Println("error: ", err)
        return
    }   
    resp.Body.Close()
    fmt.Println(resp.Request.URL)
}

Upvotes: 3

Related Questions