CODEWITHSUNDEEP
go
Jask
Jask

Reputation: 680

Urls without scheme throw error

I write a proxy and the problem is that some links on websites don't have scheme for example google:

<a class="ab_dropdownlnk" href="//www.google.com/support/websearch/?source=g&amp;hl=en">

I fetch the url via Client.Do()
How to resolve such urls in Go?

Upvotes: 2

Views: 2166

Answers (2)

peterSO
peterSO

Reputation: 166885

If there is no scheme then use a sensible default. For example,

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "net/url"
)

func main() {
    href := "//www.google.com/support/websearch/?source=g&amp;hl=en"
    url, err := url.Parse(href)
    if err != nil {
        log.Fatal(err)
    }
    if url.Scheme == "" {
        url.Scheme = "http"
    }
    req, err := http.NewRequest("GET", url.String(), nil)
    if err != nil {
        log.Fatal(err)
    }
    client := http.Client{}
    res, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    websearch, err := ioutil.ReadAll(res.Body)
    res.Body.Close()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s\n", websearch)
}

Upvotes: 3

nemo
nemo

Reputation: 57757

The missing scheme lets the browser choose the protocol and is handy for sites which offer both http and https. The browser then chooses which protocol to use depending on how he got to the page. You can use https or http as a default or act like a browser and choose the protocol you used to fetch the page.

For example, something like this:

for _, parsedLink := range parsedLinks {
    parsedLink.Scheme = requestUrl.Scheme
}

Upvotes: 2

Related Questions