Get request returns data in Thunder client/Postman but gives blank data in Golang code

Question

I am trying to fetch data from an API using Golang net/http. When I am using Thunder client from VS Code or even Postman, I am getting proper data but when I am trying to fetch the data from Golang code, I get an empty response.

The data is fetched in 2 steps:

1. Fetch cookies using an initial GET req (this part is working fine in both)
2. Use the cookies to make another GET req for fetching the required data. (This is the step which is giving blank response in Golang and is named Historical Data in the Postman link given below)

Here's the Golang code. The code might be a little long but just because of multiple lined of adding headers.

var BaseURL string = "https://www.nseindia.com"

func ReqConfig() *http.Request {
    req, _ := http.NewRequest("GET", BaseURL, nil)
    req.Header.Add("Accept", "*/*")
    req.Header.Add("Accept-Encoding", "gzip, deflate, br")
    req.Header.Add("Accept-Language", "en-GB,en-US;q=0.9,en;q=0.8")
    req.Header.Add("Connection", "keep-alive")
    req.Header.Add("Host", "www.nseindia.com")
    req.Header.Add("Referer", "https://www.nseindia.com/get-quotes/equity")
    req.Header.Add("X-Requested-With", "XMLHttpRequest")
    req.Header.Add("sec-fetch-dest", "empty")
    req.Header.Add("sec-fetch-mode", "cors")
    req.Header.Add("pragma", "no-cache")
    req.Header.Add("sec-fetch-site", "same-origin")
    req.Header.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36")
    fmt.Println(1, req.Header.Get("Cookie"))

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()
    for _, cookie := range res.Cookies() {
        req.AddCookie(cookie)
    }

    // TODO: Remove the need to call this API twice. This is just a temporary fix.
    res, err = http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()
    for _, cookie := range res.Cookies() {
        req.AddCookie(cookie)
    }


    cookies := req.Cookies()
    for i := 0; i < len(cookies); i++ {
        for j := i + 1; j < len(cookies); j++ {
            if cookies[i].Name == cookies[j].Name {
                cookies = append(cookies[:j], cookies[j+1:]...)
                j--
            }
        }
    }
    req.Header.Del("Cookie")
    for _, cookie := range cookies {
        req.AddCookie(cookie)
    }
    fmt.Println("Fetched cookies")

    return req
}


func HistoricalEQ(symbol string, from string, to string, series string) {
    req := ReqConfig()

    query := req.URL.Query()
    query.Add("symbol", symbol)
    query.Add("from", from)
    query.Add("to", to)
    query.Add("series", "[\""+series+"\"]")
    req.URL.RawQuery = query.Encode()
    req.URL.Path = "/api/historical/cm/equity"

    client := &http.Client{Timeout: 40 * time.Second}
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()

    var data map[string]interface{}
    json.NewDecoder(res.Body).Decode(&data)

        // Prints `map[]` and not the whole json data which is provided in Postman req
    fmt.Println(data)
}


func main() {
    symbol := "PAYTM"
    series := "EQ"
    from_date := time.Date(2023, 1, 1, 0, 0, 0, 0, time.Local).Format("02-01-2006")
    to_date := time.Date(2023, 1, 24, 0, 0, 0, 0, time.Local).Format("02-01-2006")
    HistoricalEQ(symbol, from_date, to_date, series)
}

If you are able to fetch data from some other way in Golang only from GET https://www.nseindia.com/api/historical/cm/equity?symbol=PAYTM&series=[%22EQ%22]&from=28-12-2022&to=28-01-2023, then that would also solve my issue. You can check out the website frontend at https://www.nseindia.com/get-quotes/equity?symbol=PAYTM. The GET req I am asking can be triggered by going to Historical data tab and clicking on filter button

Similar code in python: https://github.com/jugaad-py/jugaad-data/blob/47bbf1aa39ebec3a260579c76ff427ea06e42acd/jugaad_data/nse/history.py#L61

Answer 1

1️⃣ decoding error handling was missed

err := json.NewDecoder(res.Body).Decode(&data)
if err != nil {
    log.Fatalf("decode request: %v", err)
}

invalid character '\x1f' looking for beginning of value

2️⃣ looks like the response data has been compressed (gzip data start with the magic sequence 0x1f 0x8b). If you check response's Headers you see

...
Content-Encoding:[gzip] 👈🏻👈🏻👈🏻
Content-Length:[1890] 
Content-Type:[application/json; charset=utf-8] 
...

that it looks like the truth

3️⃣ try to manually handling compression (compress/gzip)

    client := &http.Client{Timeout: 40 * time.Second}
    res, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(res.Header)

    var reader io.ReadCloser
    switch res.Header.Get("Content-Encoding") {
    case "gzip":
        reader, err = gzip.NewReader(res.Body)
    default:
        reader = res.Body
    }
    defer reader.Close()

    var data map[string]interface{}
    err = json.NewDecoder(reader).Decode(&data)
    if err != nil {
        log.Fatalf("decode request: %v", err)
    }

    fmt.Println(data) 👈🏻 // map[data:[map[CH_52WEEK_HIGH_PRICE:994 CH_52WEEK_LOW_PRICE:438.35 CH_CLOSING_PRICE:543.55 CH_ISIN:INE982J01020 CH_LAST_TRADED_PRICE:542.2 CH_MARKET_TYPE:N ...