CODEWITHSUNDEEP
google-chromegonetwork-programminggoogle-chrome-devtools
carusyte
carusyte

Reputation: 1769

How to intercept network traffic and get the response using chromedp (or alternatives)?

I'm aiming to get a specific response from the target server which cannot be retrieve directly, but via techniques such as using web driver to fire the request initiated by inherent javascript code in the web page once loaded. The request contains some code for server-side verification which for now I'm unable to decode the generating algorithm.

The "Developer Tools" in Chrome provides a convenient way to inspect the request and response, and I need to automate the process by using powerful libs such as chromedp.

AFAIK the network package provides a GetResponseBody function, but requires a requestID parameter. How can I acquire the specific request id?

cdp.ActionFunc(func(ctxt context.Context, h cdptypes.Handler) error {
        rptn := &network.RequestPattern{
            ResourceType: page.ResourceTypeScript,
        }
        network.SetRequestInterception([]*network.RequestPattern{rptn}).Do(ctxt, h)

        //begin interception
        network.ContinueInterceptedRequest("AlphaInterceptor").Do(ctxt, h)

        //How to identify the requestID?
        network.GetResponseBody("???")

        ...
}

Upvotes: 1

Views: 4083

Answers (1)

carusyte
carusyte

Reputation: 1769

It appears that the chromedp lib wasn't complete at the time. I've implemented the event listening mechanism and submitted a pull request. For those in need, the specific server resources can be acquired by listening to network events and get the RequestID and response body respectively:

cdp.Tasks{
    cdp.ActionFunc(func(ctxt context.Context, h cdptypes.Handler) error {
        go func() {
            echan := h.Listen(cdptypes.EventNetworkRequestWillBeSent, cdptypes.EventNetworkLoadingFinished)
            for d := range echan {
                switch d.(type) {
                case *network.EventRequestWillBeSent:
                    req := d.(*network.EventRequestWillBeSent)
                    if strings.HasSuffix(req.Request.URL, "/data_I_want.js") {
                        reqId1 = req.RequestID
                    } else if strings.HasSuffix(req.Request.URL, "/another_one.js") {
                        reqId2 = req.RequestID
                    }
                case *network.EventLoadingFinished:
                    res := d.(*network.EventLoadingFinished)
                    var data []byte
                    var e error
                    if reqId1 == res.RequestID {
                        data, e = network.GetResponseBody(reqId1).Do(ctxt, h)
                    } else if reqId2 == res.RequestID {
                        data, e = network.GetResponseBody(reqId2).Do(ctxt, h)
                    }
                    if e != nil {
                        panic(e)
                    }
                    if len(data) > 0 {
                        fmt.Printf("=========data: %+v\n", string(data))
                    }
                }
            }
        }()
        return nil
    }),
    cdp.Navigate(url),
    ...
}

Upvotes: 4

Related Questions