Using CefSharp to capture Resource Response Data (body)

Question

I am trying to use CefSharp to visit a URL and capture specific resources retrieved during the loading of a given page. Presumably as a Stream or byte array per resource.

CefSharp provides the interface IRequestHandler. You can create a class which implements this interface to respond to Request/Response events, but it does not contain the response body in any way.

Answer 1

The code that works now (Version 126.2.180) is what amaitland pointed out

Create the following two classes:

public class CustomResourceRequestHandler : CefSharp.Handler.ResourceRequestHandler
{
    private readonly System.IO.MemoryStream memoryStream = new System.IO.MemoryStream();

    protected override IResponseFilter GetResourceResponseFilter(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, IResponse response)
    {
        return new CefSharp.ResponseFilter.StreamResponseFilter(memoryStream);
    }

    protected override void OnResourceLoadComplete(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, IResponse response, UrlRequestStatus status, long receivedContentLength)
    {
        //You can now get the data from the stream
        var bytes = memoryStream.ToArray();

        if (response.Charset == "utf-8")
        {
            var str = System.Text.Encoding.UTF8.GetString(bytes);
        }
        else
        {
            //Deal with different encoding here
        }
    }
}

public class CustomRequestHandler : CefSharp.Handler.RequestHandler
{
    protected override IResourceRequestHandler GetResourceRequestHandler(IWebBrowser chromiumWebBrowser, IBrowser browser, IFrame frame, IRequest request, bool isNavigation, bool isDownload, string requestInitiator, ref bool disableDefaultHandling)
    {
        //Only intercept specific Url's
        if (request.Url == "http://cefsharp.github.io/" || request.Url == "https://cefsharp.github.io/")
        {
            return new CustomResourceRequestHandler();
        }

        //Default behaviour, url will be loaded normally.
        return null;
    }
}

Then set the request handler to Custom RequestHandler class before Initialize Chromium:

browser.RequestHandler = new CustomRequestHandler();

https://github.com/cefsharp/CefSharp/wiki/General-Usage#response-filtering

Answer 2

Take a look at GetResourceResponseFilter in your implementation of IRequestHandler. GetResourceResponseFilter returns an IResponseFilter, which provides you with an opportunity to capture all responses. You'll need to implement your own IResponseFilter, thankfully there are ample examples.

Example IResponseFilter implementation taken from the CefSharp GitHub project.

public class MemoryStreamResponseFilter : IResponseFilter
{
        private MemoryStream memoryStream;

        bool IResponseFilter.InitFilter()
        {
            //NOTE: We could initialize this earlier, just one possible use of InitFilter
            memoryStream = new MemoryStream();
             
            return true;
        }

        FilterStatus IResponseFilter.Filter(Stream dataIn, out long dataInRead, Stream dataOut, out long dataOutWritten)
        {
            if (dataIn == null)
            {
                dataInRead = 0;
                dataOutWritten = 0;

                return FilterStatus.Done;
            }

            dataInRead = dataIn.Length;
            dataOutWritten = Math.Min(dataInRead, dataOut.Length);

            //Important we copy dataIn to dataOut
            dataIn.CopyTo(dataOut);

            //Copy data to stream
            dataIn.Position = 0;
            dataIn.CopyTo(memoryStream);

            return FilterStatus.Done;
        }

        void IDisposable.Dispose()
        {
            memoryStream.Dispose();
            memoryStream = null;
        }

        public byte[] Data
        {
            get { return memoryStream.ToArray(); }
        }
    }

Now in your GetResourceResponseFilter method:

 private Dictionary<ulong, MemoryStreamResponseFilter> responseDictionary = new Dictionary<ulong, MemoryStreamResponseFilter>();
 public IResponseFilter GetResourceResponseFilter(IWebBrowser browserControl, IBrowser browser, IFrame frame, IRequest request, IResponse response)
 {
       var dataFilter = new MemoryStreamResponseFilter();
       responseDictionary.Add(request.Identifier, dataFilter);
       return dataFilter;
 }

Then in OnResourceLoadComplete:

public void OnResourceLoadComplete(IWebBrowser browserControl, IBrowser browser, IFrame frame, IRequest request, IResponse response, UrlRequestStatus status, long receivedContentLength)
{      
    MemoryStreamResponseFilter filter;
    if (responseDictionary.TryGetValue(request.Identifier, out filter))
    {
        var data = filter.Data; //This returns a byte[]
        //File.WriteAllBytes("c:/save/path", data);
    }
}

You can use the properties within the IRequest and IResponse parameters to decide what you want to filter.