Java Selenium get JSON response body

Question

I using Java with Selenium webdriver and I wondering is it possible to get JSON body response? I asking because it is possible to get JSON request body using this code:

driver.manage().logs().get(LogType.PERFORMANCE);

but I can't get body from response but response exist also in this log. Is any way to get response body?

Answer 1

I was able to make it work for myself with the following code (using Selenium 4):

    File file = new File("C:\\Program Files (x86)\\ChromeDriver\\chromedriver.exe");
    ChromeDriverService service = new ChromeDriverService.Builder()
            .usingDriverExecutable(file)
            .usingAnyFreePort().build();
    ChromeOptions options = new ChromeOptions().addArguments("--incognito");
    ChromeDriver driver = new ChromeDriver(service, options);
    DevTools chromeDevTools = driver.getDevTools();
    
    chromeDevTools.createSession();
    chromeDevTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
    chromeDevTools.addListener(Network.loadingFinished(),
            entry -> {
                System.out.println("Body: " + chromeDevTools.send(Network.getResponseBody(entry.getRequestId())).getBody());
            });  

    driver.get("https://www.yoururlhere.com");
    
    chromeDevTools.send(Network.disable());
    driver.quit();

My imports:

    import org.openqa.selenium.chrome.ChromeDriver;
    import org.openqa.selenium.chrome.ChromeDriverService;
    import org.openqa.selenium.chrome.ChromeOptions;
    import org.openqa.selenium.devtools.DevTools;
    import org.openqa.selenium.devtools.v96.network.Network;

    import java.io.File;
    import java.util.*;

My Selenium dependency:

    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>4.1.0</version>
    </dependency>

Helpful sources, that helped me get to the solution:

https://applitools.com/blog/selenium-4-chrome-devtools/

https://chromedevtools.github.io/devtools-protocol/1-3/Network/

Answer 2

Edit-2

Because of a open issue with Geckodriver, https://github.com/mozilla/geckodriver/issues/764

So you need to use a workaround and do what GeckoDriver does in background for you

import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.CaptureType;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;

import java.io.File;
import java.io.IOException;

/**
 * Created by tarun.lalwani on 08/29/17.
 */
public class TestApp {


    public static void main(String [] args) {

        BrowserMobProxyServer proxyServer = new BrowserMobProxyServer();
        proxyServer.start();
        proxyServer.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
        proxyServer.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
        Proxy proxy = ClientUtil.createSeleniumProxy(proxyServer);
        FirefoxProfile profile = new FirefoxProfile();

        String host = proxy.getHttpProxy().split(":")[0];
        int port = Integer.parseInt(proxy.getHttpProxy().split(":")[1]);


        profile.setPreference("network.proxy.type", 1);
        profile.setPreference("network.proxy.http", host);
        profile.setPreference("network.proxy.http_port", port);
        profile.setPreference("network.proxy.ssl", host);
        profile.setPreference("network.proxy.ssl_port", port);
        FirefoxDriver driver = new FirefoxDriver(profile);

        proxyServer.newHar("mysite");

        driver.get("http://tarunlalwani.com");

        Har har = proxyServer.getHar();
        try {
            har.writeTo(new File("har.json"));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Edit-1

Since you want to do this in Java. You need to use BrowserMobProxy libs for Java and below kind of code should work for you

import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.CaptureType;


.....

BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);

Proxy selProxy = ClientUtil.createSeleniumProxy(proxy);

DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, selProxy);


WebDriver driver = new FirefoxDriver(capabilities);

proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);

proxy.newHar("mysite");

driver.get("http://tarunlalwani.com");


Har har = proxy.getHar();

Original Answer

You need setup a BrowserMobProxy on Firefox. Below code will work for you

import time
from selenium import webdriver

from browsermobproxy import Server
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

server = Server("/path/to/bin/browsermob-proxy")
server.start()
# If sleep is not added sometime `create_proxy` throws an error
time.sleep(2)
proxy = server.create_proxy()

sel_proxy = proxy.selenium_proxy()

profile = FirefoxProfile()
profile.set_proxy(sel_proxy)
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("mysite", options={'captureHeaders': True, 'captureContent': True} )
driver.get("http://tarunlalwani.com")
print(proxy.har)

This will give output like below

'version': '1.2',
'creator': {
  'name': 'BrowserMob Proxy',
  'version': '2.1.4',
  'comment': ''
},
'pages': [
  {
    'id': 'mysite',
    'startedDateTime': '2017-08-25T21:38:08.934+05:30',
    'title': 'mysite',
    'pageTimings': {
      'comment': ''
    },
    'comment': ''
  }
],
'entries': [
  {
    'pageref': 'mysite',
    'startedDateTime': '2017-08-25T21:38:09.367+05:30',
    'request': {
      'method': 'GET',
      'url': 'http://tarunlalwani.com/',
      'httpVersion': 'HTTP/1.1',
....

    'response': {
      'status': 200,
      'statusText': 'OK',
      'httpVersion': 'HTTP/1.1',
      'cookies': [

      ],
      'content': {
        'size': 21336,
        'mimeType': 'text/html; charset=utf-8',
        'text': '<!DOCTYPE html>\n<html lang="en">\n<head prefix="og: http://ogp.me/ns# article: http://ogp.me/ns/article# website: http://ogp.me/ns/website#">\n    <meta http-equiv="content-type" content="text/html; charset=utf-8">\n    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1">\n    \n    \n      <meta name="description" content="TARUN LALWANI">\n      <meta name="twitter:description" content="TARUN LALWANI">\n    \n\n    <meta property="og:title" content="TARUN LALWANI">\n    <meta property="twitter:title" content="TARUN LALWANI">\n    \n    <meta property="og:type" content="website">\n    \n    <meta property="og:description" content="">\n    <meta property="og:url" content="http://tarunlalwani.com/">\n    <meta property="og:site_name" content="TARUN LALWANI">\n   \n\n    \n\n    \n\n    <meta name="generator" content="Hugo 0.25.1" />\n    <title>TARUN LALWANI &middot; TARUN LALWANI</title>\n    \n    <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.3.0/css/font-awesome.min.css">\n    <link rel="stylesheet" href="http://tarunlalwani.com/css/style.css">\n\n    \n\n    \n    <link href="http://tarunlalwani.com/index.xml" rel="alternate" type="application/rss+xml" title="TARUN LALWANI" />\n    \n    \n\n    \n    \n</head>\n<body>\n\n<nav class="navbar navbar-default navbar-fixed-top visible-xs">\n\t<div class="container-fluid">\n\t\t<div class="navbar-header">\n\t\t\t<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">\n\t\t\t\t<span class="sr-only">Toggle navigation</span>\n\t\t\t\t<span class="icon-bar"></span>\n\t\t\t\t<span class="icon-bar"></span>\n\t\t\t\t<span class="icon-bar"></span>\n\t\t\t</button>\n\t\t\t\n\t\t\t\t<a class="navbar-brand" href="http://tarunlalwani.com/">TARUN LALWANI</a>\n\t\t\t\n\t\t</div>\n\t\t<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">\n\t\t\t<ul class="nav navbar-nav">\n\t\t\t\t\n\t\t\t\t\n\t\t\t</ul>\n\t\t</div>\n\t</div>\n</nav>\n<div class="container-fluid">\n\t<div class="row">\n\t\t<div id="menu" class="hidden-xs col-sm-4 col-md-3">\n\t<div id="menu-content" class="vertical-align">\n\t\t\n\t\t\t<

Answer 3

I'd recommend you to look at specialized tools for tracking network activity, like Browsermob Proxy, which could be easily integrated with Selenium. Internal Selenium logs are not good for such purposes.