In html-embedded Javascript, opening an EventSource to the server breaks Chrome's Sources and Network tabs in DevTools?

EDIT: The networks tab is the least of the concerns. Without the source tab, I cannot debug. Since I've established that the network connection is working, I can see the exact details of what is being sent without the network tab just by looking at the live log of the backend or with tcpdump.

I've been working on a new JavaScript project to render a frontend for a bunch of data that is being collected in Python and streamed to the frontend. Something that's been causing me a lot of trouble is the fact that the Chrome DevTools don't work properly while this stream is open. For example, if I bring up the Sources tab no sources are displayed. If I bring up the Network tab, no connections are displayed there.

While the Sources tab is open, if I kill the backend and thereby kill the stream's TCP connection, the source for the page pops up. Similarly, the Network tab shows nothing while the stream is open. Not only does it not show the running stream (which I know is open because the updates are being displayed in the page) but it doesn't even show the load of localhost:8000/. Killing the backend causes the Network tab to display the failed load of favicon.ico and all the retries of the stream, but not the initial load of / nor the initial run of the stream.

I've stripped this down to a very simple repro example.

#!/usr/bin/env python3

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
import time
from urllib.parse import urlparse, parse_qs

index = '''
<html>
<head>
<title>devtools lockup demo</title>
</head>
<body>
<div id='counter'>No data received yet.</div>
<script type='text/javascript' defer>

/*TODO: this doesn't really need to be a class.*/
class DataRelay {
  constructor() {
    const stream_url = '/stream/';
    this.event_source = new EventSource(stream_url);
    this.event_source.onmessage = (event) => {
      document.getElementById('counter').textContent = event.data;
    };
    this.event_source.onerror = (error) => {
      console.error('event_source.onerror:', error);
    };
    console.log('data stream handler is set up');
  }
}

let data_relay = new DataRelay();
</script>
</body>
'''

def encode_as_wire_message(data):
  # The "data: " preamble, the "\n\n" terminator, and the utf8 encoding are all
  # mandatory for streams.
  return bytes('data: ' + data + '\n\n', 'utf8')

#TODO: Get this constant in the class
class RequestHandler(BaseHTTPRequestHandler):
  def add_misc_headers(self, content_type):
    self.send_header('Content-type', content_type)
    self.send_header('Cache-Control', 'no-cache')
    self.send_header('Connection', 'keep-alive')
    self.send_header('Access-Control-Allow-Credentials', 'true')
    self.send_header('Access-Control-Allow-Origin', '*')

  def serve_index(self):
    self.send_response(200)
    self.add_misc_headers('text/html')
    self.end_headers()

    self.wfile.write(bytes(index, 'utf8'))

  def serve_stream(self):
    self.send_response(200)
    self.add_misc_headers('text/event-stream')
    self.end_headers()

    print('Beginning to serve stream...')

    for x in range(1000000):
      message = encode_as_wire_message(str(x))
      print(message)
      self.wfile.write(message)
      self.wfile.flush()
      time.sleep(1.0)

  def do_GET(self):
    parsed_url = urlparse(self.path)
    if parsed_url.path == '/':
      self.serve_index()
    elif parsed_url.path == '/stream/':
      self.serve_stream()

def run(server_class=ThreadingHTTPServer, handler_class=RequestHandler):
  server_address = ('', 8000) # serve on all interfaces, port 8000
  httpd = server_class(server_address, handler_class)
  print('starting httpd... Open a connection to http://localhost:8000')
  httpd.serve_forever()

run()

Upvotes: 3

Answers (2)

Heiko Theißen

Reputation: 17487

EDIT, by OP: The issue turned out to be that self.finish() was not being explicitly called at the end of do_GET(). The documentation for Python 3.12.3 states explicitly that finish() is called automatically at the end of do_GET() but that is clearly not the case. Adding it fixed the problem. I have marked this answer as correct so the bounty can be correctly assigned to the author because the mention of Content-Length put me on the correct path to noticing that the connection to retrieve the index was never closing. The fact that the stream was still open was a red herring in this case.... though I did not have this issue when I was only serving an index and no stream.

serve_index does not include a Content-Length header in the response. Without that, the browser cannot know when the response is finished, and unfinished responses do not appear in the Sources tab.

Despite that, the browser is able to display the potentially unfinished HTML page and consume the EventSource, as you describe.

For an example of how other programming languages approach this, Node.js with express lets you choose between

sending the entire response body with response.end, in that case the Content-Length header is set automatically, or
sending a first part of the response with response.write (and more later, asynchronously), in that case the Transfer-Encoding: chunked header is set and the response body is transferred in chunks.

I do not know if there is a similar mechanism in Python, but an easy solution is to set the Content-Length header:

def serve_index(self):
  self.send_response(200)
  self.add_misc_headers('text/html')
  self.send_header('Content-Length', len(index))
  self.end_headers()
  self.wfile.write(bytes(index, 'utf8'))

With that change the / shows up in Sources.

Addendum: The OP's EDIT taught me that (perhaps obviously) closing the connection (with self.finish()) is another way to signal the end of the response to the browser. But doing this after every request prevents the re-use of connections.

Upvotes: 1

Manas Ashwin

Reputation: 36

The networks tab of Chrome doesn't store the history unless you open it. What you should do is open the Devtools and the network tab before loading the url localhost:8000.

Or, you can also add a button that manually starts the stream instead of automatically doing when page loads. This way, you can open the devtools and once the Network Tab is open, you click the button and you'll be able to see the stream

You can replace the body tag in your index HTML with:

<body>
<h1>devtools lockup demo. Open the devtools networks tab and click the start button</h1>
<button onclick='startStream()'>Start stream</button>
<div id='counter'>No data received yet.</div>
<script type='text/javascript' defer>

/*TODO: this doesn't really need to be a class.*/
class DataRelay {
  constructor() {
    const stream_url = '/stream/';
    this.event_source = new EventSource(stream_url);
    this.event_source.onmessage = (event) => {
      document.getElementById('counter').textContent = event.data;
    };
    this.event_source.onerror = (error) => {
      console.error('event_source.onerror:', error);
    };
    console.log('data stream handler is set up');
  }
}
const startStream = () => {
let data_relay = new DataRelay();
};
</script>
</body>

Upvotes: 1

In html-embedded Javascript, opening an EventSource to the server breaks Chrome&#39;s Sources and Network tabs in DevTools?

Answers (2)

Related Questions

In html-embedded Javascript, opening an EventSource to the server breaks Chrome's Sources and Network tabs in DevTools?