Checking for open connection inside Zope thread

This won't be a ridiculously popular post, but I finally came up with a solution for an annoying problem that's been hounding me for a week, and I need to crow to somebody. As no one real is available, the Anonymous Interwebs™ will suffice (O anthropomorphized amalgam of TCP connections, I venerate your constancy!).

I've got this BrowserView that streams the contents of a file to the HTTPResponse until the file signifies (by uttering the safe word "<<<<<EOFMARKER>>>>>") that no further content is forthcoming:

    def stream(self):
        f = self.getFile()
        offset = 0
        f.seek(0, 2)
        remaining = f.tell()
        while True:
            for line in self._generate_lines(f, offset, remaining):
                if line.startswith(EOF_MARKER):
                    raise StopIteration
                yield line
            if self.finished:
                break
            offset = f.tell()
            f.seek(0, 2)
            remaining = f.tell() - offset
            del f
            time.sleep(0.1)
            f = self.getFile()

As you can see, that's a generator that yields lines until EOFMARKER is seen, and then it finishes. The view iterates over the generator and writes the lines to the response. This works very well, other things being equal.

Unfortunately, that while loop keeps the thread alive. Normally, when the client closes the connection to the server (by navigating to a different page), the thread dies. Not in this case. I'm not totally clear as to the specifics, but the cause is certainly that infinite while loop. If the loop is particularly long-running, multiple requests to view the log stream could (and do) eat up all available threads in the pool (default 4, so the phenomenon is soon evident).

I tried loads of other arrangements of the same basic pattern, and came to the conclusion that there's really no good way to have this functionality without that while. So, had to check connection status as a condition of the loop's continuation. Eventually, digging into the guts of ZServer, I found a solution: sneak up from behind.


ZPublisher.HTTPRequest.HTTPRequest objects provide (almost) no information about the connection that spawned their existence—nothing like REQUEST.connection.is_open(), for example. Since REQUEST is the only relevant information available in this context, I was forced to call upon my hidden reserve of guile. I went to the source!

ZServer uses the standard asyncore library to handle its asynchronous sockets. Cursory investigation of the module yielded the very handy dictionary of all open connections:

(Pdb) asyncore.socket_map
{6: <ZServer.HTTPServer.zhttp_server listening 0.0.0.0:8080 at 0x5bbfa8>, 
10: <ManagedClientConnection ('127.0.0.1', 8100)>, 
11: <select-trigger (pipe) at 117f378>, 
21: <ZServer.HTTPServer.zhttp_channel connected 127.0.0.1:55070 
at 0x30a3148 channel#: 15 requests:>}

That last one was the socket I cared about. It disappeared at the right time, when the browser closed the connection. I just had to find some way for Zope itself to watch the open connections and stop streaming when the right one closed. This required me to correlate the REQUEST object to the connection, so it didn't stop working when somebody else closed their browser.

Luckily, REQUEST provided one piece of relevant information: the creation time of its channel. So all I had to do was check each of the open zhttp_channel connections for the same creation time; if none matched, the connection wasn't open anymore:

def is_browser_connection_open(request):
    creation_time = request.environ['channel.creation_time']
    for cnxn in asyncore.socket_map.values():
        if (isinstance(cnxn, zhttp_channel) and 
            cnxn.creation_time==creation_time):
            return True
    return False

Then I just changed my streaming function to:

while is_browser_connection_open(REQUEST):

Worked like gangbusters. I've been sweating this issue for quite a while. Now, perhaps, I can sleep.