Seeing as how I bring Zenoss processes up and down forty times a day, I can get confused about what's running, and zenoss status gives me a bunch of stuff I don't care about. So I wrote this bash function to output a simple string representing my running daemons.

Read More...

Sorry it's taken so long, those of you who've made requests and opened tickets, but finally, there's an update to extdirect. This release adds support for batched requests as described in the Ext.Direct spec (multiple calls within a certain amount of time are gathered into a single request).

In addition, I've removed the actual ExtJS code itself so this could be released under a less restrictive license. Ext 3.x is now merely a prerequisite.

Thanks to Brian Edwards and Jon-Pierre Gentil for their work on the analogous code in Zenoss, a version of which made its way into this release.

Read More...

With ExtJS 3.0 came Ext.Direct, an excellent library for remoting server-side methods to the client side. We had already made the decision to switch to Ext with the revamped Zenoss UI, so I'd been working with Ext quite a lot; in the upcoming 2.5 release, there's a brand-new event console that makes heavy use of Ext.Direct. After learning its ins and outs with all that work, I decided to write a Python version of the server-side component to make it easier to use; I then went further and created a Zope 2 and 3 compatible component that makes it trivial. extdirect is the result. Here's some brief instruction on how to use it.

Read More...

I'm developing a feature for Zenoss that lives in its own Zope product. For the time being, it's fairly independent of Zenoss and can be demoed in a separate Zope instance. So I have a buildout for the Zope environment, also setting up a copy of the new product as a development egg (see this tutorial for an example of how to set that up). I have that development egg pulled into the Zope demo buildout via svn:externals.

At certain points in the development cycle, however, I want to send QA an RPM of the demo instance, with my package installed, so they can test it. This is kind of a hassle if I have to build the egg of the product and get it in the buildout directory on the build box in order to generate an RPM. Nor can I simply leave it as a development egg, because the package wouldn't be in the same place on the test box, so the link would fail.

So here's an easy way to turn a development egg into a production egg for the purposes of creating an RPM, using a separate buildout config file.

You'll want to read this tutorial on basic RPM creation using zc.buildout. In addition, the buildout.cfg will include a develop option in the buildout section:

[buildout]
...
develop = src/Products.MyCustomProduct

(src exists in my demo directory, and has the svn:externals definition pointing at the custom package.) The egg is also included in the instance section:

[instance]
...
eggs = 
    ${buildout:eggs}
    Products.MyCustomProduct

zcml = Products.MyCustomProduct

This gets me a development buildout, in which I can work on the package in src/Products.MyCustomProduct without building eggs or anything.

Now when it comes time to build the RPM, we need to modify the rpm config file from the tutorial to build Products.MyCustomProduct as an egg, then install it. We'll use the z3c.recipe.egg, which lets you run setup.py in a source package with arbitrary arguments. Here's the new rpm.cfg:

[buildout]
extends = buildout.cfg
develop = 
parts = 
    MyCustomProduct-build
    zope2
    instance
    zopepy
    myzope

[MyCustomProduct-build]
recipe = z3c.recipe.egg:setup
setup = src/Products.MyCustomProduct
args = bdist_egg

[instance]
find-links = 
    ${buildout:find-links}
    ${MyCustomProduct-build:setup}/dist

[myzope]
...

Explanation. So, the egg is already referenced in buildout.cfg as one of the instance eggs. All we do in this one is remove its declaration as a development egg (with develop= in the buildout section), add a part to create the egg artifact (MyCustomProduct-build), and tell the instance section where to find it (expanded find-links).

z3c.recipe.egg just needs the setup option defined, which points at a package directory containing setup.py. It'll install by default, whereas we just want it to build an egg, so we specify custom args with the args option.

Notice that we have to build the egg first. This is obvious upon reflection, but the first time I just added MyCustomProduct-build to the parts instead of making sure it was first, which meant that buildout couldn't find it in the instance step, because it didn't exist yet.

This will create a deployment instance just fine on its own, in the current directory. But in the RPM example, we build out somewhere else, so we'll have to update the buildout command in the spec file so that it can tell buildout where to find the source of the custom package. This is a modification to the %build script:

%build
...
bin/buildout -c rpm.cfg buildout:directory=%{installdir} MyCustomProduct-build:setup=%{sourcedir}/src/Products/ZenUI3

Then we build the RPM as in the tutorial. This time, it'll package up the egg in its current state and install that artifact into the Zope instance. This means that I can generate an RPM for QA on the fly with a single command, which is lots easier than assembling the parts manually.

Read More...

zc.buildout is a project that allows one to define the steps to create an application environment in config files. It is, roughly, a Makefile replacement, or at least can be used that way. It makes it very easy to reproduce deployments, both for development and for production, with nothing but a bootstrap and a config file.

The natural next step, of course, would be to use buildout to generate installable packages like RPMs. This is less straightforward than one might expect. There are probably many ways to do it, as buildout is very versatile; I'll explain one method I've come up with after the jump.

(My rpm knowledge is pretty minimal, so this may not be strictly according to doctrine, but it seems to work flawlessly in the scenarios I've tested.)

I'll use as an example the installation of a vanilla Zope 2 instance, with an rc script to control it. One may of course add one's own products to the buildout, thereby creating a custom application; that is beyond the scope of this example (see Martin Aspeli's excellent Plone-oriented but generally informative tutorial, along with the buildout docs, etc.).

Here's the config for a Zope 2.11.2 buildout:

[buildout]
parts =
    zope2
    instance
    zopepy

find-links =
    http://dist.plone.org
    http://download.zope.org/distribution/
    http://effbot.org/downloads

# Add additional eggs here
eggs =

# Reference any eggs you are developing here, one per line
# e.g.: develop = src/my.package
develop =

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.11.2/Zope-2.11.2-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
    ${buildout:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
    ${buildout:directory}/products

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

Notice I'm using Plone's recipes for setting up a Zope instance; they work as well as you might hope. If you bootstrap and buildout from this config file, you'll get yourself a Zope server listening at 8080, and a custom interpreter with the right paths and everything.

Make a new directory containing the above config file, saved as buildout.cfg, and a copy of bootstrap.py. If you like, you can make sure it works with:

$ python bootstrap.py
$ bin/buildout
$ bin/instance fg

That'll start up Zope in the foreground.

A very nice feature of zc.buildout is the ability of config files to extend others. We'll use this to create a separate buildout with the extra/different features we need for an RPM. In this minimal example, the differences are few, but you can see how it would work well for more complex buildouts. Save this as rpm.cfg in the same directory as buildout.cfg:

[buildout]
extends = buildout.cfg
parts += myzope

[myzope]
recipe = zc.recipe.rhrc
parts = zoperc

[zoperc]
run-script = ${buildout:directory}/bin/instance

Building out with this would do the same thing as our buildout.cfg, but would also install /etc/init.d/myzope as a control script, thanks to zc.recipe.rhrc.

Now we get to the (slightly) tricky part. Normally, one would build an RPM by creating a source tarball, dropping it in /usr/src/redhat/SOURCES, and pointing a spec file at it. We could do that here if we wanted to. We could also use zc.sourcerelease to create a tarball, including egg dependencies and whatnot, and do the same thing (though that irritatingly requires one to build out twice). But it's an extra step.

The directory option to buildout lets you build out into a directory other than the current. You can also override the config file from the command line. So the simplest method to create an RPM is just to pass the install path to buildout in the spec file. Here's a spec file that works (save it as myzope.spec in the same directory as rpm.cfg):

%define name myzope
%define sourcedir %(echo $PWD)
%define installdir /opt/%{name}
%define rcscript /etc/init.d/%{name}

Name: %{name}
Version: 1.0
Release: 0
Summary: My Zope Instance
URL: http://www.zope.org
License: GPL
Vendor: Me
Packager: Me <me@example.com>
Group: Applications/Database
Buildroot: /tmp/%{name}-buildroot

%description
%{summary}

%prep
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}
mkdir -p $RPM_BUILD_ROOT %{installdir}

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

%install
echo "effective-user nobody" >> %{installdir}/parts/instance/etc/zope.conf
mkdir %{installdir}/products
mkdir -p $RPM_BUILD_ROOT/etc/init.d
mkdir -p $RPM_BUILD_ROOT%{installdir}
mv %{rcscript} $RPM_BUILD_ROOT%{rcscript}
mv %{installdir}/* $RPM_BUILD_ROOT%{installdir}

%files
%defattr(-, root, root, 0755)
%{rcscript}
%defattr(-, nobody, nobody, 0755)
%{installdir}

%clean
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}

Now, step by step. First of all, the clever bit:

%define sourcedir %(echo $PWD)

This lets us find the directory with the buildout config—provided, of course, that the spec file lives in the same directory as rpm.cfg, and that you run rpmbuild from there. Notice that we also define the directory in which this will be installed, namely /opt/myzope.

The rest is basic spec file stuff, until you get to the build script:

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

So we're using the rpm config file, but overriding the directory option in the buildout section to use /opt/myzope as the location in which to build out the environment. We have to do it this way—instead of, say, building out in $RPM_BUILD_ROOT—because the paths in the buildout would then refer to /tmp/myzope-buildroot/opt/myzope instead of /opt/myzope.

Then it's as simple as doing a couple Zope-specific steps (setting the user who runs Zope), copying the resulting environment into the build root (not forgetting /etc/init.d/myzope) and letting it be packaged up.

Build it with rpmbuild -bb myzope.spec. Shizam, an RPM that you can install. Start it up with /sbin/service myzope start and check it out in the browser.

There are a few other tricks involved in separating a more complex development buildout from one that is ready for an RPM. I'll explain further in my next post.

Read More...

Zenoss uses YUI as its JavaScript framework, and until now, we've always just committed the source to the main Zenoss source tree. Seeing as how we never actually modify it, version control seems a little overkill. Also, whenever we upgrade YUI, the changeset comprises the addition and removal of hundreds of files, which is completely unnecessary overhead. We don't need to track changes to YUI; Yahoo! already does that.

The solution was to package YUI into a Zope product that consisted of a single .egg; when the package is included, the YUI source directory is available as a resource in the browser. The same solution applies, of course, to any set of static files that merely need to be traversable—CSS, images, static data, whatever. It's trivially simple, but I couldn't find any clear tutorials. Plus I'm on a plane and don't feel like reading. Here's how to wrap any directory into a Zope product.

Throughout this, I'll be using the YUI package I created as an example. Obviously, you'd replace all instances of "zenoss" and "yui" with names appropriate to your package.

1. Create the egg structure. Your best friend here is PasteScript (easy_install PasteScript if you don't have it already), which can generate skeleton structures from several different templates. In this case, you simply need an egg. You can either create the simplest kind of egg, or you can make a namespace package (for example, "yui" vs. "zenoss.yui"). I recommend the latter, if only to avoid conflicts with others who might have had the same idea:
   

   $ paster create -t basic_namespace zenoss.yui
   

   
   (If you just want a simple egg, use the basic_package template instead).
   
   Enter the namespace ("zenoss") and the package ("yui") and any other information you care to. You'll be asked for the basic egg metadata—license, description, author, etc. All is optional, but if you're going to distribute your package you should definitely provide this stuff. Don't worry, you can do it later by modifying setup.py.
   
   When you're done, you'll have an egg structure in the directory you specified (in this case, "zenoss.yui").


2. Add the payload. Go down into the egg structure (In this example, zenoss.yui/zenoss/yui) and create the directory to hold your files:
   

   $ cd zenoss.yui/zenoss/yui
   $ mkdir src
   

   
   Copy any files you want available as resources into that directory. There's no need to create a yui directory inside the src directory; the entire resource directory will be available at /++resource++yui already (see below for a way to get rid of the ++resource++ cruft if so inclined), so if you created, for example, src/myfile.js, you would use the URL /++resource++yui/myfile.js to traverse to it.
   
   Now edit the new file zenoss.yui/MANIFEST.in (it'll be a sibling of setup.py) and add the following line:
   

   graft zenoss

   
   That tells setuptools to include every file underneath the directory zenoss; otherwise, it's likely to ignore most of the files you've added.
   


3. Register src as a Zope resource directory. Edit the new file zenoss.yui/zenoss/yui/configure.zcml and add the ZCML necessary to wire the directory into Zope's publisher:
   

   
   <configure
       xmlns="http://namespaces.zope.org/zope"
       xmlns:browser="http://namespaces.zope.org/browser"
       >
   
       <browser:resourceDirectory
           name="yui"
           directory="src"
           />
   
   </configure>
   

   
   


4. You're done. Test. You can now build an egg for use in any Zope application. Go up to the root of the package, where setup.py is located. Build the egg:
   

   $ python setup.py bdist_egg

   
   That'll create an egg in dist. Install that in whatever way you like, as long as it ends up in the Python path of your Zope environment—manually, with easy_install, or by including it in a buildout recipe. Then, somewhere in the ZCML chain of your Zope application, include the package:
   

   <include package="zenoss.yui"/>

   
   Now fire up Zope and try the URL ++resource++yui/anyfile.js against the root object, where anyfile.js is some file directly under src in your package. As I explained earlier, you should be able to get to anything in the src directory by mirroring the filesystem path in the URL, replacing src with ++resource++yui.

I mentioned a technique for removing the necessity to include ++resource++ as part of the URL. You may not want to do this; I only did it because all my template files already referred to /yui and I didn't feel like changing them. In the same package in your Zope application in which you included zenoss.yui, create a file called YUIShortcut.py (you can put it anywhere, of course, but this keeps things organized). If you're using Five:

from Products.Five.browser import BrowserView

If Zope 3:

from zope.publisher.browser import BrowserView

Then make a view that passes traversal off to the zenoss.yui resourceDirectory:

class YUIShortcut(BrowserView):
    def __getitem__(self, name):
        return self.context.unrestrictedTraverse(
                                       '++resource++yui')[name]

Then wire it to the /yui URL in configure.zcml:

<browser:page
    for="*"
    name="yui"
    class=".YUIShortcut.YUIShortcut"
    permission="zope2.Public"
    >

Now you should be able to hit /yui/myfile.js instead of /++resource++yui/myfile.js.

Read More...

This won't be a ridiculously popular post, but I finally came up with a solution for an annoying problem that's been hounding me for a week, and I need to crow to somebody. As no one real is available, the Anonymous Interwebs™ will suffice (O anthropomorphized amalgam of TCP connections, I venerate your constancy!).

I've got this BrowserView that streams the contents of a file to the HTTPResponse until the file signifies (by uttering the safe word "<<<<<EOFMARKER>>>>>") that no further content is forthcoming:

    def stream(self):
        f = self.getFile()
        offset = 0
        f.seek(0, 2)
        remaining = f.tell()
        while True:
            for line in self._generate_lines(f, offset, remaining):
                if line.startswith(EOF_MARKER):
                    raise StopIteration
                yield line
            if self.finished:
                break
            offset = f.tell()
            f.seek(0, 2)
            remaining = f.tell() - offset
            del f
            time.sleep(0.1)
            f = self.getFile()

As you can see, that's a generator that yields lines until EOFMARKER is seen, and then it finishes. The view iterates over the generator and writes the lines to the response. This works very well, other things being equal.

Unfortunately, that while loop keeps the thread alive. Normally, when the client closes the connection to the server (by navigating to a different page), the thread dies. Not in this case. I'm not totally clear as to the specifics, but the cause is certainly that infinite while loop. If the loop is particularly long-running, multiple requests to view the log stream could (and do) eat up all available threads in the pool (default 4, so the phenomenon is soon evident).

I tried loads of other arrangements of the same basic pattern, and came to the conclusion that there's really no good way to have this functionality without that while. So, had to check connection status as a condition of the loop's continuation. Eventually, digging into the guts of ZServer, I found a solution: sneak up from behind.


ZPublisher.HTTPRequest.HTTPRequest objects provide (almost) no information about the connection that spawned their existence—nothing like REQUEST.connection.is_open(), for example. Since REQUEST is the only relevant information available in this context, I was forced to call upon my hidden reserve of guile. I went to the source!

ZServer uses the standard asyncore library to handle its asynchronous sockets. Cursory investigation of the module yielded the very handy dictionary of all open connections:

(Pdb) asyncore.socket_map
{6: <ZServer.HTTPServer.zhttp_server listening 0.0.0.0:8080 at 0x5bbfa8>, 
10: <ManagedClientConnection ('127.0.0.1', 8100)>, 
11: <select-trigger (pipe) at 117f378>, 
21: <ZServer.HTTPServer.zhttp_channel connected 127.0.0.1:55070 
at 0x30a3148 channel#: 15 requests:>}

That last one was the socket I cared about. It disappeared at the right time, when the browser closed the connection. I just had to find some way for Zope itself to watch the open connections and stop streaming when the right one closed. This required me to correlate the REQUEST object to the connection, so it didn't stop working when somebody else closed their browser.

Luckily, REQUEST provided one piece of relevant information: the creation time of its channel. So all I had to do was check each of the open zhttp_channel connections for the same creation time; if none matched, the connection wasn't open anymore:

def is_browser_connection_open(request):
    creation_time = request.environ['channel.creation_time']
    for cnxn in asyncore.socket_map.values():
        if (isinstance(cnxn, zhttp_channel) and 
            cnxn.creation_time==creation_time):
            return True
    return False

Then I just changed my streaming function to:

while is_browser_connection_open(REQUEST):

Worked like gangbusters. I've been sweating this issue for quite a while. Now, perhaps, I can sleep.

Read More...