Using zc.buildout to generate an RPM

zc.buildout is a project that allows one to define the steps to create an application environment in config files. It is, roughly, a Makefile replacement, or at least can be used that way. It makes it very easy to reproduce deployments, both for development and for production, with nothing but a bootstrap and a config file.

The natural next step, of course, would be to use buildout to generate installable packages like RPMs. This is less straightforward than one might expect. There are probably many ways to do it, as buildout is very versatile; I'll explain one method I've come up with after the jump.

(My rpm knowledge is pretty minimal, so this may not be strictly according to doctrine, but it seems to work flawlessly in the scenarios I've tested.)

I'll use as an example the installation of a vanilla Zope 2 instance, with an rc script to control it. One may of course add one's own products to the buildout, thereby creating a custom application; that is beyond the scope of this example (see Martin Aspeli's excellent Plone-oriented but generally informative tutorial, along with the buildout docs, etc.).

Here's the config for a Zope 2.11.2 buildout:

[buildout]
parts =
    zope2
    instance
    zopepy

find-links =
    http://dist.plone.org
    http://download.zope.org/distribution/
    http://effbot.org/downloads

# Add additional eggs here
eggs =

# Reference any eggs you are developing here, one per line
# e.g.: develop = src/my.package
develop =

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.11.2/Zope-2.11.2-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
    ${buildout:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
    ${buildout:directory}/products

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

Notice I'm using Plone's recipes for setting up a Zope instance; they work as well as you might hope. If you bootstrap and buildout from this config file, you'll get yourself a Zope server listening at 8080, and a custom interpreter with the right paths and everything.

Make a new directory containing the above config file, saved as buildout.cfg, and a copy of bootstrap.py. If you like, you can make sure it works with:

$ python bootstrap.py
$ bin/buildout
$ bin/instance fg

That'll start up Zope in the foreground.

A very nice feature of zc.buildout is the ability of config files to extend others. We'll use this to create a separate buildout with the extra/different features we need for an RPM. In this minimal example, the differences are few, but you can see how it would work well for more complex buildouts. Save this as rpm.cfg in the same directory as buildout.cfg:

[buildout]
extends = buildout.cfg
parts += myzope

[myzope]
recipe = zc.recipe.rhrc
parts = zoperc

[zoperc]
run-script = ${buildout:directory}/bin/instance

Building out with this would do the same thing as our buildout.cfg, but would also install /etc/init.d/myzope as a control script, thanks to zc.recipe.rhrc.

Now we get to the (slightly) tricky part. Normally, one would build an RPM by creating a source tarball, dropping it in /usr/src/redhat/SOURCES, and pointing a spec file at it. We could do that here if we wanted to. We could also use zc.sourcerelease to create a tarball, including egg dependencies and whatnot, and do the same thing (though that irritatingly requires one to build out twice). But it's an extra step.

The directory option to buildout lets you build out into a directory other than the current. You can also override the config file from the command line. So the simplest method to create an RPM is just to pass the install path to buildout in the spec file. Here's a spec file that works (save it as myzope.spec in the same directory as rpm.cfg):

%define name myzope
%define sourcedir %(echo $PWD)
%define installdir /opt/%{name}
%define rcscript /etc/init.d/%{name}

Name: %{name}
Version: 1.0
Release: 0
Summary: My Zope Instance
URL: http://www.zope.org
License: GPL
Vendor: Me
Packager: Me <me@example.com>
Group: Applications/Database
Buildroot: /tmp/%{name}-buildroot

%description
%{summary}

%prep
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}
mkdir -p $RPM_BUILD_ROOT %{installdir}

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

%install
echo "effective-user nobody" >> %{installdir}/parts/instance/etc/zope.conf
mkdir %{installdir}/products
mkdir -p $RPM_BUILD_ROOT/etc/init.d
mkdir -p $RPM_BUILD_ROOT%{installdir}
mv %{rcscript} $RPM_BUILD_ROOT%{rcscript}
mv %{installdir}/* $RPM_BUILD_ROOT%{installdir}

%files
%defattr(-, root, root, 0755)
%{rcscript}
%defattr(-, nobody, nobody, 0755)
%{installdir}

%clean
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}

Now, step by step. First of all, the clever bit:

%define sourcedir %(echo $PWD)

This lets us find the directory with the buildout config—provided, of course, that the spec file lives in the same directory as rpm.cfg, and that you run rpmbuild from there. Notice that we also define the directory in which this will be installed, namely /opt/myzope.

The rest is basic spec file stuff, until you get to the build script:

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

So we're using the rpm config file, but overriding the directory option in the buildout section to use /opt/myzope as the location in which to build out the environment. We have to do it this way—instead of, say, building out in $RPM_BUILD_ROOT—because the paths in the buildout would then refer to /tmp/myzope-buildroot/opt/myzope instead of /opt/myzope.

Then it's as simple as doing a couple Zope-specific steps (setting the user who runs Zope), copying the resulting environment into the build root (not forgetting /etc/init.d/myzope) and letting it be packaged up.

Build it with rpmbuild -bb myzope.spec. Shizam, an RPM that you can install. Start it up with /sbin/service myzope start and check it out in the browser.

There are a few other tricks involved in separating a more complex development buildout from one that is ready for an RPM. I'll explain further in my next post.