Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Decorator Factories (and decorator factory factories)

Decorators are a simple, handy way of implementing certain features in any language, but many people find them perplexing. When you start talking about functions that generate decorators, it gets a little twisty; when you then make helpful functions to create functions that create decorators, you might have to stop for a minute to unclench your brain. Really, though, the concept is quite simple if you take it step by step, so here it is (after the jump), hopefully demystified.

Read More...

Released: txrestapi 0.1

Sorry updates have been horribly infrequent; I both had a kid and moved to Austin (in that order) in the last few months. With that out of the way, to the code...

I got frustrated chaining twisted.web.resource.Resource classes to provide a complex REST API service, so I wrote a new package that lets you do it with regular expressions and decorators, like:

@GET('^/path/to/api/method$')
    def on_api_method(self, request):
       ...
I'm pretty fond of it already. Code is on GitHub, package is available from pypi — and, of course, via easy_install. Full docs are at both sites, quick example after the jump.

Read More...

extdirect 0.4 released: Batched requests, new licensing

Sorry it's taken so long, those of you who've made requests and opened tickets, but finally, there's an update to extdirect. This release adds support for batched requests as described in the Ext.Direct spec (multiple calls within a certain amount of time are gathered into a single request).

In addition, I've removed the actual ExtJS code itself so this could be released under a less restrictive license. Ext 3.x is now merely a prerequisite.

Thanks to Brian Edwards and Jon-Pierre Gentil for their work on the analogous code in Zenoss, a version of which made its way into this release.

Read More...

Run pyflakes/jslint automatically in Vim

As maybe you can tell from the infrequency of updates to this blog, I (and the others with whom I'm working) have been churning out a huge amount of code recently, split roughly evenly between Python and JavaScript. One nice thing I got used to during the month I tried to switch from Vim to TextMate was a plugin that ran pyflakes every time I saved a Python file. After I gave up and went back to Vim, I missed that check. Luckily, other, more vimscript-savvy coders had the same idea. Here's how to set up both pyflakes and jslint to run on save.

Read More...

Ext.Direct remoting in Django

Well, as I didn't have a lot of other things weighing on me today, I created a Django implementation of the extdirect package I released earlier today. It's slightly more involved to set up, probably due to my not knowing Django nearly as well as Zope (suggestions welcome, Django enthusiasts), but after the initial setup writing the router classes is just as easy. Here's how to do it.

Read More...

Ext.Direct remoting in Zope

With ExtJS 3.0 came Ext.Direct, an excellent library for remoting server-side methods to the client side. We had already made the decision to switch to Ext with the revamped Zenoss UI, so I'd been working with Ext quite a lot; in the upcoming 2.5 release, there's a brand-new event console that makes heavy use of Ext.Direct. After learning its ins and outs with all that work, I decided to write a Python version of the server-side component to make it easier to use; I then went further and created a Zope 2 and 3 compatible component that makes it trivial. extdirect is the result. Here's some brief instruction on how to use it.

Read More...

Vim, Part I: Improved Python syntax, Blackboard color scheme

Every few months, I notice that TextMate exists and try to use it; inevitably, the requirement that I use arrow keys defeats the endeavor. It has a few features, however, that I like, most of which I've ported to vim, my primary text editor. These features are too numerous for a single post, so I'll highlight them in a series. First up: An improved Python syntax definition and my favorite color scheme.

Python Syntax


The most popular Python syntax highlighting script is pretty complete, but it's missing a few things (Update: several of the improvements I made a year ago have been added to Dmitry Vasiliev's script, such as decorators, doctests, imports, etc. I've merged the two, so this is slightly less special than before). Specifically: docstrings, self, arguments/base classes, assignment as distinct from other operators, and a few other little things. TextMate had these, so I ported them, and the results were pretty good.

Download the improved syntax file and drop it into ~/.vim/syntax.

Blackboard Color Scheme


TextMate had a color scheme that I loved: Blackboard. The colors were primaryish without being irritatingly bright or clashing; keywords, builtins, and class definitions were given appropriate prominence. I ported it to Vim. Here's what it looks like:



That example doesn't include a doctest, but they are highlighted separately in the same color as builtins.

Download the color scheme and drop it into ~/.vim/colors, then do :colorscheme blackboard to see the results. Pretty easy on the eyes, no?


Next in my Vim series: automatic handling of pair completion (and deletion).

Read More...

Rapid development egg to RPM with buildout

I'm developing a feature for Zenoss that lives in its own Zope product. For the time being, it's fairly independent of Zenoss and can be demoed in a separate Zope instance. So I have a buildout for the Zope environment, also setting up a copy of the new product as a development egg (see this tutorial for an example of how to set that up). I have that development egg pulled into the Zope demo buildout via svn:externals.

At certain points in the development cycle, however, I want to send QA an RPM of the demo instance, with my package installed, so they can test it. This is kind of a hassle if I have to build the egg of the product and get it in the buildout directory on the build box in order to generate an RPM. Nor can I simply leave it as a development egg, because the package wouldn't be in the same place on the test box, so the link would fail.

So here's an easy way to turn a development egg into a production egg for the purposes of creating an RPM, using a separate buildout config file.

You'll want to read this tutorial on basic RPM creation using zc.buildout. In addition, the buildout.cfg will include a develop option in the buildout section:

[buildout]
...
develop = src/Products.MyCustomProduct

(src exists in my demo directory, and has the svn:externals definition pointing at the custom package.) The egg is also included in the instance section:
[instance]
...
eggs =
${buildout:eggs}
Products.MyCustomProduct

zcml = Products.MyCustomProduct

This gets me a development buildout, in which I can work on the package in src/Products.MyCustomProduct without building eggs or anything.

Now when it comes time to build the RPM, we need to modify the rpm config file from the tutorial to build Products.MyCustomProduct as an egg, then install it. We'll use the z3c.recipe.egg, which lets you run setup.py in a source package with arbitrary arguments. Here's the new rpm.cfg:
[buildout]
extends = buildout.cfg
develop =
parts =
MyCustomProduct-build
zope2
instance
zopepy
myzope

[MyCustomProduct-build]
recipe = z3c.recipe.egg:setup
setup = src/Products.MyCustomProduct
args = bdist_egg

[instance]
find-links =
${buildout:find-links}
${MyCustomProduct-build:setup}/dist

[myzope]
...

Explanation. So, the egg is already referenced in buildout.cfg as one of the instance eggs. All we do in this one is remove its declaration as a development egg (with develop= in the buildout section), add a part to create the egg artifact (MyCustomProduct-build), and tell the instance section where to find it (expanded find-links).

z3c.recipe.egg just needs the setup option defined, which points at a package directory containing setup.py. It'll install by default, whereas we just want it to build an egg, so we specify custom args with the args option.

Notice that we have to build the egg first. This is obvious upon reflection, but the first time I just added MyCustomProduct-build to the parts instead of making sure it was first, which meant that buildout couldn't find it in the instance step, because it didn't exist yet.

This will create a deployment instance just fine on its own, in the current directory. But in the RPM example, we build out somewhere else, so we'll have to update the buildout command in the spec file so that it can tell buildout where to find the source of the custom package. This is a modification to the %build script:
%build
...
bin/buildout -c rpm.cfg buildout:directory=%{installdir} MyCustomProduct-build:setup=%{sourcedir}/src/Products/ZenUI3


Then we build the RPM as in the tutorial. This time, it'll package up the egg in its current state and install that artifact into the Zope instance. This means that I can generate an RPM for QA on the fly with a single command, which is lots easier than assembling the parts manually.

Read More...

Using zc.buildout to generate an RPM

zc.buildout is a project that allows one to define the steps to create an application environment in config files. It is, roughly, a Makefile replacement, or at least can be used that way. It makes it very easy to reproduce deployments, both for development and for production, with nothing but a bootstrap and a config file.

The natural next step, of course, would be to use buildout to generate installable packages like RPMs. This is less straightforward than one might expect. There are probably many ways to do it, as buildout is very versatile; I'll explain one method I've come up with after the jump.

(My rpm knowledge is pretty minimal, so this may not be strictly according to doctrine, but it seems to work flawlessly in the scenarios I've tested.)

I'll use as an example the installation of a vanilla Zope 2 instance, with an rc script to control it. One may of course add one's own products to the buildout, thereby creating a custom application; that is beyond the scope of this example (see Martin Aspeli's excellent Plone-oriented but generally informative tutorial, along with the buildout docs, etc.).

Here's the config for a Zope 2.11.2 buildout:

[buildout]
parts =
zope2
instance
zopepy

find-links =
http://dist.plone.org
http://download.zope.org/distribution/
http://effbot.org/downloads

# Add additional eggs here
eggs =

# Reference any eggs you are developing here, one per line
# e.g.: develop = src/my.package
develop =

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.11.2/Zope-2.11.2-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
${buildout:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
${buildout:directory}/products

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy


Notice I'm using Plone's recipes for setting up a Zope instance; they work as well as you might hope. If you bootstrap and buildout from this config file, you'll get yourself a Zope server listening at 8080, and a custom interpreter with the right paths and everything.

Make a new directory containing the above config file, saved as buildout.cfg, and a copy of bootstrap.py. If you like, you can make sure it works with:

$ python bootstrap.py
$ bin/buildout
$ bin/instance fg

That'll start up Zope in the foreground.

A very nice feature of zc.buildout is the ability of config files to extend others. We'll use this to create a separate buildout with the extra/different features we need for an RPM. In this minimal example, the differences are few, but you can see how it would work well for more complex buildouts. Save this as rpm.cfg in the same directory as buildout.cfg:

[buildout]
extends = buildout.cfg
parts += myzope

[myzope]
recipe = zc.recipe.rhrc
parts = zoperc

[zoperc]
run-script = ${buildout:directory}/bin/instance


Building out with this would do the same thing as our buildout.cfg, but would also install /etc/init.d/myzope as a control script, thanks to zc.recipe.rhrc.

Now we get to the (slightly) tricky part. Normally, one would build an RPM by creating a source tarball, dropping it in /usr/src/redhat/SOURCES, and pointing a spec file at it. We could do that here if we wanted to. We could also use zc.sourcerelease to create a tarball, including egg dependencies and whatnot, and do the same thing (though that irritatingly requires one to build out twice). But it's an extra step.

The directory option to buildout lets you build out into a directory other than the current. You can also override the config file from the command line. So the simplest method to create an RPM is just to pass the install path to buildout in the spec file. Here's a spec file that works (save it as myzope.spec in the same directory as rpm.cfg):

%define name myzope
%define sourcedir %(echo $PWD)
%define installdir /opt/%{name}
%define rcscript /etc/init.d/%{name}

Name: %{name}
Version: 1.0
Release: 0
Summary: My Zope Instance
URL: http://www.zope.org
License: GPL
Vendor: Me
Packager: Me <me@example.com>
Group: Applications/Database
Buildroot: /tmp/%{name}-buildroot

%description
%{summary}

%prep
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}
mkdir -p $RPM_BUILD_ROOT %{installdir}

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

%install
echo "effective-user nobody" >> %{installdir}/parts/instance/etc/zope.conf
mkdir %{installdir}/products
mkdir -p $RPM_BUILD_ROOT/etc/init.d
mkdir -p $RPM_BUILD_ROOT%{installdir}
mv %{rcscript} $RPM_BUILD_ROOT%{rcscript}
mv %{installdir}/* $RPM_BUILD_ROOT%{installdir}

%files
%defattr(-, root, root, 0755)
%{rcscript}
%defattr(-, nobody, nobody, 0755)
%{installdir}

%clean
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}


Now, step by step. First of all, the clever bit:

%define sourcedir %(echo $PWD)

This lets us find the directory with the buildout config—provided, of course, that the spec file lives in the same directory as rpm.cfg, and that you run rpmbuild from there. Notice that we also define the directory in which this will be installed, namely /opt/myzope.

The rest is basic spec file stuff, until you get to the build script:
%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

So we're using the rpm config file, but overriding the directory option in the buildout section to use /opt/myzope as the location in which to build out the environment. We have to do it this way—instead of, say, building out in $RPM_BUILD_ROOT—because the paths in the buildout would then refer to /tmp/myzope-buildroot/opt/myzope instead of /opt/myzope.

Then it's as simple as doing a couple Zope-specific steps (setting the user who runs Zope), copying the resulting environment into the build root (not forgetting /etc/init.d/myzope) and letting it be packaged up.

Build it with rpmbuild -bb myzope.spec. Shizam, an RPM that you can install. Start it up with /sbin/service myzope start and check it out in the browser.

There are a few other tricks involved in separating a more complex development buildout from one that is ready for an RPM. I'll explain further in my next post.

Read More...

cliutils has a user not named Ian!

Jason Friedland has very kindly submitted my cliutils package for inclusion in Fedora. Thanks! Pretty exciting to learn that somebody is using it. Now I'm trying to come up with ways to improve it. Any ideas for features to add, tens of readers?

Read More...

Human-comparison sort in Python

I recently had a need to sort a large list of items (I admit freely that this list comprised episodes of Battlestar Galactica) by a subjective criterion (my liking). This turned into a minor interface problem; how best to sort these without a) a Netflix queue interface or b) dragging Excel rows around? Also, how could I implement an efficient sorting algorithm, rather than waste time with the simplest bubble sort?

The solution I came up with was almost too simple to feel proud of. Python uses an adaptive mergesort by default, which was plenty good enough for my purposes. All I had to do, then, was have it ask a human to compare each pair of elements.


QUESTION_TEMPLATE = """
1. %s or
2. %s or
3. neither? [neither] """

def human_cmp(a, b):
answer = raw_input(
QUESTION_TEMPLATE % (a, b))
if answer=="1": return -1
elif answer=="2": return 1
else: return 0

EPISODES = open('bsg_episodes.txt').readlines()
EPISODES.sort(human_cmp)
for i, title in EPISODES:
print '%s. %s' % (i, title)

So it just uses the built-in algorithm and asks you for each comparison. Let me tell you, way better than creating a single-elimination tournament bracket in Excel.

Read More...

How Joseph Method Stole My Day

My friend Method shared this with me via Google Reader this morning, accompanied by the provocative message, "Here you go, Ian." The problem, in short, was to discover the first ten-digit prime in consecutive digits of e. And a day of planned work-catching-up dissolved before it began.

I had a solution in twenty minutes or so, using third-party libraries pycrypto (for isPrime) and mpmath (for e), but that made the whole enterprise trivial and didn't quite seem cricket. Also, it was slightly inefficient, because, being limited by the code of others, I had to calculate e to n digits first, and then check that for primes; I figured if I could calculate digits of e lazily I'd get an extremely minor but nonetheless pride-inducing speed increase.

To Wikipedia! where I learned scads about primality tests (luckily I didn't read the page on the constant e until later, or I would have learned that this was a common puzzle that pretty much everybody tries to solve. Also I would have learned the answer). I remembered how to calculate e using the Taylor series 1 + 1/1! + 1/2! + 1/3!..., so that wasn't a problem—until I ran into precision woes.

That led me down a dark path whereon I attempted to program arbitrary-precision arithmetic from scratch, and learned all about mantissas and bitwise operators over the course of five hours. I got pretty far with that before I realized that I was spending my entire day trying to comprehend what I imagine amounts to at least a month of college-level CS. I went to school for other things, like French poetry.

Also at this point I remembered the existence of Python's own decimal floating-point arithmetic package, decimal, which had apparently dropped completely out of my brain exactly when I needed it. Thankfully, it resurfaced before I had to give up on this psychotic enterprise altogether.

After that it was easy. I post the code below not because I believe it is in any way notable—I'm sure it's old hat to those who went to school for this—but because I just want someone to know that I actually did something today. Thanks, Method.


from random import randint
from operator import mul
from collections import deque
from decimal import Decimal, getcontext

# Allow enough precision to get an answer
getcontext().prec = 1000

def factorial(x):
if x==1: return x
return reduce(mul, xrange(2, x+1))

def taylor_series():
"""
e == 1/1! + 1/2! + 1/3! ...
"""
s = Decimal(1)
_e = Decimal(1)
while True:
_e += (Decimal(1)/factorial(s))
yield _e
s += 1

def digits_of_e():
"""
Yield digits of e lazily by noticing when
they stop changing as the Taylor series
converges.
"""
idx = 2 # skip '2.'
last = None
for iteration in taylor_series():
i = str(iteration)
try:
cur = i[idx]
except IndexError:
continue
if cur==last:
idx += 1
yield cur
last = i[idx]

def ten_digit_chunks():
"""
Collect digits of e as they are discovered
and yield them in ten-digit chunks.
"""
l = deque()
for digit in digits_of_e():
l.append(digit)
# Keep the stack at 10 by popping
# from the left
while len(l)>10:
l.popleft()
if len(l)==10:
yield ''.join(l)

def is_prime(n, k=20):
"""
Simple implementation of the Miller-Rabin
primality test. Thanks, Wikipedia!

n is the integer to be tested; k is the
precision (number of random possible
witnesses to test against)
"""
# Find s and d such that (2**s)*d==n
d, s = n-1, 0
while not d & 1: # d & 1 == d % 2, only faster
d >>= 1 # d >> 1 == d / 2, only faster
s += 1
# Test for k random numbers
for i in xrange(k):
_c = False
a = randint(2, n-2)
x = pow(a, d, n) # a**d % n
if x==1 or x==n-1:
continue
for r in xrange(1, s):
x = pow(x, 2, n) # x**2 % n
if x==1:
return False
elif x==n-1:
_c = True
break
if _c: continue
return False
return True

def main():
for chunk in ten_digit_chunks():
print "Trying %s..." % chunk
if is_prime(long(chunk)):
print "="*20
print
print "Found! First ten-digit"
print "prime in consecutive digits"
print "of e is %s!" % chunk
print
print "="*20
break

if __name__=="__main__":
main()


UPDATE: In order to get through this ridiculous set of problems to the unsatisfying and insulting end, you can use this implementation of the Brent-Salamin algorithm to find primes in digits of pi, replacing taylor_series in the digits_of_e function in the code above:

def brent_salamin():
"""
A series whose sum converges on pi
incredibly quickly.
"""
a = Decimal("1.")
b = Decimal("1.")/(Decimal("2.").sqrt())
t = Decimal("1.")/Decimal("4.")
p = Decimal("1.")
while True:
a1 = Decimal(a + b)/Decimal("2.")
b1 = Decimal(a*b).sqrt()
t1 = t - p*((a - a1)**2)
p1 = 2*p
yield ((a1+b1)**2)/(t1*Decimal("4."))
a, b, t, p = a1, b1, t1, p1

Read More...

How to create a Zope resource package

Zenoss uses YUI as its JavaScript framework, and until now, we've always just committed the source to the main Zenoss source tree. Seeing as how we never actually modify it, version control seems a little overkill. Also, whenever we upgrade YUI, the changeset comprises the addition and removal of hundreds of files, which is completely unnecessary overhead. We don't need to track changes to YUI; Yahoo! already does that.

The solution was to package YUI into a Zope product that consisted of a single .egg; when the package is included, the YUI source directory is available as a resource in the browser. The same solution applies, of course, to any set of static files that merely need to be traversable—CSS, images, static data, whatever. It's trivially simple, but I couldn't find any clear tutorials. Plus I'm on a plane and don't feel like reading. Here's how to wrap any directory into a Zope product.

Throughout this, I'll be using the YUI package I created as an example. Obviously, you'd replace all instances of "zenoss" and "yui" with names appropriate to your package.


  1. Create the egg structure. Your best friend here is PasteScript (easy_install PasteScript if you don't have it already), which can generate skeleton structures from several different templates. In this case, you simply need an egg. You can either create the simplest kind of egg, or you can make a namespace package (for example, "yui" vs. "zenoss.yui"). I recommend the latter, if only to avoid conflicts with others who might have had the same idea:
    $ paster create -t basic_namespace zenoss.yui

    (If you just want a simple egg, use the basic_package template instead).

    Enter the namespace ("zenoss") and the package ("yui") and any other information you care to. You'll be asked for the basic egg metadata—license, description, author, etc. All is optional, but if you're going to distribute your package you should definitely provide this stuff. Don't worry, you can do it later by modifying setup.py.

    When you're done, you'll have an egg structure in the directory you specified (in this case, "zenoss.yui").

  2. Add the payload. Go down into the egg structure (In this example, zenoss.yui/zenoss/yui) and create the directory to hold your files:
    $ cd zenoss.yui/zenoss/yui
    $ mkdir src

    Copy any files you want available as resources into that directory. There's no need to create a yui directory inside the src directory; the entire resource directory will be available at /++resource++yui already (see below for a way to get rid of the ++resource++ cruft if so inclined), so if you created, for example, src/myfile.js, you would use the URL /++resource++yui/myfile.js to traverse to it.

    Now edit the new file zenoss.yui/MANIFEST.in (it'll be a sibling of setup.py) and add the following line:
    graft zenoss

    That tells setuptools to include every file underneath the directory zenoss; otherwise, it's likely to ignore most of the files you've added.

  3. Register src as a Zope resource directory. Edit the new file zenoss.yui/zenoss/yui/configure.zcml and add the ZCML necessary to wire the directory into Zope's publisher:

    <configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:browser="http://namespaces.zope.org/browser"
    >

    <browser:resourceDirectory
    name="yui"
    directory="src"
    />

    </configure>


  4. You're done. Test. You can now build an egg for use in any Zope application. Go up to the root of the package, where setup.py is located. Build the egg:
    $ python setup.py bdist_egg

    That'll create an egg in dist. Install that in whatever way you like, as long as it ends up in the Python path of your Zope environment—manually, with easy_install, or by including it in a buildout recipe. Then, somewhere in the ZCML chain of your Zope application, include the package:
    <include package="zenoss.yui"/>

    Now fire up Zope and try the URL ++resource++yui/anyfile.js against the root object, where anyfile.js is some file directly under src in your package. As I explained earlier, you should be able to get to anything in the src directory by mirroring the filesystem path in the URL, replacing src with ++resource++yui.


I mentioned a technique for removing the necessity to include ++resource++ as part of the URL. You may not want to do this; I only did it because all my template files already referred to /yui and I didn't feel like changing them. In the same package in your Zope application in which you included zenoss.yui, create a file called YUIShortcut.py (you can put it anywhere, of course, but this keeps things organized). If you're using Five:
from Products.Five.browser import BrowserView

If Zope 3:
from zope.publisher.browser import BrowserView

Then make a view that passes traversal off to the zenoss.yui resourceDirectory:
class YUIShortcut(BrowserView):
def __getitem__(self, name):
return self.context.unrestrictedTraverse(
'++resource++yui')[name]

Then wire it to the /yui URL in configure.zcml:
<browser:page
for="*"
name="yui"
class=".YUIShortcut.YUIShortcut"
permission="zope2.Public"
>

Now you should be able to hit /yui/myfile.js instead of /++resource++yui/myfile.js.

Read More...

Checking for open connection inside Zope thread

This won't be a ridiculously popular post, but I finally came up with a solution for an annoying problem that's been hounding me for a week, and I need to crow to somebody. As no one real is available, the Anonymous Interwebs™ will suffice (O anthropomorphized amalgam of TCP connections, I venerate your constancy!).

I've got this BrowserView that streams the contents of a file to the HTTPResponse until the file signifies (by uttering the safe word "<<<<<EOFMARKER>>>>>") that no further content is forthcoming:


def stream(self):
f = self.getFile()
offset = 0
f.seek(0, 2)
remaining = f.tell()
while True:
for line in self._generate_lines(f, offset, remaining):
if line.startswith(EOF_MARKER):
raise StopIteration
yield line
if self.finished:
break
offset = f.tell()
f.seek(0, 2)
remaining = f.tell() - offset
del f
time.sleep(0.1)
f = self.getFile()

As you can see, that's a generator that yields lines until EOFMARKER is seen, and then it finishes. The view iterates over the generator and writes the lines to the response. This works very well, other things being equal.

Unfortunately, that while loop keeps the thread alive. Normally, when the client closes the connection to the server (by navigating to a different page), the thread dies. Not in this case. I'm not totally clear as to the specifics, but the cause is certainly that infinite while loop. If the loop is particularly long-running, multiple requests to view the log stream could (and do) eat up all available threads in the pool (default 4, so the phenomenon is soon evident).

I tried loads of other arrangements of the same basic pattern, and came to the conclusion that there's really no good way to have this functionality without that while. So, had to check connection status as a condition of the loop's continuation. Eventually, digging into the guts of ZServer, I found a solution: sneak up from behind.


ZPublisher.HTTPRequest.HTTPRequest objects provide (almost) no information about the connection that spawned their existence—nothing like REQUEST.connection.is_open(), for example. Since REQUEST is the only relevant information available in this context, I was forced to call upon my hidden reserve of guile. I went to the source!

ZServer uses the standard asyncore library to handle its asynchronous sockets. Cursory investigation of the module yielded the very handy dictionary of all open connections:

(Pdb) asyncore.socket_map
{6: <ZServer.HTTPServer.zhttp_server listening 0.0.0.0:8080 at 0x5bbfa8>,
10: <ManagedClientConnection ('127.0.0.1', 8100)>,
11: <select-trigger (pipe) at 117f378>,
21: <ZServer.HTTPServer.zhttp_channel connected 127.0.0.1:55070
at 0x30a3148 channel#: 15 requests:>}

That last one was the socket I cared about. It disappeared at the right time, when the browser closed the connection. I just had to find some way for Zope itself to watch the open connections and stop streaming when the right one closed. This required me to correlate the REQUEST object to the connection, so it didn't stop working when somebody else closed their browser.

Luckily, REQUEST provided one piece of relevant information: the creation time of its channel. So all I had to do was check each of the open zhttp_channel connections for the same creation time; if none matched, the connection wasn't open anymore:

def is_browser_connection_open(request):
creation_time = request.environ['channel.creation_time']
for cnxn in asyncore.socket_map.values():
if (isinstance(cnxn, zhttp_channel) and
cnxn.creation_time==creation_time):
return True
return False

Then I just changed my streaming function to:

while is_browser_connection_open(REQUEST):

Worked like gangbusters. I've been sweating this issue for quite a while. Now, perhaps, I can sleep.

Read More...

RPM-style version comparison in Python

We've been going over some changes to our versioning scheme at Zenoss. Anything we choose has to provide a clear upgrade path on RHEL-based systems, which means we needed to check out how RPM compares two versions to see which is newer. Google yielded this description, which it says is deprecated, but I can't find anything newer. That led me to search for the comparison code itself, which I eventually tracked down. Eventually, I decided the most direct path to understanding involved porting it to a language in which I am moderately fluent, namely Python. You'll find the code at the bottom of this post.

As it turns out, RPM version comparison is pretty stupid. Letters are counted as older than numbers (1.2.3 > 1.2.a), which is good. Unfortunately, they count as a new segment as well, and things with more segments are newer. This means that 1.2.3a > 1.2.3 (because it splits into ('1', '2', '3', 'a') and ('1', '2', '3')). Thus there isn't a very good way to put descriptions of prereleases into an artifact name—at least, not if you want to be able to upgrade to the final.

After reading a bunch about this (including Wikipedia's thoughts on the subject, which were pretty enlightening), I recommended we go to an odd/even scheme: odd-numbered versions are unstable, even are stable. This would allow us to go from 1.2.3 (the development branch) to 1.2.3a (alpha release) to 1.2.3b to 1.2.3rc1 to 1.2.4 (final release), and RPM would know how to upgrade the whole way. Also, it's apparent from the name of the artifact how much trust one can place in it.

We'll see if it takes hold. If nothing else, we'll throw the build number in the artifact name, so 1.2.3-1234 will upgrade to 1.2.3-1250 just fine, except you'll have to know ahead of time that 1234 was the beta and 1250 the final.



import re
isalnum = re.compile('[^a-zA-Z0-9]')

def rpmvercmp(a, b):
# If they're the same, we're done
if a==b: return 0

def _gen_segments(val):
"""
Generator that splits a string into segments.
e.g., '2xFg33.+f.5' => ('2', 'xFg', '33', 'f', '5')
"""
val = isalnum.split(val)
for dot in val:
res = ''
for s in dot:
if not res:
res += s
elif (res.isdigit() and s.isdigit()) or \
(res.isalpha() and s.isalpha()):
res += s
else:
if res:
yield res
res = s
if res:
yield res

ver1, ver2 = a, b

# Get rid of the release number
ver1_rel, ver2_rel = None, None
if '-' in ver1: ver1, ver1_rel = ver1.rsplit('-')
if '-' in ver2: ver2, ver2_rel = ver2.rsplit('-')

l1, l2 = map(_gen_segments, (ver1, ver2))
while l1 and l2:
# Get the next segment; if none exists, done
try: s1 = l1.next()
except StopIteration: s1 = None
try: s2 = l2.next()
except StopIteration: s2 = None

if s1 is None and s2 is None: break
if (s1 and not s2): return 1
if (s2 and not s1): return -1

# Check for type mismatch
if s1.isdigit() and not s2.isdigit(): return 1
if s2.isdigit() and not s1.isdigit(): return -1

# Cast as ints if possible
if s1.isdigit(): s1 = int(s1)
if s2.isdigit(): s2 = int(s2)

rc = cmp(s1, s2)
if rc: return rc

# If we've gotten this far, check release numbers
if ver1_rel is not None and ver2_rel is not None:
return rpmvercmp(ver1_rel, ver2_rel)

return 0

Read More...

Chaining monkey patches in Python: partial legitimization of a venial sin

Monkey patching is a spectacular way to modify third-party code, but a terrible way to make friends (I here of course discount the pervasive problems coders have with socialization in general). Not only does it make issues notoriously difficult to track down, dirtying otherwise clean code with obscurity, but it's arrogant to boot. You've overridden, in perpetuity, any changes made to that third-party code that tread on the territory your monkey patch has conquered. More likely than not, you've rendered upgrades of the patchee untenable without breaking your code or their code or both.

Nonetheless, it's bloody useful, when one exerts a certain degree of control over the process. We use the technique in Zenoss quite a lot, in our ZenPack plugins, which often need to extend the functionality of the main product; since ZenPacks can be created by anyone with Zenoss and a dream, adding hooks into Core manually isn't an option, so ZenPacks have to monkey-patch in their features. It's fine, really. We only feel a little dirty. Less dirty since we started using a decorator!

def monkeypatch(target):
def patcher(func):
setattr(target, func.__name__, func)
return func
return patcher

import MyClass

@monkeypatch(MyClass)
def myfunc():
...

It's pretty trivial, and I'm sure there are myriad identical decorators out there; I think Guido even wrote one at some point. Nonetheless, it makes the process look much less ugly (which is, interestingly, one of the reasons anti-monkey-patching zealots reject such a decorator. They believe that such an ugly practice should look ugly, to deter usage. As if).

Today I was presented with a case where monkey-patching could actually be considered beneficial. I needed to validate manual addition of devices to Zenoss, to make sure they weren't put into a reserved device class. Whatever, no big deal, except it occurred to me that other ZenPacks were likely to override the function as well, which would knock out my validation (since monkey-patching involves replacing existing methods wholesale, he who monkey-patches last wins, and all other patches are discarded). Thus I needed some way to chain monkey patches.

This could, in fact, be considered something of a feature. If a few standard hooks were set up, meant to be monkey-patched an arbitrary number of times, aggregating results, ZenPacks could play nicely with each other—in this case, monkey patching is really the best way to do it.

It only took a few minutes to come up with the decorator:

def monkeypatch_reduce(target, reducefunc):
"""
A decorator factory that chains monkey patches, in the
case where separate patches may modify the same class.
"""
def patcher(func):
try:
_oldfunc = getattr(target, func.__name__)
except AttributeError:
setattr(target, func.__name__, func)
return func
def inner(*args, **kwargs):
return reducefunc(
_oldfunc(*args, **kwargs),
func(*args, **kwargs))
setattr(target, func.__name__, inner)
return inner
return patcher

With this decorator, you can monkey patch a class over and over, using any reduction function you want, and get the reduced result. For example, my validation function would use a reduction function that returns True if all criteria validate, and False otherwise:

from Products.ZenModel import Device

@monkeypatch_reduce(Device, reducefunc=lambda x,y:x and y)
def validate_device(dev):
return dev.fulfills_some_criteria()

@monkeypatch_reduce(Device, reducefunc=lambda x,y:x and y)
def validate_device(dev):
return dev.fulfills_other_criteria()

Any ZenPack can add their criteria to the validation process by monkey patching with this decorator, without worrying at all about anybody else's monkey patches.

One could also create a hook that would be monkey-patched to provide a kind of registry:

ZenPack.py:
-----------
def zenpack_list(self):
return []

A ZenPack:
----------
from Products.ZenModel import ZenPack
@monkeypatch_reduce(ZenPack, operator.add)
def zenpack_list(self):
return ["A Pack"]

A different ZenPack:
--------------------
from Products.ZenModel import ZenPack
@monkeypatch_reduce(ZenPack, operator.add)
def zenpack_list(self):
return ["A Different Pack"]


See? Each acts as if it's providing the only result, and the patch chain reduces them into a list:

>>> ZenPack.zenpack_list()
["A Pack", "A Different Pack"]

I dunno, maybe it's crazy. But I think it's got legs. I'm going to play around with it a bit more.

Read More...

cliutils 0.1.3 released: Persistence, home directories

This release of cliutils adds a few handy items. The new persistence module provides a function (storage_dir) that will, on Windows or *nix, find a directory where settings files should be stored (namely, the roaming Application Data directory on Windows, or the current user's home directory on *nix). Not too complex, but it removes a bit of overhead in locating the proper storage path. Also handy for creating directories; if the result doesn't exist, it'll be created.

The persistence module also provides a function that uses the inimitable shelve module to create a persistent dictionary, getting its path from storage_dir. This lets one create very simple databases on the fly for storing settings and whatnot, like:

>>> db('.myscript')
{}


Since the dictionary is saved whenever changed, it can be used just like a regular dictionary, only persistent. I've found this very useful.

Finally, the persistence module provides a simple dictionary-like interface to ConfigParser instances, again getting a path through storage_dir:

>>> import tempfile; filename = tempfile.mkstemp()[1]
>>> cfg = config(filename)
>>> cfg['sec1']['option2'] = 75
>>> cfg['sec2']['option1'] = "Some String"
>>> cfg['sec2']['option2'] = "Another value"
>>> f = file(filename)
>>> print f.read()
[sec1]
option2 = 75

[sec2]
option2 = Another value
option1 = Some String


Lastly, at the suggestion of a commenter, the logged decorator has been renamed to the more accurate redirect. Also, log_decorator has been deprecated, since it was really just a synonym of logged.

Download from Google Code or pypi. Upgrade or install anew with:

easy_install -U cliutils


API docs are here.

Read More...

cliutils 0.1.1 released: Working directory decorator factory

Came up with a handy decorator factory: @indir.  Say you want a shell script to execute in a different working directory—for example, you want to do complex config file manipulation, or batch operations on a lot of files, or you just don't want to pass absolute paths all the time. Just decorate the function in question with @indir(workdir), where workdir is the directory to which you want to change:

>>> import tempfile
>>> realpath = os.path.realpath
>>> new, cur = map(realpath, (tempfile.mkdtemp(), os.curdir))
>>> @indir(new)
... def whereami():
...     return realpath(os.curdir)
...
>> whereami() == new
True
>>> realpath(os.curdir) == cur
True

Notice that it'll change back to the original directory upon the completion of the function.

Download from Google Code or pypi.  Upgrade or install anew with easy_install -U cliutils. API docs are here.

Read More...

How-to: Simplify Python shell scripts with setuptools

A lot of people don't know how easy it is to use setuptools to create shell scripts. I didn't know how to use it properly until a couple of weeks ago; since discovering how, I've taken to repackaging all my Python scripts into an egg.

Here's how I was doing it before. I had modified PYTHONPATH to include ~/lib/python, and PATH to include ~/bin; this is where I would put scripts and modules I'd written. Since my home directory is checked into my Subversion repository, I could access these new scripts on remote boxes by making a user and checking them out.

This works fine, except for a few things. First, the scripts aren't usable by other users without their checking out my home directory. Second, modifying paths can be tricky. Third, it's a hassle setting everything up. Fourth, managing dependencies boils down to trusting that my home directory will be set up in the right way. Finally, using environment variables limits the use of the scripts to my own environment.

Enter setuptools. If I package my scripts up into eggs, I can install them as root and make them available to everyone, other users can install the egg in whatever way they see fit, and setuptools can handle all the dependencies and PYTHONPATH hackery for me. Once I learned that setuptools will actually create shell scripts for me pointing at functions I specify and install them in the correct place on the system, I was totally sold. This pattern was so useful for me that I even wrote and released a package making it easier.

Here's how I do it. Normally, I package related shell scripts together. In this example, we'll make an egg that installs a single shell script, which does nothing but return the titles of the HTML documents at the URLs passed in. I assure you that I recognize the total lack of utility this script provides.

  1. Create the egg structure. I used to do this by hand until I discovered a utility that does it for me. Install PasteScript:

    $ sudo easy_install PasteScript

    PasteScript provides the command paster, which, among other things, will create an empty egg for you to modify. Make that egg now:

    $ paster create -t basic_package TitleGetter

    It'll ask you a bunch of question that you can answer or not at your discretion. You can always change them later, and unless you're going to be releasing this to the world they're unncessary. At the end, you'll have a directory with an egg structure inside!

  2. Write the code. Naturally, this is the part you'll care about most once you've got the process down. For now, it's irrelevant. Here's some code that gets a web page and pulls out a title:

    import re, urllib2
    pattern = re.compile(r'<title[^<>]*>(.*)</title>', re.I|re.S)
    def print_titles(*urls):
    for url in urls:
    html = urllib2.urlopen(url).read()
    match = pattern.search(html)
    if match:
    title = match.groups()[0]
    else:
    title = "** Could not find a <title> **"
    print url, '\t', title

    Inside your egg is a directory called "titlegetter", and inside that is where you put your code. Make a file in there called titles.py and paste the above code into it.

  3. Test the code. Normally, I write unit tests before writing the code, placing them in a tests directory inside the package. That's up to you. Some people don't write tests. setuptools does allow you to run tests on an egg easily. I use nosetests, which you can install with:

    $ sudo easy_install nose

    You can then run them by going to the egg's top directory, where setup.py is, and running:

    $ python setup.py nosetests

    That'll build the egg in place and run the tests. Right now, though, we'll just test it manually. Install a development version of the egg:

    $ sudo python setup.py develop

    Now test your function:

    $ python
    >>> from titlegetter.titles import print_titles
    >>> print_titles('http://www.google.com')
    http://www.google.com Google


  4. Have setup.py create a shell script. This is the key bit of utility setuptools provides. Open up setup.py and find the entry_points definition. Change it to look like this:

    ...
    entry_points={
    "console_scripts": [
    'gettitle = titlegetter.titles:print_titles'
    ]},
    ...

    Now re-install the code with sudo python setup.py develop. setuptools will create a script called gettitle and install it somewhere on your PATH; that script will call titlegetter.titles.print_titles. Try it out:

    $ gettitle http://www.google.com http://www.yahoo.com
    $

    Notice nothing is printed out. Why not? Well, the script that setuptools creates doesn't do any argument parsing or anything, it just calls the function with no arguments. This makes sense, actually; if you're doing something fancy with args, you don't want to have to override the setuptools mechanism. On the other hand, for really simple scripts, you have to plop in a bunch of sys.argv parsing overhead.

    Luckily, I wrote a decorator that handles this for you. Add cliutils as a dependency by including the string in the install_requires definition in setup.py:

    ...
    install_requires = ['cliutils'],
    ...

    Then go down into titles.py and add the decorator:

    ...
    from cliutils import cliargs

    @cliargs
    def print_titles(*urls):
    ...

    That decorator will parse the command line arguments into args and keyword args and pass the result into the decorated function. Now that that's done, re-install your development egg with sudo python setup.py develop. You'll see that cliutils is downloaded and installed. Now try your script again:

    $ gettitle http://www.google.com http://www.yahoo.com
    http://www.google.com Google
    http://www.yahoo.com Yahoo!



And just like that, you've got a portable command-line utility. Check it in somewhere. If you want to build an actual egg, you can just do the usual setuptools thing and run:


$ python setup.py bdist_egg

Which will build and egg and dump it in the dist directory.

So that's that. I have two or three eggs that I keep modifying and updating with new functions. Ones that have the same dependencies or are generally related I put all in one egg, but organization is up to you. That cliutils package has a few other good things in it; read its documentation for more info.

Read More...

cliutils 0.1 released: Helpful utilities for Python shell scripts

Last weekend I had the idea to make a decorator that parsed sys.argv and passed the results as args to functions referenced by setuptools-created shell scripts. That turned out to be trivially easy. For some reason, I ended up writing a logging decorator, that redirects stdout to a given file-like object, and a much more ambitious Process object, which wraps subprocess.Popen objects and allows them to be piped into each other like in bash:

>>> p = Process("echo 'blah blah blah'") | Process("wc -w")
>>> print p.stdout
3


I packaged these up into an egg and released it in the Cheese Shop, so you can get it with:

$ easy_install cliutils


Code hosted at Google Code. And here are the API docs.

I'll be adding to this package as I come up with other tidbits that make shell scripts easier to write. After work today, provided I finish the project I'm working on, I plan to write a tutorial on packaging up one's own shell scripts in eggs for easier deployment and PYTHONPATH mungery. Setuptools really does make it incredibly easy.

Read More...