Seeing as how I bring Zenoss processes up and down forty times a day, I can get confused about what's running, and zenoss status gives me a bunch of stuff I don't care about. So I wrote this bash function to output a simple string representing my running daemons.

Read More...

bash 4.0 was released last month, and I'm only now getting around to messing with it. So far the things I use most are the '**' recursive globbing and the 'autocd' option. Here's how I upgraded my system and a few ways I use the new features.

1. Download, build and install. Get the source tarball:
   

   $ wget ftp://ftp.cwru.edu/pub/bash/bash-4.0.tar.gz

   
   Build and install:
   

   $ tar xzf bash-4.0.tar.gz
   $ cd bash-4.0
   $ ./configure && make && sudo make install

   
   On my system (Leopard + Xcode) I had no trouble compiling. The binary is installed at /usr/local/bin/bash. You can also install via MacPorts or Fink, I imagine, but I try to do these things myself when possible, if only for the sake of transparency.


2. Configure to use the new shell. First, register the new binary as a valid shell:
   

   $ sudo bash -c "echo /usr/local/bin/bash >> /private/etc/shells"

   
   Then change your user to use it as its shell with chsh (this is blatantly obvious, but modify the "Shell" line to point to /usr/local/bin/bash).
   Now open a new shell (restarting Terminal.app will do, or just Cmd-N for a new window) and make sure your changes took:
   

   [ian@iansmbp] ~/> echo $SHELL
   /usr/local/bin/bash
   [ian@iansmbp] ~/> echo $BASH_VERSION
   4.0.0(1)-release



3. Enable the new features. There's a bunch of features in the 4.0 release that I haven't gone through yet, including coproc, improved programmable completion, a new &>> redirect operator (synonym of the old >>myfile 2>&1 pattern) and new case-insensitive expansion options. You can peruse a list of the new stuff at your leisure. For immediate gratification, however, just turn on recursive globbing and autocd:
   

   $ echo "shopt -s globstar autocd" >> ~/.profile
   $ source ~/.profile

   
   ** matches contents recursively. For example, where you might previously have recursively removed all your byte-compiled Python modules with:
   

   $ find . -name \*.pyc | xargs rm -f

   
   You can now simply:
   

   $ rm -f **/*.pyc

   
   If you've got a deep directory structure, and you want to get into a subdirectory whose name you know, you can replace:
   

   $ cd path/to/the/directory/i/want/named/mydir

   
   with:
   

   $ cd **/mydir

   
   Hey, and if those extra three characters at the beginning of that last command are too much for you, then you'll love the autocd option, which, when enabled, permits you to cd to a directory merely by typing its name:
   

   [ian@iansmbp] ~/src/zenoss/core/Products/> **/yui
   cd ZenWidgets/skins/zenui/yui
   [ian@iansmbp] ~/src/zenoss/core/Products/ZenWidgets/skins/zenui/yui/>

   
   

More on new features as warranted; these two, however, are those of zsh that I missed most.

Read More...

I've been a little office-upgrade jag the past couple of days; Monday I ordered a Mirra, and today I set up a third monitor on my desk:



It's driven by my previously headless Ubuntu server; I use Synergy to share input devices. I'm using it mostly for things I want visible constantly—Skype, Pidgin, mail, iTunes—which yesterday consumed a dedicated virtual desktop using Spaces.app. If OmniFocus weren't OSX only it'd be there too.

Even with the annex, space is at a premium. I typically require two browser windows, at least two terminal windows, and the triptych MacVim to code effectively (at least, when focusing on UI work as I am now). This morning, I decided that Twitter client (I vacillate between Twhirl and TweetDeck) took up a tract of pixelage too juicy to ignore.

A quickly aborted foray into writing my own barebones plaintext Twitter client, followed by a more sober Google for such an obviously extant project, led me to Python Twitter Tools, exactly what I was looking for. Its basic usage is simple:

ian@iansmbp:~$ twitter
samuelmcc I have to revise an essay and read a 150 page book by 2pm tomorrow.
hb123 Now organizing my Google contacts (synced with iPhone) as a procrastination device.
pollack Every day loving New Mexican food, but just realized I also miss red beans & rice in Jackson.
marcambinder For 2010, O wants to cap non-defense disc. spending increases by .5 ... Federal workers get one percent raises.
hb123 @pollack I definitely didn't see them and I grew up in a 90% Catholic town. They all went in the evening, though.
marcambinder The significance of not being a tv correspondent, a newspaper writer or a lifestyle mag reporter dawns on me when i try to deal with the ...
chezaristote @neagle @hb123 Pretty much everything we said about J. Stewart as an ill-fitting Oscar host was recapped by John Oliver on Monday's D.Show.
pollack @hb123 So what is that a sign of european death?
chezaristote You know, I might be attracted to the Kindle if it showed me two pages at once. And would let me write notes in the margins. But maybe not.
KatherineD RT @bianconeri4ever: Saudi Arabia Starts Blocking Sections on @Digg!  http://bit.ly/12B5de inexcusable! #Digg #Saudi #Freedom #netneutra ...

You can also ask for replies, DMs, etc. Posting to Twitter is equally simple:

ian@iansmbp:~$ twitter set My quippy one-liners are matchless!

The only gotcha I've run across so far is the necessity to escape apostrophes, but that comes naturally in a command-line environment.

Installation is trivial, thanks to the Cheese Shop:

ian@iansmbp:~$ sudo easy_install twitter

Also comes with an IRC bot. The man page does a decent job of outlining things.

So, if, like me, you decide that Twitter's not the kind of thing that necessitates a dedicated chunk of screen, this is a pretty good way to go. I've got a few enhancement ideas I may code up and see if they want. More on that if and when it happens.

Read More...

For some reason, a lot of people have issues setting up ssh key authentication on remote boxes; sometimes it's cut-and-paste problems, sometimes mistakes editing the file manually. Also some aren't aware that you don't need to push your pubkey file up to the box, then ssh in and copy/paste the contents into authorized_keys. You can do it all in one go:

$ cat ~/.ssh/id_dsa.pub | ssh user@remotebox \
    "cat - >> ~/.ssh/authorized_keys"

A minor improvement to a small problem, perhaps, but it removes a bunch of failure-prone steps.

Update: I wrote a function that pushes your key out to the remote box, then modifies your ~/.ssh/config to use the specified username:

setupssh () {
    USER=${1%@*}
    BOX=${1#*@}
    if [ "$USER" = "$1" ]; then
        USER=`whoami`
    else
        # Set up user
        echo "
Host $BOX
   User $USER" >> $HOME/.ssh/config
    fi  
    # Install the key
    cat ~/.ssh/id_dsa.pub | ssh $USER@$BOX "cat - >> ~/.ssh/authorized_keys"
}

Shove that in your ~/.bashrc, source it, and type:

$ setupssh remoteuser@my.remotebox.com

You'll have to enter remoteuser's password once when it pushes out your pubkey. From then on, you can just ssh my.remotebox.com, no passwords.

Read More...

I don't know how long ack has been around, but I only discovered it yesterday (via $tail -f findings.out). It does basically the same thing as the eternal paradigm:

$ find . -name \*.py | grep -v '.svn' | xargs grep 'my search string'

...only more maneuverable, and wrapped up in a shell script, plus highlighting and other good bits. It has a bunch of options I haven't really explored yet. It's looking like I can retire my collection of aliases (pyfind, jsfind, etc.).
Remember: no real man uses an IDE.

Read More...

A lot of people don't know how easy it is to use setuptools to create shell scripts. I didn't know how to use it properly until a couple of weeks ago; since discovering how, I've taken to repackaging all my Python scripts into an egg.

Here's how I was doing it before. I had modified PYTHONPATH to include ~/lib/python, and PATH to include ~/bin; this is where I would put scripts and modules I'd written. Since my home directory is checked into my Subversion repository, I could access these new scripts on remote boxes by making a user and checking them out.

This works fine, except for a few things. First, the scripts aren't usable by other users without their checking out my home directory. Second, modifying paths can be tricky. Third, it's a hassle setting everything up. Fourth, managing dependencies boils down to trusting that my home directory will be set up in the right way. Finally, using environment variables limits the use of the scripts to my own environment.

Enter setuptools. If I package my scripts up into eggs, I can install them as root and make them available to everyone, other users can install the egg in whatever way they see fit, and setuptools can handle all the dependencies and PYTHONPATH hackery for me. Once I learned that setuptools will actually create shell scripts for me pointing at functions I specify and install them in the correct place on the system, I was totally sold. This pattern was so useful for me that I even wrote and released a package making it easier.

Here's how I do it. Normally, I package related shell scripts together. In this example, we'll make an egg that installs a single shell script, which does nothing but return the titles of the HTML documents at the URLs passed in. I assure you that I recognize the total lack of utility this script provides.

1. Create the egg structure. I used to do this by hand until I discovered a utility that does it for me. Install PasteScript:
   

   
   $ sudo easy_install PasteScript
   

   
   PasteScript provides the command paster, which, among other things, will create an empty egg for you to modify. Make that egg now:
   

   
   $ paster create -t basic_package TitleGetter
   

   
   It'll ask you a bunch of question that you can answer or not at your discretion. You can always change them later, and unless you're going to be releasing this to the world they're unncessary. At the end, you'll have a directory with an egg structure inside!
   


2. Write the code. Naturally, this is the part you'll care about most once you've got the process down. For now, it's irrelevant. Here's some code that gets a web page and pulls out a title:
   

   
   import re, urllib2
   pattern = re.compile(r'<title[^<>]*>(.*)</title>', re.I|re.S)
   def print_titles(*urls):
       for url in urls:
           html = urllib2.urlopen(url).read()
           match = pattern.search(html)
           if match:
               title = match.groups()[0]
           else:
               title = "** Could not find a <title> **"
           print url, '\t', title
   

   
   Inside your egg is a directory called "titlegetter", and inside that is where you put your code. Make a file in there called titles.py and paste the above code into it.
   


3. Test the code. Normally, I write unit tests before writing the code, placing them in a tests directory inside the package. That's up to you. Some people don't write tests. setuptools does allow you to run tests on an egg easily. I use nosetests, which you can install with:
   

   
   $ sudo easy_install nose
   

   
   You can then run them by going to the egg's top directory, where setup.py is, and running:
   

   
   $ python setup.py nosetests
   

   
   That'll build the egg in place and run the tests. Right now, though, we'll just test it manually. Install a development version of the egg:
   

   
   $ sudo python setup.py develop
   

   
   Now test your function:
   

   
   $ python
   >>> from titlegetter.titles import print_titles
   >>> print_titles('http://www.google.com')
   http://www.google.com Google
   

   
   


4. Have setup.py create a shell script. This is the key bit of utility setuptools provides. Open up setup.py and find the entry_points definition. Change it to look like this:
   

   
   ...
   entry_points={
       "console_scripts": [
           'gettitle = titlegetter.titles:print_titles'
       ]},
   ...
   

   
   Now re-install the code with sudo python setup.py develop. setuptools will create a script called gettitle and install it somewhere on your PATH; that script will call titlegetter.titles.print_titles. Try it out:
   

   
   $ gettitle http://www.google.com http://www.yahoo.com
   $
   

   
   Notice nothing is printed out. Why not? Well, the script that setuptools creates doesn't do any argument parsing or anything, it just calls the function with no arguments. This makes sense, actually; if you're doing something fancy with args, you don't want to have to override the setuptools mechanism. On the other hand, for really simple scripts, you have to plop in a bunch of sys.argv parsing overhead.
   
   Luckily, I wrote a decorator that handles this for you. Add cliutils as a dependency by including the string in the install_requires definition in setup.py:
   

   
   ...
   install_requires = ['cliutils'],
   ...
   

   
   Then go down into titles.py and add the decorator:
   

   
   ...
   from cliutils import cliargs
   
   @cliargs
   def print_titles(*urls):
   ...
   

   
   That decorator will parse the command line arguments into args and keyword args and pass the result into the decorated function. Now that that's done, re-install your development egg with sudo python setup.py develop. You'll see that cliutils is downloaded and installed. Now try your script again:
   

   
   $ gettitle http://www.google.com http://www.yahoo.com
   http://www.google.com Google
   http://www.yahoo.com Yahoo!
   

   
   

And just like that, you've got a portable command-line utility. Check it in somewhere. If you want to build an actual egg, you can just do the usual setuptools thing and run:

$ python setup.py bdist_egg

Which will build and egg and dump it in the dist directory.

So that's that. I have two or three eggs that I keep modifying and updating with new functions. Ones that have the same dependencies or are generally related I put all in one egg, but organization is up to you. That cliutils package has a few other good things in it; read its documentation for more info.

Read More...

Here at Zenoss, we had a need to run Python in a modified environment. A script was written (by a third party) wrapping the python binary, and became `which python`:

#!/bin/sh
. /path/to/setenv.sh
/path/to/.python.bin "$@"

This replaced the binary as the path to the interpreter.

Then we tested it. Most of the scripts in $ZENHOME/bin are shell scripts that call Python scripts elsewhere, but a few, including most of the ones laid down by Zope during the mkzopeinstance installation step, are straight up Python scripts with the shebang #!/usr/local/zenoss/bin/python ($ZENHOME in this case is /usr/local/zenoss; distutils does a string replace of shebang lines during the install-scripts step). When we tried to execute these scripts with the new Python wrapper in place, we got things like:

runtests: line 3: import: command not found<br />from: can't read /var/mail/subprocess<br />runtests: line 5: import: command not found<br />runtests: line 6: import: command not found<br />runtests: line 8: ZENHOME: command not found<br />runtests: line 9: syntax error near unexpected token `('<br />runtests: line 9: `ZOPEHOME = os.environ.get('ZOPEHOME', ZENHOME)'

/bin/sh was trying to interpret the script. When we executed the shebang on the command line, we were thrown into the interpreter, so it wasn't a simple path problem. When we asked around, one of our more experienced developers knew the cause immediately: the shebang line has to point to a binary.

We figured we were pretty much sunk; among other things, the setenv.sh script set the LD_LIBRARY_PATH against which that Python binary had been compiled (the idea was that the binary could be portable that way), so we couldn't call it directly and set environment variables via Python, for example. Eventually we came up with a hacky but totally workable solution:

#!/usr/bin/env /path/to/my/wrapper/python

env is a command meant to interact with environment variables and run other commands in modified environments. If you don't give it a modified environment, it'll use the current one, which makes it basically a pass-through; on the command line, env python and python are synonymous. /usr/bin/env is, needless to say, a binary file.

I don't know why the shebang line has this limitation; presumably it's some security issue. But there's the solution, if you absolutely have to circumvent it: call your shell script through env.

Read More...

I didn't learn about svn switch until recently. Turns out it's pretty handy! Basically, it's a subset of svn up that switches your working copy to a different URL. So if you're in the trunk, and you want to work in a branch but don't want to check out all the code fresh, you can just:

svn switch http://repo/branch/url .

And your working copy will update to the HEAD of the branch you specify (or, if you like, you can specify a revision with the usual -r option.

Example: Zenoss's build system does an svn checkout of the source trunk/branch/tag as an initial step. In order to save time (rpm builds took an hour), we trimmed down tasks that could be avoided; among other things, instead of checking out the code fresh each time, we would just svn update if the working copy already existed.

This worked great, until I noticed the other day that builds would occur against the trunk even if we were trying to build a release against a tag. The reason for this, of course, was that the working directory already existed, from our nightly rpm builds, so specifying a tag didn't matter. The build script would just svn up the trunk.

I made a quick change. svn up /my/working/copy became svn switch http://dev.zenoss.org/svn/path/to/branch /my/working/copy. If the working copy has the same URL as the one specified, it's a synonym of svn up; otherwise, we switch to the tag. Now we get the time-saving benefits of reusing already-checked-out code, while still building off of our branch of choice.

Read More...