Nomad

27 Jul 2012 by Robbie

Installing Xapian in virtualenv (django)

The problem

So, we recently took on some amends and changes to an existing Django based website. When the time came for us to start development work, we had to first set up a local environment for it. This was easier said than done due to the lack of documentation left by the previous dev team. Additionally, the live server wasn't running the site inside of a virtualenv.

Virtualenv is the standard tool for creating isolated python environments. For example, this means that you can run a Django 1.2 site on the same server as a Django 1.4 site - and changes to one will not affect the other. Which is infinitely better than installing all your libraries server wide and acccidently breaking every site by upgrading one lib.

Whilst going through the trial and error of installing the correct packages (and correct versions) required for the site, we came across the need for the Haystack lib. Haystack is a library that allows for modular search in Django - specifically it plugs into a number of search backends such as Solr, Whoosh and for our purposes Xapian.

This required two things: Installing the xapian-haystack lib (easy via pip) and installing/compiling Xapian itself (the awkward bit).

Now, if we were installing this globally on our server, it would have been a simple case of using apt to install xapian from the repos. However, we are running inside a virtual environment remember. The python in our virtual environment is isolated from the system and therefore can't see our easily installed Xapian.

I tracked down the answer via this gist. Essentially, you need to install Xapian core and bindings manually into your virtualenv. A relatively simple solution. The gist works with an older version of xapian, so you may need to update a few lines to get it installing the version you want. I've broken down the various stages of the script below, so you can see what it's doing at each stage.

This script it designed to be run after you've activated your environment on the command line (so your shell is using the enviroment's libs) - this is acheived by running:

source env/bin/activate

Where 'env' is your virtualenv folder.

The script

Obligatory disclaimer: As ever, if you run this and it breaks the world, your computer/server or anything else, it's your own fault.

apt-get install zlib1g-dev
apt-get install g++

export VENV=$VIRTUAL_ENV
mkdir $VENV/packages && cd $VENV/packages

curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-core-1.0.16.tar.gz
curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-bindings-1.0.16.tar.gz

tar xzvf xapian-core-1.0.16.tar.gz
tar xzvf xapian-bindings-1.0.16.tar.gz

cd $VENV/packages/xapian-core-1.0.16
./configure --prefix=$VENV && make && make install

export LD_LIBRARY_PATH=$VENV/lib

cd $VENV/packages/xapian-bindings-1.0.16
./configure --prefix=$VENV --with-python && make && make install

python -c "import xapian" 

The breakdown

Time to break the script down into bite sized chunks, so that we know what it's doing (always best with code you didn't write).

Firstly, we install the tools needed to compile Xapian.

apt-get install zlib1g-dev
apt-get install g++

Next, we set up our path to the virtualenv (going to need this later), and make a few folders that are required.

export VENV=$VIRTUAL_ENV
mkdir $VENV/packages && cd $VENV/packages

Now download and unpack the xapian core and bindings. You might want to change the URLs here to get the versions you require. Just remember to make sure the core and bindings packages match.

curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-core-1.0.16.tar.gz
curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-bindings-1.0.16.tar.gz

tar xzvf xapian-core-1.0.16.tar.gz
tar xzvf xapian-bindings-1.0.16.tar.gz

Configure and build xapian core (change the paths if you changed URLs), using the custom virtualenv path we created earlier.

cd $VENV/packages/xapian-core-1.0.16
./configure --prefix=$VENV && make && make install

This bit sets up an environment variable for later use - specifically the shared library path (so the binding package can find the core).

export LD_LIBRARY_PATH=$VENV/lib

Now we configure and build the bindings with the custom path.

cd $VENV/packages/xapian-bindings-1.0.16
./configure --prefix=$VENV --with-python && make && make install

Lastly, we test importing the xapian library by running a short command line program with python. This will show up with an error if the lib isn't there.

python -c "import xapian"

This should leave you with a working, isolated, xapian installation for your virtualenv. Remember you'll need to use pip to install django-haystack and xapian-haystack to use this with django.

As ever, if there is anything wrong, or anything that can be improved upon in this post, then please let us know in the comments.