The Nomad blog is where we share news of our projects, thoughts on web development and a weekly round up of things on the Internet that have caught our eye - Tales from the Internet
Browse by Date
- April 2013
- November 2012
- October 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
Blog » Installing Xapian in virtualenv (django)
Installing Xapian in virtualenv (django)
So, we recently took on some amends and changes to an existing Django based website. When the time came for us to start development work, we had to first set up a local environment for it. This was easier said than done due to the lack of documentation left by the previous dev team. Additionally, the live server wasn't running the site inside of a virtualenv.
Virtualenv is the standard tool for creating isolated python environments. For example, this means that you can run a Django 1.2 site on the same server as a Django 1.4 site - and changes to one will not affect the other. Which is infinitely better than installing all your libraries server wide and acccidently breaking every site by upgrading one lib.
Whilst going through the trial and error of installing the correct packages (and correct versions) required for the site, we came across the need for the Haystack lib. Haystack is a library that allows for modular search in Django - specifically it plugs into a number of search backends such as Solr, Whoosh and for our purposes Xapian.
This required two things: Installing the xapian-haystack lib (easy via pip) and installing/compiling Xapian itself (the awkward bit).
Now, if we were installing this globally on our server, it would have been a simple case of using apt to install xapian from the repos. However, we are running inside a virtual environment remember. The python in our virtual environment is isolated from the system and therefore can't see our easily installed Xapian.
I tracked down the answer via this gist. Essentially, you need to install Xapian core and bindings manually into your virtualenv. A relatively simple solution. The gist works with an older version of xapian, so you may need to update a few lines to get it installing the version you want. I've broken down the various stages of the script below, so you can see what it's doing at each stage.
This script it designed to be run after you've activated your environment on the command line (so your shell is using the enviroment's libs) - this is acheived by running:
Where 'env' is your virtualenv folder.
Obligatory disclaimer: As ever, if you run this and it breaks the world, your computer/server or anything else, it's your own fault.
apt-get install zlib1g-dev apt-get install g++ export VENV=$VIRTUAL_ENV mkdir $VENV/packages && cd $VENV/packages curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-core-1.0.16.tar.gz curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-bindings-1.0.16.tar.gz tar xzvf xapian-core-1.0.16.tar.gz tar xzvf xapian-bindings-1.0.16.tar.gz cd $VENV/packages/xapian-core-1.0.16 ./configure --prefix=$VENV && make && make install export LD_LIBRARY_PATH=$VENV/lib cd $VENV/packages/xapian-bindings-1.0.16 ./configure --prefix=$VENV --with-python && make && make install python -c "import xapian"
Time to break the script down into bite sized chunks, so that we know what it's doing (always best with code you didn't write).
Firstly, we install the tools needed to compile Xapian.
apt-get install zlib1g-dev apt-get install g++
Next, we set up our path to the virtualenv (going to need this later), and make a few folders that are required.
export VENV=$VIRTUAL_ENV mkdir $VENV/packages && cd $VENV/packages
Now download and unpack the xapian core and bindings. You might want to change the URLs here to get the versions you require. Just remember to make sure the core and bindings packages match.
curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-core-1.0.16.tar.gz curl -O http://oligarchy.co.uk/xapian/1.0.16/xapian-bindings-1.0.16.tar.gz tar xzvf xapian-core-1.0.16.tar.gz tar xzvf xapian-bindings-1.0.16.tar.gz
Configure and build xapian core (change the paths if you changed URLs), using the custom virtualenv path we created earlier.
cd $VENV/packages/xapian-core-1.0.16 ./configure --prefix=$VENV && make && make install
This bit sets up an environment variable for later use - specifically the shared library path (so the binding package can find the core).
Now we configure and build the bindings with the custom path.
cd $VENV/packages/xapian-bindings-1.0.16 ./configure --prefix=$VENV --with-python && make && make install
Lastly, we test importing the xapian library by running a short command line program with python. This will show up with an error if the lib isn't there.
python -c "import xapian"
This should leave you with a working, isolated, xapian installation for your virtualenv. Remember you'll need to use pip to install django-haystack and xapian-haystack to use this with django.
As ever, if there is anything wrong, or anything that can be improved upon in this post, then please let us know in the comments.