How to replace virtualenv + pip by buildout
I was used to virtualenv and pip, and buildout first seems hard to understand. In fact, I thought that I didn't need buildout.
Then, with some practice, I came to think the opposite. I mean I no longer need virtualenv or pip. And buildout has strong features I really need.
Thus, I am wondering why so many people keep on using virtualenv and pip, and even often refuse to use buildout. One hypothesis is that buildout is hard to discover. During first steps with buildout, you don't understand what happens. And documentation about available options is a bit obscure.
So, here is an article about reproducing a virtualenv+pip workflow with buildout. Let's try it
Use case: project deployment workflow
Since we are about to compare two tools, let's use them in comparable situations. We will deploy a project:
- See https://github.com/benoitbryon/python-from-virtualenv-and-pip-to-buildout
- This project contains a sphinx documentation.
- We want to download the project, install dependencies (sphinx and nose) then build the documentation to html.
So, let's compare what we have to write in our documentation (INSTALL procedure) and which commands will users (contributors) have to execute.
Note
I won't cover all aspects of buildout here. I will focus on reproducing virtualenv+pip workflow.
Virtualenv and pip
Preliminary notes:
- Python is a prerequisite should be already installed.
- We use Debian-based packages here, adapt it to your needs.
- We use "sudo" to show which commands require superuser privileges.
# Let's work somewhere we own files cd ~ # Begin deployment process... # 1. Install virtualenv sudo aptitude install virtualenv # 2. Download project git clone https://github.com/benoitbryon/python-from-virtualenv-and-pip-to-buildout from-virtualenv-and-pip # 3. Create isolated environment directly within project virtualenv --distribute --no-site-packages from-virtualenv-and-pip # 4. Enter project cd from-virtualenv-and-pip/ # 5. Install requirements bin/pip install -r requirements.txt # End deployment process. From now, project is installed, let's use it... # Build documentation cd docs make html cd ..
5 commands in the deployment process. Really simple, isn't it?
Additional notes:
- We could have installed virtualenv with easy_install, but it is not the recommended way, since upgrades or uninstalls are not easy.
- The --distribute option is the default in recent versions of virtualenv.
- we didn't activate the virtual environment. It removes two steps from the procedure (activate and deactivate).
Let's do the same with buildout...
Buildout
Preliminary notes:
- Python is a prerequisite should be already installed.
# Let's work somewhere we own files cd ~ # Begin deployment process... # 1. Download project git clone https://github.com/benoitbryon/python-from-virtualenv-and-pip-to-buildout to-buildout # 2. Enter project cd to-buildout/ # 3. Install buildout python bootstrap.py --distribute # 4. Install requirements bin/buildout # End deployment process. From now, project is installed, let's use it... # Build documentation cd docs make html cd ..
4 commands in the deployment process...
Conclusion: buildout is simpler than virtualenv + pip
If you need a straight and simple INSTALL procedure, then I recommend buildout.
<Tom> Hey! I already have virtualenv installed system wide, so I don't need to reinstall it! <George> Can you assert that all the contributors of your project (i.e. your team) have virtualenv already installed system wide? <Tom> I can't. <George> So you **must** tell them it is a prerequisite in your project's documentation. <Tom> Okay I'm adding a note about prerequisites in the documentation. <George> Then there is a risk that they can't install virtualenv because they have not sufficient privileges. Small but real. <Tom> Okay I'm adding a note about privileges in the documentation. <George> How do contributors know how to install virtualenv? <Tom> They search the web! <George> ... <Tom> Okay, I'm adding the command to install virtualenv. <George> On Debian, Fedora, Solaris or FreeBSD? <Tom> Wait, I can't maintain installation notes for all operating systems... <George> Is there a generic recipe to install virtualenv? <Tom> sudo easy_install virtualenv! <George> Is sudo another prerequisite? <Tom> Okay, I'm replacing the commands to install virtualenv by a link to http://pypi.python.org/pypi/virtualenv/. <George> That is a long documentation to read. If they didn't know about virtualenv, contributors have to read and learn about it. It could puzzle them, or at least take some time... <Tom> ... <George> With buildout, documentation would be lighter, i.e. simpler for a human to read and understand. <George> And we wouldn't have had this conversation... <Tom> Okay, let's try buildout... But I don't know how to create a project with buildout! <George> Read the notes below ;)
Use case: creating projects
Virtualenv + pip
Here is the synopsis:
- Create an empty repository
- Create a virtualenv in it
- Pip install packages
- Export requirements.txt with "pip freeze > requirements.txt"
- Ignore virtualenv generated files in repository
- Commit and push
Buildout
Here is the synopsis:
- Create an empty repository
- Download buildout's bootstrap.py
- Create buildout.cfg configuration file
- Install buildout with "python bootstrap.py"
- Deploy project with "bin/buildout"
- Ignore buildout generated files in repository
- Commit and push
Let's explain some steps...
Download buildout's bootstrap.py
Download buildout's bootstrap.py manually or create a script with the command like this:
wget http://svn.zope.org/*checkout*/zc.buildout/tags/1.5.2/bootstrap/bootstrap.py?content-type=text%2Fplain -O bootstrap.py
Create buildout.cfg configuration file
Here is a simple buildout.cfg file which reproduces virtualenv + pip requirements to install sphinx, nose and a python interpreter.
# Sample buildout configuration file to install sphinx and nose. [buildout] parts = sphinx-and-python eggs = sphinx nose versions = versions [sphinx-and-python] recipe = zc.recipe.egg eggs = ${buildout:eggs} interpreter = python [versions] sphinx = 1.1.2 nose = 1.1.2
Yes, pip's requirements files are simpler. But, once you learned the basics of buildout, you see that buildout's configuration files are not so complex:
- As any ConfigParser file, this one is made of sections.
- [buildout] is the main buildout section:
- The "parts" directive lists the sections to execute.
- If "parts" is blank, then buildout will do nothing.
- The "eggs" directive is a list of eggs (i.e. python dependencies). It does nothing "magic" by itself. But it can be referenced by other parts via the ${buildout:eggs} variable (see the [sphinx-and-python] part). So, consider it as a set of eggs that **can** be shared between several (none to all) parts.
- The "versions" directive indicates the part where you freeze (pin) versions.
- [sphinx-and-python] part installs a bin/python interpreter with eggs declared in the main [buildout] section.
- [versions] section contains some version numbers frozen manually. Keep in mind that extensions like buildout-versions or dumppickedversions can help you generate it!
- Notice that it is recommended to add comments in buildout configuration files.
Ignore buildout generated files in revisions
Here is a .gitignore file I use in some Python projects:
# Buildout's directories /bin /develop-eggs /eggs /parts /.installed.cfg /downloads /lib # Setuptools/Distribute's files *.egg-info /dist /build # Python's precompiled files *.pyc *.pyo # Sphinx's builds /docs/_build
Additional tips
Some important buildout options
I recommend reading the buildout help at bin/buildout -h. Pay attention at least to the following options:
- bin/buildout -N
- Run buildout in "non newest" mode. Use it if you don't want to upgrade installed packages. In fact, it is recommended to always use it, except when you explicitely want to upgrade packages.
- bin/buildout -vvvvvvv
- Many "v" means more verbose... If something goes wrong, messages may be useful.
- bin/buildout -c path/to/buildout.cfg
- The location of your buildout configuration file. It is "buildout.cfg" by default. Notice that, by default, buildout's directories (eggs, parts...) are relative to configuration file.
Developing Python modules
- Create eggs (this is also a best pratice with virtualenv and pip). Have a look at PasteScript and ZopeSkel for this.
- If your project is the egg (i.e. you want an alternative to "setup.py develop"), search the web for the "develop" option in the [buildout] section.
- If you want to edit external modules (i.e. you want an alternative to "pip install -e") have a look at mr.developer extension.
Les sous-répertoires de buildout ?
Avec virtualenv, j'ai :
mon-projet/ env/ src/ .git/ READMEQuand je commence à travailler, je fais juste :
Avec buildout, j'ai toujours l'impression qu'il m'en fiche partout, avec 7 répertoires. Peut-on cloisonner dans un seul répertoire ? Comme virtualenv ?
Re: Les sous-répertoires de buildout ?
La réponse est oui, en jouant avec les options "directory". Ça pourrait faire l'objet d'un autre article. D'ici là, le fait que buildout crée des répertoires à la racine du projet n'est aucunement bloquant pour le travail au quotidien.