How to replace virtualenv + pip by buildout

I was used to virtualenv and pip, and buildout first seems hard to understand. In fact, I thought that I didn't need buildout.

Then, with some practice, I came to think the opposite. I mean I no longer need virtualenv or pip. And buildout has strong features I really need.

Thus, I am wondering why so many people keep on using virtualenv and pip, and even often refuse to use buildout. One hypothesis is that buildout is hard to discover. During first steps with buildout, you don't understand what happens. And documentation about available options is a bit obscure.

So, here is an article about reproducing a virtualenv+pip workflow with buildout. Let's try it

 

Use case: project deployment workflow

Since we are about to compare two tools, let's use them in comparable situations. We will deploy a project:

So, let's compare what we have to write in our documentation (INSTALL procedure) and which commands will users (contributors) have to execute.

Note

I won't cover all aspects of buildout here. I will focus on reproducing virtualenv+pip workflow.

Virtualenv and pip

Preliminary notes:

  • Python is a prerequisite should be already installed.
  • We use Debian-based packages here, adapt it to your needs.
  • We use "sudo" to show which commands require superuser privileges.
# Let's work somewhere we own files
cd ~

# Begin deployment process...

# 1. Install virtualenv
sudo aptitude install virtualenv
# 2. Download project
git clone https://github.com/benoitbryon/python-from-virtualenv-and-pip-to-buildout from-virtualenv-and-pip
# 3. Create isolated environment directly within project
virtualenv --distribute --no-site-packages from-virtualenv-and-pip
# 4. Enter project
cd from-virtualenv-and-pip/
# 5. Install requirements
bin/pip install -r requirements.txt

# End deployment process. From now, project is installed, let's use it...

# Build documentation
cd docs
make html
cd ..

5 commands in the deployment process. Really simple, isn't it?

Additional notes:

  • We could have installed virtualenv with easy_install, but it is not the recommended way, since upgrades or uninstalls are not easy.
  • The --distribute option is the default in recent versions of virtualenv.
  • we didn't activate the virtual environment. It removes two steps from the procedure (activate and deactivate).

Let's do the same with buildout...

Buildout

Preliminary notes:

  • Python is a prerequisite should be already installed.
# Let's work somewhere we own files
cd ~

# Begin deployment process...

# 1. Download project
git clone https://github.com/benoitbryon/python-from-virtualenv-and-pip-to-buildout to-buildout
# 2. Enter project
cd to-buildout/
# 3. Install buildout
python bootstrap.py --distribute
# 4. Install requirements
bin/buildout

# End deployment process. From now, project is installed, let's use it...

# Build documentation
cd docs
make html
cd ..

4 commands in the deployment process...

Conclusion: buildout is simpler than virtualenv + pip

If you need a straight and simple INSTALL procedure, then I recommend buildout.

<Tom>  Hey! I already have virtualenv installed system wide, so I don't
       need to reinstall it!
<George>  Can you assert that all the contributors of your project (i.e. your
          team) have virtualenv already installed system wide?
<Tom>  I can't.
<George>  So you **must** tell them it is a prerequisite in your project's
          documentation.
<Tom>  Okay I'm adding a note about prerequisites in the documentation.
<George>  Then there is a risk that they can't install virtualenv because
          they have not sufficient privileges. Small but real.
<Tom>  Okay I'm adding a note about privileges in the documentation.
<George>  How do contributors know how to install virtualenv?
<Tom>  They search the web!
<George>  ...
<Tom>  Okay, I'm adding the command to install virtualenv.
<George>  On Debian, Fedora, Solaris or FreeBSD?
<Tom>  Wait, I can't maintain installation notes for all operating systems...
<George>  Is there a generic recipe to install virtualenv?
<Tom>  sudo easy_install virtualenv!
<George>  Is sudo another prerequisite?
<Tom>  Okay, I'm replacing the commands to install virtualenv by a link to
       http://pypi.python.org/pypi/virtualenv/.
<George>  That is a long documentation to read. If they didn't know about
          virtualenv, contributors have to read and learn about it. It could
          puzzle them, or at least take some time...
<Tom>  ...
<George>  With buildout, documentation would be lighter, i.e. simpler for a
          human to read and understand.
<George>  And we wouldn't have had this conversation...
<Tom>  Okay, let's try buildout... But I don't know how to create a project
       with buildout!
<George>  Read the notes below ;)

Use case: creating projects

Virtualenv + pip

Here is the synopsis:

  • Create an empty repository
  • Create a virtualenv in it
  • Pip install packages
  • Export requirements.txt with "pip freeze > requirements.txt"
  • Ignore virtualenv generated files in repository
  • Commit and push

Buildout

Here is the synopsis:

  • Create an empty repository
  • Download buildout's bootstrap.py
  • Create buildout.cfg configuration file
  • Install buildout with "python bootstrap.py"
  • Deploy project with "bin/buildout"
  • Ignore buildout generated files in repository
  • Commit and push

Let's explain some steps...

Download buildout's bootstrap.py

Download buildout's bootstrap.py manually or create a script with the command like this:

wget http://svn.zope.org/*checkout*/zc.buildout/tags/1.5.2/bootstrap/bootstrap.py?content-type=text%2Fplain -O bootstrap.py

Create buildout.cfg configuration file

Here is a simple buildout.cfg file which reproduces virtualenv + pip requirements to install sphinx, nose and a python interpreter.

# Sample buildout configuration file to install sphinx and nose.

[buildout]
parts =
    sphinx-and-python
eggs =
    sphinx
    nose
versions = versions

[sphinx-and-python]
recipe = zc.recipe.egg
eggs = ${buildout:eggs}
interpreter = python

[versions]
sphinx = 1.1.2
nose = 1.1.2

Yes, pip's requirements files are simpler. But, once you learned the basics of buildout, you see that buildout's configuration files are not so complex:

  • As any ConfigParser file, this one is made of sections.
  • [buildout] is the main buildout section:
    • The "parts" directive lists the sections to execute.
    • If "parts" is blank, then buildout will do nothing.
    • The "eggs" directive is a list of eggs (i.e. python dependencies). It does nothing "magic" by itself. But it can be referenced by other parts via the ${buildout:eggs} variable (see the [sphinx-and-python] part). So, consider it as a set of eggs that **can** be shared between several (none to all) parts.
    • The "versions" directive indicates the part where you freeze (pin) versions.
  • [sphinx-and-python] part installs a bin/python interpreter with eggs declared in the main [buildout] section.
  • [versions] section contains some version numbers frozen manually. Keep in mind that extensions like buildout-versions or dumppickedversions can help you generate it!
  • Notice that it is recommended to add comments in buildout configuration files.

Ignore buildout generated files in revisions

Here is a .gitignore file I use in some Python projects:

# Buildout's directories
/bin
/develop-eggs
/eggs
/parts
/.installed.cfg
/downloads
/lib

# Setuptools/Distribute's files
*.egg-info
/dist
/build

# Python's precompiled files
*.pyc
*.pyo

# Sphinx's builds
/docs/_build

Additional tips

Some important buildout options

I recommend reading the buildout help at bin/buildout -h. Pay attention at least to the following options:

bin/buildout -N
Run buildout in "non newest" mode. Use it if you don't want to upgrade installed packages. In fact, it is recommended to always use it, except when you explicitely want to upgrade packages.
bin/buildout -vvvvvvv
Many "v" means more verbose... If something goes wrong, messages may be useful.
bin/buildout -c path/to/buildout.cfg
The location of your buildout configuration file. It is "buildout.cfg" by default. Notice that, by default, buildout's directories (eggs, parts...) are relative to configuration file.

Developing Python modules

  • Create eggs (this is also a best pratice with virtualenv and pip). Have a look at PasteScript and ZopeSkel for this.
  • If your project is the egg (i.e. you want an alternative to "setup.py develop"), search the web for the "develop" option in the [buildout] section.
  • If you want to edit external modules (i.e. you want an alternative to "pip install -e") have a look at mr.developer extension.
Votre notation : Aucun Moyenne : 5 (8 votes)

Les sous-répertoires de buildout ?

Avec virtualenv, j'ai :

  mon-projet/
     env/
     src/ 
        .git/
        README

Quand je commence à travailler, je fais juste :

cd mon-projet/src
../env/bin/activate
pip install -r requirements.txt

Avec buildout, j'ai toujours l'impression qu'il m'en fiche partout, avec 7 répertoires. Peut-on cloisonner dans un seul répertoire ? Comme virtualenv ?

Re: Les sous-répertoires de buildout ?

Peut-on cloisonner dans un seul répertoire ? Comme virtualenv ?

La réponse est oui, en jouant avec les options "directory". Ça pourrait faire l'objet d'un autre article. D'ici là, le fait que buildout crée des répertoires à la racine du projet n'est aucunement bloquant pour le travail au quotidien.