"…mais ce serait peut-être l'une des plus grandes opportunités manquées de notre époque si le logiciel libre ne libérait rien d'autre que du code…"

Archive for the ‘package_management’ Category

Python package management

Python packaging : Hitchhiker guide, toydist, envbuilder

Posted by patrick sur avril 22, 2010

Toujours beaucoup de mouvement en ce qui concerne la distribution de modules python.

  • http://cournape.wordpress.com/2010/04/22/first-public-release-of-toydist/ (‘The goals of toydist are simplicity and extensibility. There should be only one way to package simple packages (ideally, relying only on the toysetup.info file), while being flexible enough to handle complex softwares. The ultimate goal of toydist is to replace the hideous distutils extensions to build NumPy and SciPy. ‘)
  • http://pypi.python.org/pypi/envbuilder/0.3.0 (‘ A package for automatic generation of virtualenvs Envbuilder is a system for automatically building virtualenvs in Python. To do this, it uses a .env config file to define parcels, which are individual pieces of software to be checked out and installed into the virtualenv. It is mainly tested under Linux, but it should work under any platform that supports unix-y commands (including cygwin). In fact, you might even be able to make one config file work on both Windows and *nix if you’re careful.’)

Posted in 2010, Développement logiciel, distribute, Distribution de logiciel, Doc_sphinx, Génie logiciel, package_management, packaging, toydist | Leave a Comment »

Python : Distribute 0.6.8 (http://python-distribute.org/) , pip 0.6 , virtualenv 1.4.1 : make the transition today :)

Posted by patrick sur novembre 10, 2009

Source: http://s3.pixane.com/pip_distribute.png

distribute, pip virtualenv
curl  -0 http://python-distribute.org/distribute_setup.py > distribute_setup.py
sudo python distribute_setup.py
sudo easy_install -U pip

http://pypi.python.org/pypi/pip/0.6 (‘pip is a replacement for easy_install. It uses mostly the same techniques for finding packages, so packages that were made easy_installable should be pip-installable as well…The main website for pip is pip.openplans.org. You can also install the in-development version of pip with easy_install pip==dev. ‘)

http://pypi.python.org/pypi/virtualenv/1.4.2 (‘virtualenv is a successor to workingenv, and an extension of virtual-python.’)

More informations

source: http://mail.python.org/pipermail/distutils-sig/2009-November/014229.html 

On behalf of the Distribute team, I am pleased to announce the 0.6.7
release of Distribute.

As usual, availabe at PyPI: http://pypi.python.org/pypi/distribute

Most noticeable changes in 0.6.7 are:

- now the develop command supports the --user option, so it can use
the per-user site packages (PEP 370)
- the generated scripts now wrap their call to the script entry point
in the standard "if name == 'main'"
- better errors handling in PackageIndex when files and pages are
visited by easy_install
- a virtualenv-compatible version, so the next virtualenv release will
be able to provide a --distribute option.

You can visit http://pypi.python.org/pypi/distribute#id2 for a full

We are now starting the 0.6.8 work in parallel of 0.7.x development,
with more bugfixes coming up.


This is just a placeholder for bootstrap files for Distribute, and nightly builds for Distutils

Useful links:

http://faassen.n–tree.net/blog/view/weblog/2009/11/09/0 (‘A history of Python packaging’)


guido van rossum

Posted in 2009, distribute, package_management, python | Tagué: , | Leave a Comment »

Python package management: ‘Distribute’ le fork de ‘setuptools’ est sur pypi

Posted by patrick sur août 9, 2009

Grâce au travail de Tarek Ziadé (http://twitter.com/tarek_ziade, http://ziade.org/blog)  le fork amical de setuptools est sorti en version 0.6 le dimanche 9 août 2009:

  • http://pypi.python.org/pypi/distribute (’Download, build, install, upgrade, and uninstall Python packages — easily! Distribute is a friendly fork of the Setuptools project. Distribute is a friendly fork of the Setuptools project. )
    • http://bitbucket.org/tarek/distribute/wiki/Home
    • http://tarekziade.wordpress.com/2009/07/22/preparing-to-release-distribute-0-6/ (’According to the poll, The name of the fork will be Distribute ! The code should not be changed anymore at this point, and I am working on the bootstraping so installing Distribute will work with an existing Setuptools installation and will replace it for the applications that requires it. This is done by detecting an installation of Setuptools, and replacing it with a fake installation. This means that once you’ve installed Distribute, applications and especially installers will think that setuptools 0.6c9 is installed.That’s pretty strong and intrusive, but required for a simple switch : even if the programs you are using have a setuptools dependency, they will work without requiring any change on the code or in their setup.py files. Same goes for zc.buildout apps.

Autres liens:

Posted in package_management, python | Tagué: , | Leave a Comment »

Quelques articles sur des frameworks Web Python : Django, Pylons, Web2py, Repoze.bfg, Pyxer

Posted by patrick sur février 8, 2009


  • http://plope.com/whats_your_web_framework_doing (‘It can be a bit useless to benchmark web application frameworks. When you’re commmitted to a particular framework, either it works or it doesn’t for your particular application; often raw speed is not really a concern. You’re probably not going to switch web frameworks in the middle of a project in order to get a 15% or even a 50% or 100% speed increase: you’ve got too much investment in the code that works under the framework to consider it. In my experience, very few people truly understand more than one web framework, and they tend to use that framework for everything even it it’s slightly less optimal for any specific task; this is because the « switching cost » to go to another one is so high. So benchmarks aren’t really all that interesting in the « real » web world; it all depends on context. But if you haven’t chosen a web framework yet (is there anyone?), or if you’re falling out of love with your current web framework and you’re considering using a different one, you might be able to learn something from profiling an application running under various frameworks nonetheless, even if you ignore the raw speed of the framework itself’Here are the results:
  • http://compoundthinking.com/blog/index.php/2009/02/04/wsgi-and-cooperation/ (‘…The point of all this is that the wider python web world is seeing a huge increase in cross-polination of ideas, of working together on projects, and sharing code across major frameworks from TurboGears to Zope, to Pylons. And at the core of all this is the WSGI specification. WSGI may not be pefect, but it’s still great, because it provides a defined way for web framework developers to work together. And, that coupled with the fact that we’re all pretty friendly people, and we all like each other and want to work together on things, has created a renasance of sorts in python web tools world…’)
  • http://code.google.com/p/pyxer/ (‘Yet another Python Framework! The goal of this project to make web development as easy as possible to enable the developer to start quickly with a new project. This project should be distributable via PasteDeploy for normal servers and for Google App Engine (GAE), without the need of using different technologies and approaches. The Pyxer Server is a very simple Python Web Framework that aims to makes starting a new project as easy as it can be. It still works respecting the MVC concept but the files can be mixed in one directory. For a high end solution you should maybe consider using Pylons, Django, TurboGears and similar. This work is inspired by http://pythonpaste.org/webob/do-it-yourself.htmlTechnical background. The Google App Engine (GAE) in version 1.1 offers a very restricted Python environment and the developer as to ship arround a lot of limitations. Pyxer helps on this point by providing solutions that also work together with the WSGI Framework Paste by Ian Bicking. This way you get the best from both sides: GAE and Paste. To achieve this, some other common third party tools are used like WebOb and VirtualEnv also by Ian Bicking. The templating s based on Genshi. .‘)
  • http://groups.google.com/group/comp.lang.python.announce/browse_thread/thread/09cfe8decd549127/1382cac331ce7805?show_docid=1382cac331ce7805 (‘web2py 1.56 is out, including a new web site with better documentation http://www.web2py.com.
    • New features Include:
      – Authentication
      – Authorization (Role Based Access Controller)
      – CRUD
      – portable url fetch function
      – portable geocoding function
      – PEP8 compliant
      – Python 2.5 and Python 2.6 compliant
      – Runs on Jython (although without db drivers)
      – Runs on IronPython (although without CSV, db drivers and internal
      – DAL shortcuts
      – SQLFORM has default image preview
      – new generic.html view
      – more examples and documentation
      – always backward compatible ‘)
  • http://kiznet.fr/blog/2008/11/21/django–python-cest-plus-fort-que-toi/ (‘Aujourd »hui, je vais vous présenter Django, « Le framework web pour les perfectionnistes sous pression » ! Avouez que c »est déjà pas mal comme slogan. Je vais parler ici de mon expérience personnelle d’amateur qui pratique l »informatique sur son temps libre. Donc tout ce que je peux raconter ici peut apparaître totalement FAUX à un expert en la matière, mais bon, faut bien se lancer un jour hein 🙂 …J’ai ensuite installé Django en lui même (ce qui se résume à télécharger la version en cours de développement puis créer 2-3 liens symboliques, du gâteau quoi), puis paramétrer mon httpd.conf (de même, processus très simple grâce à la très bonne documentation très bien traduite par David Larlet. Résultat, en partant de rien, un framework fonctionnel disponible en production en tout juste 5 petites minutesBienvenue dans le monde merveilleux de Django! 🙂 Pour tes accents, vérifie que tu as bien créé ta base MySQL avec le bon encodage. Il y a en fait deux encodages qui interviennent: celui de la base et celui de la connexion client. Tu peux mettre les deux en UTF-8. L’encodage de la base détermine comment ton texte est stocké physiquement sur le disque; si tu mets un encodage non-universel, alors certains caractères ne pourront être stockés. L’encodage de la connexion client détermine comment ton texte est transféré physiquement entre le serveur SQL et ton applicatif. Là aussi, il est important que l’encodage choisi soit capable de gérer les caractères de ton texte, mais il est également très important que l’applicatif et le serveur SQL soient d’accord sur cet encodage (c’est une erreur que je vois trop souvent sur des sites PHP). Normalement Django pose tout seul les bons encodages sur la connexion client, donc de ce point de vue là ce n’est probablement pas un souci pour toi. Mais fais gaffe à l’encodage de la base. ‘)
  • http://www.screeley.com/entries/2008/aug/21/enterprise-django/ (‘…If there’s one thing I love about Alfresco it would be webscripts. They make it easy to retrieve, create and modify content from a repository RESTfully. Webscripts are also very technology agnostic, you can put just about anything on top. We looked at Symphony, Ruby on Rails and Django, but because I have a hammer and everything looks like a nail, we went with Django. This might have been short sighted. The developer working on this django-jython project only got Django running on Jython in TomCat 4 days ago. It also doesn’t help that the client asked if we misspelled banjo in our presentation. Needless to say there will be a few challenges going forward. Tech Stuff: If you have an existing Django project it takes less than an hour to make the conversion. Before you get started you will need three things: Java 6, Tomcat(I am using 5.5) and the Postgres JDBC. The initial install directions are here and to create the WAR the docs are here. The only issue I ran into was a Java class version issue, which is why you need Java 6. ‘)

A voir:

  • http://wiki.python.org/moin/WebFrameworks
  • http://mdp.cti.depaul.edu/ (‘web2pyTM Enterprise Web Framework.Free and open source full-stack enterprise framework for agile development of fast, secure and portable database-driven web-based applications. Written and programmable in Python.Version 1.56.1 (2009-02-05 11:49:10) Created by Massimo Di Pierro. © 2007-2009. ‘)
  • http://en.wikipedia.org/wiki/Django_(web_framework)
  • http://www.django-fr.org/liens/ (‘La documentation officielle est complète mais ne suffit parfois pas à certaines spécificités, les liens présents sur cette page sont là pour combler ce manque. ‘)
  • http://plope.com/Members/chrism/repoze-bfg (‘…So, for the last few weeks (at the expense of much else), with a lot of help from Paul Everitt and Tres Seaver, I’ve been working on a new web framework. This framework is named (provisionally) repoze.bfg (« Big Fine Gun« ). It uses the Zope CA, and Zope interfaces, and a bunch of other Zope libraries, but doesn’t use the Z2 or Z3 publishers; it has its own publisher. It also does not use any form of Zope security; it implements its own security model. It has the same concept of views that Zope and Django have. Its development model is closer to Pylons’ or Django’s than it is to Zope’s. It uses the z3c.pt templating system, which is a reimplementation of ZPT, or you can also use XSL, or plug in whatever templating system you like really. It doesn’t assume any particular persistence mechanism; you provide your own (ZODB, relational, filesystem, etc); in a tip of the hat to Pylons, that’s considered an application decision. It depends heavily on a variety of Ian Bicking creations, including WebOb and Paste. It’s very small, currently clocking in at a little over 2000 lines of code, minus dependencies. That said, it does about what I want out of a web framework. It’s still in a pretty early stage, but it has definitely moved its way out of « toy » phase. We’ve managed to create several applications using the framework so far. The very first one was repoze.virginia , which is the application which serves up Repoze.org these days. It’s a simple file hierarchy publisher with slight dynamicism. The second one Paul wrote, and it’s named repoze.lxmlgraph , where he demonstrates how one might publish a website that was represented entirely by a single XML document. The third one is named repoze.cluegun , which is a port of Rocky Burt’s ClueBin application to repoze.bfg. That code runs the ClueGun website . Of the three, ClueGun is probably the most real-world app (albeit small). I’m sort of excited about it. It’s sort of like getting out of jail or something to be able to have your own web framework to write as you see fit. We’ll likely continue to develop applications using repoze.bfg, pushing features into the framework as we find them useful.’)
  • http://en.wikipedia.org/wiki/Pylons_(web_framework)
  • http://static.repoze.org/bfgdocs/ (‘repoze.bfg is a Python web application framework. It is inspired by Zope, and uses Zope libraries to do much of its work. However, it is simpler than any released version of Zope. repoze.bfg uses the WSGI protocol to handle request and responses.’)
    • http://static.repoze.org/bfgdocs/narr/introduction.html#similarities-to-other-frameworks (‘The repoze.bfg concept of traversal is inspired by Zope. Additionally, repoze.bfg uses the Zope Component Architecture (“CA”) internally, as do Zope 2, Zope 3, and Grok. Application repoze.bfg developers use either ZCML (an XML dialect, used in Zope) or decorators to perform various configuration tasks. The decorator support is provided by the Grok project. Like Pylons, repoze.bfg is mostly policy-free. It makes no assertions about which database you should use, and its built-in templating facilities are only for convenience. In essence, it only supplies a mechanism to map URLs to view code, along with a convention for calling those views. You are free to use third-party components in your application that fit your needs. Also like Pylons, repoze.bfg is heavily dependent on WSGI.The “Django docs state that Django is an “MTV” framework in their FAQ. This also happens to be true for repoze.bfg:‘)
    • http://static.repoze.org/bfgdocs/narr/introduction.html#differences-from-other-frameworks (‘ Like Zope, the repoze.bfg framework imposes slightly more control inversion upon application developers than other Python frameworks such as Pylons. For example repoze.bfg assumes that you’re wiling to resolve a URL to a context object before passing it to a view. Pylons and other Python “MVC” frameworks have no such intermediate step; they resolve a URL directly to a controller. Another example: using the repoze.bfg security subsystem assumes that you’re willing to attach an ACL to a context object; the ACL is checked by the framework itself instead of by user code, and access is permitted or denied by the framework itself rather than by user code. Such a task would typically be performed by user-space decorators in other Python web frameworks. Unlike application development using Zope, application developers don’t interact with the Zope Component Architecture (“CA”) very much during repoze.bfg application development. Instead, the repoze.bfg framework tends to “hide” most interaction with the CA behind special-purpose API functions. Also unlike Zope and unlike other “full-featured” frameworks such as Django, repoze.bfg makes no assumptions about what persistence mechanisms you want to use to build an application. Zope applications are typically reliant on ZODB; repoze.bfg allows you to build ZODB applications, but it has no reliance on the ZODB package. Likewise, Django tends to make the assumption that you’re going to want to store your application’s data in a relational database. repoze.bfg makes no such assumption; it allows you to use a relational database but doesn’t enourage or discourage an application developer about such a decision.’)
  • http://en.wikipedia.org/wiki/Grok_(web_framework) (‘ Grok is a web framework based on Zope 3 technology. The project was started in 2006 by a number of Zope 3 developers.Grok has since then seen regular releases. Its core technologies (Martian, grokcore.component) are also finding uptake in other Zope 3 and Zope 2 based projects. The primary motive behind Grok is to make the powerful, but not quite easily approachable Zope 3 technology more accessible and easier to use for newcomers and, at the same time, speed up application development, in accordance with the agile programming paradigm.To achieve this, Grok gives up some Zope 3 principles. Instead of using a strictly explicit, XML based configuration language (ZCML) like Zope 3 does, Grok uses Python code for component configuration, and has many implicit defaults and conventions so that you don’t have to have so much configuration, in the hope that it is easier to learn than standard Zope 3. Grok aims to be more similar in feel to popular Python web frameworks like TurboGears, Pylons and Django, while at the same time it retains much of Zope 3’s codebase and its component approach to development. ‘)

Posted in 2009, Ironpython, jython, package_management, python, Python Web Frameworks | Tagué: , , , , , | Leave a Comment »

Infos python: bytecodes, python parser, AST, gajim

Posted by patrick sur décembre 19, 2008

http://spyrit.svn.sourceforge.net/viewvc/spyrit/trunk/dist (‘A la question: Comment concentrer en un seul script toutes les classes et fonctions nécessaires (mais seulement ce qui est nécessaire) pour obtenir le résultat ? Sundance  a répondu : j’ai développé un petit système qui permet de faire ce genre de chose: tu lances ce programme en lui passant en paramètre le nom du script à packager, et ça cherche les modules dont dépend ce script, ça les compile en bytecode, et ça génère un seul script contenant 1) le bytecode des modules dans un dictionnaire, et 2) un hook d’import qui cherche les imports dans ce dictionnaire. »)

## -*- coding: utf-8 -*-

import sys
import imp
import bz2
import time
import struct
import marshal
import os.path

LAUNCHER_STUB = """#!/usr/bin/python
## -*- coding: utf-8 -*-


import sys
import imp
import bz2
import marshal

class embedded_module_importer:

  def find_module( s, fullname, path=None ):

    if fullname == "MAIN":
      fullname = "__main__"

    return EMBEDDED_MODULES.has_key( fullname ) and s or None

  def load_module( s, fullname ):

    if fullname == "MAIN":
      fullname = "__main__"

    filename, code = EMBEDDED_MODULES[ fullname ]
    code =  marshal.loads( bz2.decompress( code ) )

    mod = sys.modules.setdefault( fullname, imp.new_module( fullname ) )
    mod.__file__ = filename
    mod.__loader__ = s

    exec code in mod.__dict__

    return mod

sys.meta_path.append( embedded_module_importer() )

import MAIN

def get_timestamp_long():
  return struct.pack( "l", long( time.time() ) )

def make_bytecode( filename ):

  m  = compile( file( filename ).read(), filename, 'exec' )
#  bc = imp.get_magic() + get_timestamp_long() + marshal.dumps( m )
  bc = bz2.compress( marshal.dumps( m ) )

  return bc

def make_module_dict( modules ):

  mods = []

  for ( modulename, f ) in modules:
    bc = make_bytecode( f )
    mods.append( "  %s: ( %s, %s )" % ( modulename.__repr__(),
                                        make_bytecode( f ).__repr__() ) )

  return ",\n".join( mods )

def make_launcher( main, modules ):

  return LAUNCHER_STUB % ( make_module_dict( modules ) )

def build( scriptname, outputname=None ):

  mainname, ext = os.path.splitext( scriptname )

  from modulefinder import ModuleFinder

  p = os.path.abspath( os.getcwd() )

  if os.path.dirname( scriptname ):
    os.chdir( os.path.dirname( scriptname ) )

  mf = ModuleFinder( [ "." ] )
  mf.run_script( os.path.basename( scriptname ) )

  libs = [ ( name, mod.__file__ ) for ( name, mod ) in mf.modules.iteritems()
                                  if mod.__file__ ]

  output = make_launcher( mainname, libs )

  os.chdir( p )

  if not outputname:
    print output

    file( outputname, "wb" ).write( output )

def main():

  if not len( sys.argv ) > 1:

  if len( sys.argv ) > 2:
    build( sys.argv[1], sys.argv[2] )

    build( sys.argv[1] )

if __name__ == "__main__":

Dans le même genre de choses:

http://svn.python.org/view/python/branches/release30-maint/Tools/freeze/README?rev=67499&view=markup (‘What is Freeze ? Freeze make it possible to ship arbitrary Python programs to people who don’t have Python. The shipped file (called a « frozen » version of your Python program) is an executable, so this only works if your platform is compatible with that on the receiving end (this is usually a matter of having the same major operating system revision and CPU type). ‘)

http://svn.python.org/view/python/branches/release30-maint/Python/compile.c?rev=67499&view=markup(‘This file compiles an abstract syntax tree (AST) into Python bytecode‘)

http://svn.python.org/view/python/branches/release30-maint/Python/ast.c?rev=67499&view=markup(‘This file includes functions to transform a concrete syntax tree (CST) to * an abstract syntax tree (AST). The main function is PyAST_FromNode()‘)

http://pypi.python.org/pypi/sourcecodegen/0.6.2 (‘This package provides a module-level source-code generator which
operates on the AST from the built-in compiler.ast module.Note that this AST is not compatible with the new ast module in
Python 2.6

The generator works on AST parse trees.

from compiler import parse
tree = parse("""\
...     print 'Hello, world!'
... """)

We can now generate Python-code equivalent to the original using the source-code generator.

from sourcecodegen import ModuleSourceCodeGenerator
generator = ModuleSourceCodeGenerator(tree)
print generator.getSourceCode()
print 'Hello, world!'


http://svn.python.org/view/python/branches/release30-maint/Demo/parser/README?rev=67499&view=markup (‘These files are from the large example of using the `parser’ module. Refer
to the Python Library Reference for more information.
It also contains examples for the AST parser.


FILES — list of files associated with the parser module.

README — this file.

example.py — module that uses the `parser’ module to extract
information from the parse tree of Python source

docstring.py — sample source file containing only a module docstring.

simple.py — sample source containing a « short form » definition.

source.py — sample source code used to demonstrate ability to
handle nested constructs easily using the functions
and classes in example.py.

test_parser.py program to put the parser module through its paces.

unparse.py AST (2.5) based example to recreate source code
from an AST. This is incomplete; contributions
are welcome.


http://svn.python.org/view/python/branches/release30-maint/Doc/library/parser.rst?rev=67499&view=markup (‘Access parse trees for Python source code…From Python 2.5 onward, it’s much more convenient to cut in at the Abstract  Syntax Tree (AST) generation and compilation stage, using the :mod:`ast` module’)

http://linuxfr.org/2008/12/18/24812.html(‘Gajim, le client Jabber aérien (écrit en Python et GTK+ pour ceux que ça intéresse) vient de sortir en version 0.12, quasiment un an jour pour jour après la 0.11.4 et deux ans après la 0.11.
L’avantage par rapport à d’autres de ce client de messagerie instantanée, sous licence GPLv3, est qu’il ne fait que le protocole Jabber (XMPP), et le fait bien. On a même accès à une console XML dans laquelle on peut injecter le XML qu’on veut, très fun pour jouer avec les copains pendant ces froides soirées d’hiver. Gros plus non négligeable, il est l’un des rares à prendre en charge GnuPG (peut-être même le seul en GTK+), pour la confidentialité de vos conversations ou pour authentifier votre interlocuteur.
Au programme des nouveautés :

  • Amélioration du support des sessions ;
  • Chiffrement point à point (transparent pour l’utilisateur) ;
  • Amélioration de l’interface graphique en général ;
  • Validation des certificats SSL ;
  • Intégration de Kerberos ;
  • Salons de discussion dans le roster ;
  • Conversion d’une conversation privée en salon de discussion ;
  • Événements personnels (PEP) : humeur, activité, musique écoutée ;
  • Et bien sûr, la classique corrections des bugs.

Il ne reste qu’à l’installer 😉

Posted in package_management, python | Tagué: , , , | Leave a Comment »

Python: news du 17 décembre 2008

Posted by patrick sur décembre 18, 2008

  • http://tarekziade.wordpress.com/2008/12/16/pycon-2009-talks/ (‘The talks that made it:
    • How AlterWay releases web applications using zc.buildout
    • On the importance of PyPI in delivering and building Python softwares – mirroring, fail-over and third-party package indexes »)
  • http://jessenoller.com/2008/12/16/pycon-2009-talks-accepted/ (‘Last night I got two emails – both of my talks I proposed for PyCon 2009 were accepted, here’s the title and abstract from both:
    • Introduction to Multiprocessing in Python
      • This talk will cover the new multiprocessing package included with Python 2.6 (and 3.0) focusing on design, benefits, practical usage, application construction, gotchas and how to use it to build multi-core and distributed applications.
    • Concurrency and Distributed Computing with Python Today
      • This talk will cover the recent changes to Python 2.6, including a brief introduction to the threading module and multiprocessing inclusion and changes but will primarily focus on the concurrent and distributed ecosystem for Python today.‘)

Posted in 2008, DotNet, Ironpython, Multiprocessing, package_management, python | Tagué: | Leave a Comment »

Python package management: « using pip requirements » from Ian Bicking

Posted by patrick sur décembre 17, 2008

Un article très dense et très instructif de Ian Bicking reproduit dans son intégralité:

Source: http://blog.ianbicking.org/2008/12/16/using-pip-requirements/


Following onto a set of recent posts (from James, me, then James again), Martijn Faassen wrote a description of Grok’s version management. Our ideas are pretty close, but he’s using buildout, and I’ll describe out to do the same things with pip.

Here’s a kind of development workflow that I think works well:

  • A framework release is prepared. Ideally there’s a buildbot that has been running (as Pylons has, for example), so the integration has been running for a while.
  • People make sure there are released versions of all the important components. If there are known conflicts between pieces, libraries and the framework update their install_requires in their setup.py files to make sure people don’t use conflicting pieces together.
  • Once everything has been released, there is a known set of packages that work together. Using a buildbot maybe future versions will also work together, but they won’t necessarily work together with applications built on the framework. And breakage can also occur regardless of a buildbot.
  • Also, people may have versions of libraries already installed, but just because they’ve installed something doesn’t mean they really mean to stay with an old version. While known conflicts have been noted, there’s going to be lots of unknown conflicts and future conflicts.
  • When starting development with a framework, the developer would like to start with some known-good set, which is a set that can be developed by the framework developers, or potentially by any person. For instance, if you extend a public framework with an internal framework (or even a public sub-framework like Pinax) then the known-good set will be developed by a different set of people.
  • As an application is developed, the developer will add on other libraries, or use some of their own libraries. Development will probably occur at the trunk/tip of several libraries as they are developed together.
  • A developer might upgrade the entire framework, or just upgrade one piece (for instance, to get a bug fix they are interested in, or follow a branch that has functionality they care about). The developer doesn’t necessarily have the same notion of « stable » and « released » as the core framework developers have.
  • At the time of deployment the developer wants to make sure all the pieces are deployed together as they’ve tested them, and how they know them to work. At any time, another developer may want to clone the same set of libraries.
  • After initial deployment, the developer may want to upgrade a single component, if only to test that an upgrade works, or if it resolves a bug. They may test out combinations only to throw them away, and they don’t want to bump versions of libraries in order to deploy new combinations.

This is the kind of development pattern that requirement files are meant to assist with. They can provide a known-good set of packages. Or they can provide a starting point for an active line of development. Or they can provide a historical record of how something was put together.

The easy way to start a requirement file for pip is just to put the packages you know you want to work with. For instance, we’ll call this project-start.txt:

-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

You can plug away for a while, and maybe you decide you want to freeze the file. So you do:

$ pip freeze -r project-start.txt project-frozen.txt

By using -r project-start.txt you give pip freeze a template for it to start with. From that, you’ll get project-frozen.txt that will look like:

-e svn+http://mycompany/svn/MyApp/trunk@1045#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk@1058#egg=MyLibrary

## The following requirements were added by pip –freeze:

# Installing as editable to satisfy requirement INITools==0.2.1dev-r3488:
-e svn+http://svn.colorstudy.com/INITools/trunk@3488#egg=INITools-0.2.1dev_r3488

At that point you might decide that you don’t care about the nose version, or you might have installed something from trunk when you could have used the last release. So you go and adjust some things.

Martijn also asks: how do you have framework developers maintain one file, and then also have developers maintain their own lists for their projects?

You could start with a file like this for the framework itself. Pylons for instance could ship with something like this. To install Pylons you could then do:

$ pip -E MyProject install  -r http://pylonshq.com/0.9.7-requirements.txt

You can also download that file yourself, add some comments, rename the file and add your project to it, and use that. When you freeze the order of the packages and any comments will be preserved, so you can keep track of what changed. Also it should be ameniable to source control, and diffs would be sensible.

You could also use indirection, creating a file like this for your project:

-r http://pylonshq.com/0.9.7-requirements.txt
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp

-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

That is, requirements files can refer to each other. So if you want to maintain your own requirements file alongside the development of an upstream requirements file, you could do that.

Posted in 2008, Distribution de logiciel, Génie logiciel, Gestion de version, Intégration continue, package_management, python | Tagué: , , | Leave a Comment »

Python package management: critiques de setuptools et louanges de pip et virtualenv

Posted by patrick sur décembre 16, 2008

La compréhension de la gestion des packages Python est devenue essentielle avec l’arrivée de 2 nouvelles versions de Python: python 2.6 et Python 3000 (ou Python 3k)

Voici quelques articles récents:

http://blog.ianbicking.org/2008/12/14/a-few-corrections-to-on-packaging/ (‘James Bennett recently wrote an article on Python packaging and installation, and Setuptools. There’s a lot of issues, and writing up my thoughts could take a long time, but I thought at least I should correct some errors, specifically category errors. Figuring out where all the pieces in Setuptools (and pip and virtualenv) fit is difficult

This is an alternative to easy_install. It works somewhat differently than easy_install, but not much. Mostly it is better than easy_install, in that it has some extra features and is easier to use. Unlike easy_install, it downloads all distributions up-front, and generates the metadata to read distribution and version requirements. It uses Setuptools to generate this metadata from a setup.py file, and uses pkg_resources to parse this metadata. It then installs packages with the setuptools monkeypatches applied. It just happens to use an option python setup.py –single-version-externally-managed, which gets Setuptools to install packages in a more flat manner, with Distro.egg-info/ directories alongside the package. Pip installs eggs! I’ve heard the many complaints about easy_install (and I’ve had many myself), but ultimately I think pip does well by just fixing a few small issues. Pip is not a repudiation of Setuptools or the basic mechanisms that easy_install uses.
This is a little hack that creates isolated Python environments. It’s based on virtual-python.py, which is something I wrote based on some documentation notes PJE wrote for Setuptools. Basically virtualenv just creates a bin/python interpreter that has its own value of sys.prefix, but uses the system Python and standard library. It also installs Setuptools to make it easier to bootstrap the environment (because bootstrapping Setuptools is itself a bit tedious). I’ll add pip to it too sometime. Using virtualenv you don’t have to worry about different library versions, because for any one environment you will probably only need one version of a library. On any one machine you probably need different versions, which is why installing packages system-wide is problematic for most libraries. (I’ve been meaning to write a post on why I think using system packaging for libraries is counter-productive, but that’ll wait for another time.

…I don’t think zipping libraries up is all that useful, and while it should work, it doesn’t always, and it makes code harder to inspect and understand. So since it’s not that useful, I’ve disabled it when pip installs packages. I also have had it disabled on my own system for years now, by creating a distutils.cfg file with [easy_install] zip_ok = False in it. Sadly App Engine is forcing me to use zip files again, because of its absurdly small file limits… but that’s a different topic. (There is an experimental pip zip command mostly intended for App Engine.

…Another pain point is version management with setup.py and Setuptools. Indeed it is easy to get things messed up, and it is easy to piss people off by overspecifying, and sometimes things can get in a weird state for no good reason (often because of easy_install’s rather naive leap-before-you-look installation order). Pip fixes that last point, but it also tries to suggest more constructive and less painful ways to manage other pieces.

Pip requirement files are an assertion of versions that work together. setup.py requirements (the Setuptools requirements) should contain two things: 1: all the libraries used by the distribution (without which there’s no way it’ll work) and 2: exclusions of the versions of those libraries that are known not to work. setup.py requirements should not be viewed as an assertion that by satisfying those requirements everything will work, just that it might work. Only the end developer, testing the system together, can figure out if it really works. Then pip gives you a way to record that working set (using pip freeze), separate from any single distribution or library….

  1. I am new to pip, so I hope I don’t write silly things ;)It’s a good idea to have the requirements outside the package, instead of what setuptools does. So you can share them amongst several packages.

    So basically, pip’s requirements files are what Zope calls the “Known Good Set” and what Turbogears 2 does by maintaining its own PyPI server to distribute TG2 packages : a list of versions that are known to interact well in the same environment, right ?

    But it’s not really different from setuptools there, except that you change the requirements in a different place if something goes wrong.

    So to simplify the problem, couldn’t we have juste ONE requirement file in the whole Python ?

    This could be the clue for os packagers : they would be able to tweak this file, while developers would be able to try out their package over different requirements files (”the debian etch python requirement file” “the debian unstable python requirement file”, etc). And for specific, isolated stuff, using virtualenv would allow developer to have their own custome requirement files….

  1. As Brett points out, apt and other package managers are inappropriate because they don’t allow multiple versions of packages to be installed, local environments, and ad hoc packages

  1. Hey Ian,

    This is an excellent and unbiased summary. More or less the missing manual of the current state of packaging under Python.

    I need to add that I really like pip. It doesn’t change much to easy_install, but it is day and night to me. Thanks for making it available.)

  • http://www.b-list.org/weblog/2008/dec/15/pip/ (‘Why I like pip. So yesterday I explained some of the reasons why I don’t like setuptools. In essence, my objections boil down to one idea: application packagingdevelopment should be orthogonal concerns. The way setuptools works, however, seems to tend, inevitably, toward coupling them to each other…But toward the end of yesterday’s article I suggested pip as an alternative to pkg_resources/easy_install toolchain, and today I’d like to explain a bit more about why I prefer pip and some of the concrete benefits it offers and application as an alternative to the setuptools… in his response to my article yesterday Ian clarified that pip just uses the setuptools APIs to do this. Which is saying something: glossing over setuptools’ warts to the point where you don’t even realize it’s being used is a pretty big deal… pip looks before it leaps, can bail out early if it’s not going to be able to install your package and will leave behind a useful log file explaining what went wrong…The point where pip really shines, though, is in the ease of specifying and creating reproducible builds. If you’ve ever dealt with having to deploy the same code base across multiple machines, you know what a headache this can be, since a huge number of factors (operating system and version, pre-installed packages and versions, system package managers and configuration, etc., etc.) can change the results of your deployment process, sometimes in subtle and difficult-to-debug ways. With pip, this is not (so much of) a problem…I mentioned pip requirements files yesterday as an alternative to the way setuptools specifies dependencies directly in setup.py, and that’s certainly one useful application of the feature, but you can take it much further: once you know which packages (and, just as important, which versions of which packages) you need, you can write them down in a simple, plain-text file, point pip at it, and it’ll install them…The last piece of the puzzle, for me, is virtualenv; virtualenv is a tool for creating and working with isolated Python environments, and is basically the only way I work with Python these days…pip integrates quite nicely with virtualenv; normally, when working in an active virtualenv, Python packaging/installation tools (pip included) will install into that virtualenv, but pip also lets you:
  • Specify a virtualenv to install into (using the -E flag), and
  • Create a new virtualenv and install into it.

The second one is really the killer feature, because it means you can set up a requirements file specifying a list of packages, and get pip to create a virtualenv for you and install the packages into it…… well, there’s a heck of a lot more I could write here about pip (and about virtualenv, and about some other interesting tools), but I think this is a good start and hopefully I’ve at least got you interested enough to explore a bit on your own. And I hope I’ve managed to communicate some of the practical reasons why I’ve ditched easy_install for package installation; compared to what pip can do right now (not even considering what it might be able to do in the future), easy_install just doesn’t measure up enough to justify the headaches it can create… »)

  • http://tarekziade.wordpress.com/2008/12/15/python-isolated-environment-pie/ (‘Right now, when Python is loaded, it uses the site module to browse the site-packages directory to populate the path with packages it find there.  .pth files are also parsed to provide extra paths. Python 2.6 has introduced per-user site-packages directory, where you can define an extra directory, which is added in the path like the central one. But both will append new paths to the environment without any rule of exclusion or version checking…A few workarounds exist to be able to express what packages (and version) an application needs to run, or to set up an isolated environment for it:
  • setuptools provides the install_requires mechanism where you can define dependencies directly inside the package, as a new metadata. It also provides a way to install two different versions of one package and let you pick by code or when the program starts, which one you want to activate.
  • virtualenv will let you create an isolated Python environment, where you can define your own site-packages. This allows you to make sure you are not conflicting with a incompatible version of a given package.
  • zc.buildout relies on setuptools and provides an isolated environment a bit similar in some aspects to virtualenv.
  • pip provides a way to describe requirements in a file, which can be used to define bundles, which are very similar to what zc.buildout provides.

But they all aim at the same goal : define a specific execution context for a specific application, and declare dependencies with no respect to other applications or to the OS environment…This proposal describes a solution that can be added to Python to provide that feature. A new file called a  Python Isolated Environment file (PIE file) can be provided by any  application to define the list of dependencies and their versions…’)

Posted in 2008, Distribution de logiciel, Génie logiciel, Gestion de version, package_management, python | Tagué: , , , , , , , | Leave a Comment »

« Python 3.0 and 2.6 and Executable Zipfiles »: a great way of distributing applications as a single file

Posted by patrick sur décembre 8, 2008

Source: http://www.voidspace.org.uk/python/weblog/arch_d7_2008_12_06.shtml#e1038

A new feature that was quietly sneaked into Python 2.6 (NDLR, and Python 3.0), without the fanfare it deserves, is the ability to distribute Python applications as executable zipfiles.

What is new is the ability to make zip archives executable. If you call the Python 2.6+ (or 3.0+) interpreter passing in a zip file instead of a Python file – the interpreter looks inside the zip file for a Python file named __main__.py (at the top-level) and executes it. The zip file can also contain all the (pure-Python only) modules and packages your app depends on.

This is a great way of distributing applications as a single file. The nice thing is that the Python interpreter doesn’t depend on the extension to recognise zipfiles, instead recognising them automagically.

Floris Bruynooghe notes in the comments that you can add a hash-bang line to a zipfile and make it executable:

$ cat > __main__.py
print('hi there')
$ zip test.zip __main__.py
adding: __main__.py (stored 0%)
$ cat > hashbang.txt
#!/usr/bin/env python3.0
$ cat hashbang.txt test.zip > my_exec
$ chmod +x my_exec
$ ./my_exec
hi there

Posted in 2008, package_management, python, Python 3000 | Leave a Comment »