"…mais ce serait peut-être l'une des plus grandes opportunités manquées de notre époque si le logiciel libre ne libérait rien d'autre que du code…"

Archive for the ‘Distribution de logiciel’ Category

Python packaging : Hitchhiker guide, toydist, envbuilder

Posted by patrick sur avril 22, 2010

Toujours beaucoup de mouvement en ce qui concerne la distribution de modules python.

  • http://cournape.wordpress.com/2010/04/22/first-public-release-of-toydist/ (‘The goals of toydist are simplicity and extensibility. There should be only one way to package simple packages (ideally, relying only on the toysetup.info file), while being flexible enough to handle complex softwares. The ultimate goal of toydist is to replace the hideous distutils extensions to build NumPy and SciPy. ‘)
  • http://pypi.python.org/pypi/envbuilder/0.3.0 (‘ A package for automatic generation of virtualenvs Envbuilder is a system for automatically building virtualenvs in Python. To do this, it uses a .env config file to define parcels, which are individual pieces of software to be checked out and installed into the virtualenv. It is mainly tested under Linux, but it should work under any platform that supports unix-y commands (including cygwin). In fact, you might even be able to make one config file work on both Windows and *nix if you’re careful.’)

Posted in 2010, Développement logiciel, distribute, Distribution de logiciel, Doc_sphinx, Génie logiciel, package_management, packaging, toydist | Leave a Comment »

Le service de construction de paquetages d’openSUSE : openSUSE build service (OBS)

Posted by patrick sur avril 7, 2010

  • http://en.wikipedia.org/wiki/OpenSUSE_Build_Service (« …The openSUSE Build Service is an open and complete distribution development platform designed to encourage developers to compile packages for multiple Linux distributions including openSUSE, Red Hat, Mandriva, Ubuntu, Fedora and Debian. It typically simplifies the packaging process, so developers can more easily package a single program for many distributions, and many openSUSE releases, making more packages available to users regardless of what distribution version they use. The build service software is published under the GPL. In an acknowledgement of its usefulness to the wider Linux community, the Linux Foundation has announced that the project will be added to the Linux Developer Network (LDN) »)
  • http://fr.opensuse.org/Build_Service ( » … OpenSUSE Build Service (OBS) est une plateforme complète de développement de distribution fournissant l’infrastructure nécessaire pour le développement des futures distributions basées sur openSUSE. Il comporte également les services qui permettent la compilation et la création de paquets pour les autres distributions Linux, comme Fedora, Debian, Ubuntu, et bien d’autres. Les utilisateurs d’openSUSE peuvent facilement passer en revue les différents paquets via l’interface web http://software.opensuse.org/ et télécharger les derniers paquets. Les interfaces ouvertes permettent aux services externes (par ex: SourceForge) et aux pages web d’interagir avec le Build Service et d’utiliser ses ressources…..)

Posted in 2010, command line interface, Développement logiciel, Distribution de logiciel, logiciel libre, openSUSE build service, packaging, python | Tagué: , , , , , , | Leave a Comment »

Le nouveau cahier de l’administrateur Debian GNU/Linux « Lenny » de Raphaël Hertzog et Roland Mas est sorti

Posted by patrick sur mars 22, 2009

Debian GNU/Linux lenny administration

Debian GNU/Linux lenny administration

Références :

Sources:

http://www.ouaza.com/livre/admin-debian/(‘Ce livre a été rédigé par deux développeurs Debian de longue date (Raphaël contribue depuis 1997, Roland depuis 2000). Nous avons essayé de vous faire découvrir toutes les facettes de cette distribution fascinante, autant d’un point de vue technique qu’organisationnel. Jugez-en par vous même ! ‘)

http://roland.entierement.nu/blog/2009/03/02/debian-lenny-le-livre-le-jeu.html (‘Si vous n’avez pas passé les trois dernières semaines sous un caillou ou dans un monastère, vous savez déjà que la version 5.0 de Debian GNU/Linux, nom de code « Lenny », est sortie le 14 février dernier. Pour accompagner la publication de cette nouvelle version majeure de Debian, Raphaël Hertzog et moi-même avons également mis à jour le Cahier de l’admin Debian. L’édition portant sur Lenny est déjà disponible en version électronique sur Izibook, et la version papier sera chez les libraires le 19 mars. Elle ne constitue pas une révolution (on ne change pas une formule qui est devenue au fil des ans la référence en français), mais principalement une mise à jour, avec des ajouts pour combler quelques vides (par exemple, de nouvelles sections sur OpenVPN et les partitions chiffrées. Pour certaines des précédentes éditions, il avait été lancé un concours où les personnes faisant la meilleure promotion étaient récompensées par des exemplaires du livre. Pour cette édition, nous avons gardé l’idée, mais l’objet du concours est différent : il s’agit non plus de faire la promotion du livre, mais de participer à la communauté Debian. Pas besoin d’être un super-développeur, il suffit d’apporter une contribution qui va faire progresser Debian d’une manière ou d’une autre, à condition que ce soit tangible. Raphaël écrit une série d’articles sur le sujet, vous pouvez donc aller y piocher des idées. J’en ajouterai deux, complémentaires :

  • Publier un blog qui donne une visibilité aux évolutions récentes, en cours ou prévues de Debian unstable et/ou testing, voire experimental. Quelques exemples de ce qu’on pourrait y voir serait l’arrivée d’une nouvelle version majeure du noyau ou de Gnome, la disparition d’un paquet au profit d’un autre, l’arrivée de nouveaux paquets intéressants, leur migration vers testing, etc. Pas une liste de tous les changements de chaque bibliothèque mineure, mais juste une vue macroscopique de ce qui se passe de visible pour les utilisateurs.
  • Dans un registre un peu différent, il serait intéressant de publier (peut-être aussi sous forme de blog) un baromètre de l’état actuel d’unstable, avec les migrations en cours, les paquets importants connus pour être cassés, etc. Le but est de faciliter l’usage d’unstable par des utilisateurs courageux-mais-pas-téméraires. Un exemple de ce que j’y verrais bien est une mention du bug 511009, assorti d’une mention que c’est pas une bonne idée d’upgrader CUPS pour l’instant (ça m’aurait évité quelques heures d’interrogations) ; et un autre billet quand c’est résolu. Là encore, trop de débit ne servirait à rien, il suffirait d’une vision macroscopique de l’état des différents sous-systèmes.

Bien entendu, ce ne sont que des idées, et les plus originales seront peut-être les meilleures, puisqu’elles seront celles que nous n’aurons pas prévues et apporteront donc une réelle nouveauté.’  voir http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/ )

http://www.eyrolles.com/Informatique/Livre/debian-lenny-9782212124439 (‘Debian GNU/Linux, distribution Linux non commerciale extrêmement populaire, est réputée pour sa fiabilité et sa richesse. Créée et maintenue par un impressionnant réseau de plusieurs milliers de développeurs dans le monde, elle a pour principes l’engagement vis-à-vis de ses utilisateurs et la qualité. Ses technologies concernent un nombre toujours croissant d’administrateurs, notamment par le biais de la distribution dérivée Ubuntu.

Ce cahier de l’Admin entièrement revu pour Debian Lenny perpétue le succès des premières éditions : accessible à tous, il fournit les connaissances indispensables pour qui souhaite devenir un administrateur Debian GNU/Linux efficace et indépendant. Il traite des méthodes que tout administrateur Linux compétent maîtrise, depuis l’installation et la mise à jour du système jusqu’à la création de paquetages et la compilation d’un noyau Linux, en passant par la supervision, la sauvegarde et les migrations, sans oublier des techniques avancées telles que la mise en place de SELinux pour sécuriser des services, l’automatisation des installations ou encore la virtualisation avec Xen.

Au sommaire

  • Le projet GNU/Debian
  • Présentation de l’étude de cas
  • Prise en compte de l’existant et migration
  • Installation
  • Système de paquetage, outils et principes fondamentaux
  • Maintenance et mise à jour : les outils APT
  • Résolution de problèmes et sources d’information
  • Configuration de base : réseau, comptes, impression
  • Services Unix
  • Infrastructure réseau
  • Services réseau : Postfix, Apache, NFS, Samba, Squid, LDAP
  • Administration avancée
  • Station de travail
  • Sécurité
  • Conception d’un paquet Debian
  • Conclusion : l’avenir de Debian
  • A. Distributions dérivées
  • B. Petit cours de rattrapage

Extraits du livre

  • Table des matières – Fichier PDF (248.8 Ko)
  • Avant-propos – Fichier PDF (255 Ko), « Linux commence à bénéficier d’une couverture médiatique non négligeable, profitant essentiellement aux distributions commerciales (RedHat, SuSE, Mandriva…). Debian, souvent placée par les sondages dans le trio de tête des distributions les plus populaires, est pourtant loin d’être marginale, surtout si l’on y inclut les distributions dérivées dont certaines — comme Ubuntu — connaissent un succès très important.
    Ce n’est pas un hasard si Hewlett-Packard a vu ses ventes de matériel augmenter de 25 millions de dollars en 2006 après avoir proposé du sup-
    port pour Debian. Ce livre a ainsi pour vocation de faire découvrir cette distribution. Nous espérons vous faire profiter de toute l’expérience acquise depuis que nous avons rejoint le projet en tant que développeurs-contributeurs, en 1998 pour Raphaël et en 2000 pour Roland. Peut-être parviendrons-nous à
    vous communiquer notre enthousiasme, et vous donner l’envie de rejoindre nos rangs d’ici quelque temps, qui sait..
    . »
  • Préface – Fichier PDF (169.2 Ko)
  • Chapitre 1 : Le projet Debian – Fichier PDF (599.3 Ko)
  • Chapitre 6 : Maintenance et mise à jour : les outils APT – Fichier PDF (553.9 Ko), (« Ce qui rend Debian si populaire auprès des administrateurs,
    c’est la facilité avec laquelle il est possible d’y installer des logiciels et de mettre à jour le système complet. Cet avantage unique est dû en grande partie au programme APT, outil dont les administrateurs de Falcot SA se sont empressés d’étudier les possibilités. »)
Voir aussi:
  • http://www.debian.org/News/2009/20090214 (‘Le projet Debian est heureux d’annoncer la publication officielle de la version 5.0 de Debian GNU/Linux, surnommée Lenny, après 22 mois de développement constant. Debian GNU/Linux est un système d’exploitation libre qui prend en charge au total douze architectures de processeurs et contient les environnements de bureau KDE, GNOME, Xfce et LXDE. C’est un système compatible avec la version 2.3 du FHS (« Filesystem Hierarchy Standard », ou standard sur l’organisation des systèmes de fichiers) et les logiciels développés pour la version 3.2 de la LSB (« Linux Standard Base », ou base standard de Linux). Debian GNU/Linux fonctionne sur de nombreux ordinateurs, depuis les assistants personnels (« palmtops ») jusqu’aux superordinateurs y compris quasiment tous les systèmes intermédiaires. Au total, douze architectures sont prises en charge : Sun SPARC (sparc), HP Alpha (alpha), Motorola/IBM PowerPC (powerpc), Intel IA-32 (i386) et IA-64 (ia64), HP PA-RISC (hppa), MIPS (mips, mipsel), ARM (arm, armel), IBM S/390 (s390) et AMD64 et Intel EM64T (amd64). Debian GNU/Linux 5.0 Lenny gère également la plateforme Orion de Marvell qui est utilisée dans de nombreux équipements de stockage, parmi lesquels on trouve la série QNAP Turbo Station, HP Media Vault mv2120, et Buffalo Kurobox Pro. De plus, Lenny gère maintenant plusieurs ordinateurs ultra-portables comme le Eee PC d’Asus. Debian GNU/Linux 5.0 Lenny inclut aussi les outils de construction pour Emdebian, qui permet de construire et les paquets source de Debian sur une autre architecture et de les réduire afin qu’ils conviennent aux systèmes embarqués ARM. Debian GNU/Linux 5.0 Lenny comporte le nouveau portage ARM EABI, Armel qui utilise plus efficacement les processeurs ARM actuels. L’ancien portage d’ARM (arm) est maintenant déconseillé.’)
  • http://fr.wikipedia.org/wiki/Debian (”Debian est un système d’exploitation reposant sur les principes du logiciel libre et développé de façon bénévole par l’intermédiaire d’Internet. Ce système permet un mode d’unification des nombreux éléments pouvant être développés indépendamment les uns des autres. Ils sont associés sous forme de « paquets » modulables en fonction des besoins (on en dénombre plus de 23 000 en 2009). On l’assimile généralement à Debian GNU/Linux (la distribution GNU/Linux de Debian), car c’est la seule branche parfaitement fonctionnelle. Toutefois, d’autres versions du système Debian sont possibles comme le prouve le projet Debian GNU/Hurd[1] ou Debian GNU/kFreeBSD[2]. Debian a été utilisée comme base de nombreuses autres distributions comme Knoppix ou Ubuntu qui rencontrent un grand succès..”)
  • http://en.wikipedia.org/wiki/Debian#Debian_releases (” As of February 2009, the latest stable release is version 5.0, code name lenny.[41] When a new version is released, the previous stable is labeled oldstable; currently, this is version 4.0, code name etch. In addition, a stable release gets minor updates (called point releases). The numbering scheme up for the point releases to Debian 4.0 was to include the letter r (for release) after the main version number (eg 4.0) and then the number of the point release; for example, the latest point release of version 4.0 (etch) as of 15 February 2009 is 4.0r7.[42] From Debian 5.0 (lenny) and the numbering scheme of point releases has been changed and conforms to the GNU version numbering standard; so, for example, the first point release of Debian 5.0 is going to be 5.0.1 (instead of 5.0r1).[43].“)
  • http://fr.wikipedia.org/wiki/Debian#Historique_des_versions (”Les différentes versions de la distribution empruntent leurs noms aux personnages du film d’animation Toy Story des studios Pixar “)
  • http://fr.wikipedia.org/wiki/Logiciel_libre (”Un logiciel libre est un logiciel dont la licence dite libre donne à chacun le droit d’utiliser, d’étudier, de modifier, de dupliquer, de donner et de vendre le logiciel. Richard Stallman a formalisé la notion de logiciel libre dans la première moitié des années 1980 puis l’a popularisée avec le projet GNU et la Free Software Foundation (FSF). Depuis la fin des années 1990, le succès des logiciels libres, notamment de Linux, suscite un vif intérêt dans l’industrie informatique et les médias. Les logiciels libres constituent une option face à ceux qui ne le sont pas, qualifiés de « propriétaires ». La notion de logiciel libre ne se confond ni avec celle de logiciel gratuit (freewares), ni avec celle de sharewares, ni avec celle de domaine public. De même, les libertés définies par un logiciel libre sont beaucoup plus étendues que le simple accès aux sources, ce qu’on appelle souvent logiciel Open Source telle qu’elle est définie par l’Open source initiative. Toutefois, la notion formelle de logiciel Open Source est reconnue comme techniquement comparable au logiciel libre.“)
  • Flux RSS:

Posted in 2009, Administration système, Debian, Distribution de logiciel, GNU/Linux, logiciel libre, système embarqué, Ultra-portables | Tagué: , , , , | Leave a Comment »

Python package management: « using pip requirements » from Ian Bicking

Posted by patrick sur décembre 17, 2008

Un article très dense et très instructif de Ian Bicking reproduit dans son intégralité:

Source: http://blog.ianbicking.org/2008/12/16/using-pip-requirements/

================================

Following onto a set of recent posts (from James, me, then James again), Martijn Faassen wrote a description of Grok’s version management. Our ideas are pretty close, but he’s using buildout, and I’ll describe out to do the same things with pip.

Here’s a kind of development workflow that I think works well:

  • A framework release is prepared. Ideally there’s a buildbot that has been running (as Pylons has, for example), so the integration has been running for a while.
  • People make sure there are released versions of all the important components. If there are known conflicts between pieces, libraries and the framework update their install_requires in their setup.py files to make sure people don’t use conflicting pieces together.
  • Once everything has been released, there is a known set of packages that work together. Using a buildbot maybe future versions will also work together, but they won’t necessarily work together with applications built on the framework. And breakage can also occur regardless of a buildbot.
  • Also, people may have versions of libraries already installed, but just because they’ve installed something doesn’t mean they really mean to stay with an old version. While known conflicts have been noted, there’s going to be lots of unknown conflicts and future conflicts.
  • When starting development with a framework, the developer would like to start with some known-good set, which is a set that can be developed by the framework developers, or potentially by any person. For instance, if you extend a public framework with an internal framework (or even a public sub-framework like Pinax) then the known-good set will be developed by a different set of people.
  • As an application is developed, the developer will add on other libraries, or use some of their own libraries. Development will probably occur at the trunk/tip of several libraries as they are developed together.
  • A developer might upgrade the entire framework, or just upgrade one piece (for instance, to get a bug fix they are interested in, or follow a branch that has functionality they care about). The developer doesn’t necessarily have the same notion of « stable » and « released » as the core framework developers have.
  • At the time of deployment the developer wants to make sure all the pieces are deployed together as they’ve tested them, and how they know them to work. At any time, another developer may want to clone the same set of libraries.
  • After initial deployment, the developer may want to upgrade a single component, if only to test that an upgrade works, or if it resolves a bug. They may test out combinations only to throw them away, and they don’t want to bump versions of libraries in order to deploy new combinations.

This is the kind of development pattern that requirement files are meant to assist with. They can provide a known-good set of packages. Or they can provide a starting point for an active line of development. Or they can provide a historical record of how something was put together.

The easy way to start a requirement file for pip is just to put the packages you know you want to work with. For instance, we’ll call this project-start.txt:

Pylons
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

You can plug away for a while, and maybe you decide you want to freeze the file. So you do:

$ pip freeze -r project-start.txt project-frozen.txt

By using -r project-start.txt you give pip freeze a template for it to start with. From that, you’ll get project-frozen.txt that will look like:

Pylons==0.9.7
-e svn+http://mycompany/svn/MyApp/trunk@1045#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk@1058#egg=MyLibrary

## The following requirements were added by pip –freeze:

Beaker==0.2.1
WebHelpers==0.9.1
nose==1.4
# Installing as editable to satisfy requirement INITools==0.2.1dev-r3488:
-e svn+http://svn.colorstudy.com/INITools/trunk@3488#egg=INITools-0.2.1dev_r3488

At that point you might decide that you don’t care about the nose version, or you might have installed something from trunk when you could have used the last release. So you go and adjust some things.

Martijn also asks: how do you have framework developers maintain one file, and then also have developers maintain their own lists for their projects?

You could start with a file like this for the framework itself. Pylons for instance could ship with something like this. To install Pylons you could then do:

$ pip -E MyProject install  -r http://pylonshq.com/0.9.7-requirements.txt

You can also download that file yourself, add some comments, rename the file and add your project to it, and use that. When you freeze the order of the packages and any comments will be preserved, so you can keep track of what changed. Also it should be ameniable to source control, and diffs would be sensible.

You could also use indirection, creating a file like this for your project:

-r http://pylonshq.com/0.9.7-requirements.txt
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp

-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

That is, requirements files can refer to each other. So if you want to maintain your own requirements file alongside the development of an upstream requirements file, you could do that.

Posted in 2008, Distribution de logiciel, Génie logiciel, Gestion de version, Intégration continue, package_management, python | Tagué: , , | Leave a Comment »

Python package management: critiques de setuptools et louanges de pip et virtualenv

Posted by patrick sur décembre 16, 2008

La compréhension de la gestion des packages Python est devenue essentielle avec l’arrivée de 2 nouvelles versions de Python: python 2.6 et Python 3000 (ou Python 3k)

Voici quelques articles récents:

http://blog.ianbicking.org/2008/12/14/a-few-corrections-to-on-packaging/ (‘James Bennett recently wrote an article on Python packaging and installation, and Setuptools. There’s a lot of issues, and writing up my thoughts could take a long time, but I thought at least I should correct some errors, specifically category errors. Figuring out where all the pieces in Setuptools (and pip and virtualenv) fit is difficult

pip:
This is an alternative to easy_install. It works somewhat differently than easy_install, but not much. Mostly it is better than easy_install, in that it has some extra features and is easier to use. Unlike easy_install, it downloads all distributions up-front, and generates the metadata to read distribution and version requirements. It uses Setuptools to generate this metadata from a setup.py file, and uses pkg_resources to parse this metadata. It then installs packages with the setuptools monkeypatches applied. It just happens to use an option python setup.py –single-version-externally-managed, which gets Setuptools to install packages in a more flat manner, with Distro.egg-info/ directories alongside the package. Pip installs eggs! I’ve heard the many complaints about easy_install (and I’ve had many myself), but ultimately I think pip does well by just fixing a few small issues. Pip is not a repudiation of Setuptools or the basic mechanisms that easy_install uses.
virtualenv:
This is a little hack that creates isolated Python environments. It’s based on virtual-python.py, which is something I wrote based on some documentation notes PJE wrote for Setuptools. Basically virtualenv just creates a bin/python interpreter that has its own value of sys.prefix, but uses the system Python and standard library. It also installs Setuptools to make it easier to bootstrap the environment (because bootstrapping Setuptools is itself a bit tedious). I’ll add pip to it too sometime. Using virtualenv you don’t have to worry about different library versions, because for any one environment you will probably only need one version of a library. On any one machine you probably need different versions, which is why installing packages system-wide is problematic for most libraries. (I’ve been meaning to write a post on why I think using system packaging for libraries is counter-productive, but that’ll wait for another time.

…I don’t think zipping libraries up is all that useful, and while it should work, it doesn’t always, and it makes code harder to inspect and understand. So since it’s not that useful, I’ve disabled it when pip installs packages. I also have had it disabled on my own system for years now, by creating a distutils.cfg file with [easy_install] zip_ok = False in it. Sadly App Engine is forcing me to use zip files again, because of its absurdly small file limits… but that’s a different topic. (There is an experimental pip zip command mostly intended for App Engine.

…Another pain point is version management with setup.py and Setuptools. Indeed it is easy to get things messed up, and it is easy to piss people off by overspecifying, and sometimes things can get in a weird state for no good reason (often because of easy_install’s rather naive leap-before-you-look installation order). Pip fixes that last point, but it also tries to suggest more constructive and less painful ways to manage other pieces.

Pip requirement files are an assertion of versions that work together. setup.py requirements (the Setuptools requirements) should contain two things: 1: all the libraries used by the distribution (without which there’s no way it’ll work) and 2: exclusions of the versions of those libraries that are known not to work. setup.py requirements should not be viewed as an assertion that by satisfying those requirements everything will work, just that it might work. Only the end developer, testing the system together, can figure out if it really works. Then pip gives you a way to record that working set (using pip freeze), separate from any single distribution or library….

  1. I am new to pip, so I hope I don’t write silly things ;)It’s a good idea to have the requirements outside the package, instead of what setuptools does. So you can share them amongst several packages.

    So basically, pip’s requirements files are what Zope calls the “Known Good Set” and what Turbogears 2 does by maintaining its own PyPI server to distribute TG2 packages : a list of versions that are known to interact well in the same environment, right ?

    But it’s not really different from setuptools there, except that you change the requirements in a different place if something goes wrong.

    So to simplify the problem, couldn’t we have juste ONE requirement file in the whole Python ?

    This could be the clue for os packagers : they would be able to tweak this file, while developers would be able to try out their package over different requirements files (”the debian etch python requirement file” “the debian unstable python requirement file”, etc). And for specific, isolated stuff, using virtualenv would allow developer to have their own custome requirement files….

  1. As Brett points out, apt and other package managers are inappropriate because they don’t allow multiple versions of packages to be installed, local environments, and ad hoc packages

  1. Hey Ian,

    This is an excellent and unbiased summary. More or less the missing manual of the current state of packaging under Python.

    I need to add that I really like pip. It doesn’t change much to easy_install, but it is day and night to me. Thanks for making it available.)

  • http://www.b-list.org/weblog/2008/dec/15/pip/ (‘Why I like pip. So yesterday I explained some of the reasons why I don’t like setuptools. In essence, my objections boil down to one idea: application packagingdevelopment should be orthogonal concerns. The way setuptools works, however, seems to tend, inevitably, toward coupling them to each other…But toward the end of yesterday’s article I suggested pip as an alternative to pkg_resources/easy_install toolchain, and today I’d like to explain a bit more about why I prefer pip and some of the concrete benefits it offers and application as an alternative to the setuptools… in his response to my article yesterday Ian clarified that pip just uses the setuptools APIs to do this. Which is saying something: glossing over setuptools’ warts to the point where you don’t even realize it’s being used is a pretty big deal… pip looks before it leaps, can bail out early if it’s not going to be able to install your package and will leave behind a useful log file explaining what went wrong…The point where pip really shines, though, is in the ease of specifying and creating reproducible builds. If you’ve ever dealt with having to deploy the same code base across multiple machines, you know what a headache this can be, since a huge number of factors (operating system and version, pre-installed packages and versions, system package managers and configuration, etc., etc.) can change the results of your deployment process, sometimes in subtle and difficult-to-debug ways. With pip, this is not (so much of) a problem…I mentioned pip requirements files yesterday as an alternative to the way setuptools specifies dependencies directly in setup.py, and that’s certainly one useful application of the feature, but you can take it much further: once you know which packages (and, just as important, which versions of which packages) you need, you can write them down in a simple, plain-text file, point pip at it, and it’ll install them…The last piece of the puzzle, for me, is virtualenv; virtualenv is a tool for creating and working with isolated Python environments, and is basically the only way I work with Python these days…pip integrates quite nicely with virtualenv; normally, when working in an active virtualenv, Python packaging/installation tools (pip included) will install into that virtualenv, but pip also lets you:
  • Specify a virtualenv to install into (using the -E flag), and
  • Create a new virtualenv and install into it.

The second one is really the killer feature, because it means you can set up a requirements file specifying a list of packages, and get pip to create a virtualenv for you and install the packages into it…… well, there’s a heck of a lot more I could write here about pip (and about virtualenv, and about some other interesting tools), but I think this is a good start and hopefully I’ve at least got you interested enough to explore a bit on your own. And I hope I’ve managed to communicate some of the practical reasons why I’ve ditched easy_install for package installation; compared to what pip can do right now (not even considering what it might be able to do in the future), easy_install just doesn’t measure up enough to justify the headaches it can create… »)

  • http://tarekziade.wordpress.com/2008/12/15/python-isolated-environment-pie/ (‘Right now, when Python is loaded, it uses the site module to browse the site-packages directory to populate the path with packages it find there.  .pth files are also parsed to provide extra paths. Python 2.6 has introduced per-user site-packages directory, where you can define an extra directory, which is added in the path like the central one. But both will append new paths to the environment without any rule of exclusion or version checking…A few workarounds exist to be able to express what packages (and version) an application needs to run, or to set up an isolated environment for it:
  • setuptools provides the install_requires mechanism where you can define dependencies directly inside the package, as a new metadata. It also provides a way to install two different versions of one package and let you pick by code or when the program starts, which one you want to activate.
  • virtualenv will let you create an isolated Python environment, where you can define your own site-packages. This allows you to make sure you are not conflicting with a incompatible version of a given package.
  • zc.buildout relies on setuptools and provides an isolated environment a bit similar in some aspects to virtualenv.
  • pip provides a way to describe requirements in a file, which can be used to define bundles, which are very similar to what zc.buildout provides.

But they all aim at the same goal : define a specific execution context for a specific application, and declare dependencies with no respect to other applications or to the OS environment…This proposal describes a solution that can be added to Python to provide that feature. A new file called a  Python Isolated Environment file (PIE file) can be provided by any  application to define the list of dependencies and their versions…’)

Posted in 2008, Distribution de logiciel, Génie logiciel, Gestion de version, package_management, python | Tagué: , , , , , , , | Leave a Comment »

Les nouveaux projets python: codereview/rietveld, Paver, Sphinx

Posted by patrick sur mai 12, 2008

Codereview

http://googleappengine.blogspot.com/2008/05/open-source-app-rietveld-code-review.html (« My first project as a Google engineer was an internal web app for code review. According to Wikipedia, code review is « systematic examination (often as peer review) of computer source code intended to find and fix mistakes overlooked in the initial development phase, improving both the overall quality of software and the developers’ skills. » Not an exciting topic, perhaps, but the internal web app, which I code-named Mondrian after one of my favorite Dutch painters, was an overnight success among Google engineers (who evidently value software quality and skills development :-). I even gave a public presentation about it: you can watch the video on YouTube.« )

http://groups.google.com/group/codereview-discuss?hl=en (« Discussion of the code review tool « Rietveld », http://codereview.appspot.com, open source at http://code.google.com/p/rietveld/.« )

Paver

http://www.blueskyonmars.com/projects/paver/ (« Paver is a Python-based build/distribution/deployment scripting tool along the lines of Make or Rake. What makes Paver unique is its integration with commonly used Python libraries. Common tasks that were easy before remain easy. More importantly, dealing with yourKevin Dangoor of SitePen applications specific needs and requirements is now much easier…Paver is currently alpha release software. There is one major feature (zc.buildout integration) planned for 1.0. At this point, it is unlikely that there will be significant changes to the pavement syntax, but there are no guarantees. If there are breaking changes, they will almost certainly be minor…Paver was created by . »)

http://groups.google.com/group/comp.lang.python.announce/browse_thread/thread/1b84e15dab7f02d6?hl=en (« With Version 0.7, Paver is now a full stand-in for the traditional distutils- or setuptools-based setup.py. Need to perform some extra work before an sdist runs? No problem:

@task
def sdist():
…move files around, etc….
call_task(« distutils.command.sdist »)

You put that in your pavement.py file and now running « paver sdist » will perform your logic and then run the distutils sdist command. Paver can even generate a setup.py so « python setup.py sdist » or « python setup.py install » work just as they always have. And to make it easier for people who don’t yet have Paver, you can include a small zip file that enables « python setup.py install » to work off of a pavement.py file even without Paver. Paver 0.7 has a bunch of new tools to help with project documentation (taking advantage of Georg Brandl’s Sphinx and also including Ned Batchelder’s Cog). Paver now includes much better docs, too.« )

http://nedbatchelder.com/code/cog/ (« Cog is a code generation tool. It lets you use pieces of Python code as generators in your source files to generate whatever code you need…Cog transforms files in a very simple way: it finds chunks of Python code embedded in them, executes the Python code, and inserts its output back into the original file. The file can contain whatever text you like around the Python code. It will usually be source code« )

Sphinx

http://sphinx.pocoo.org/ (« Sphinx is a tool that makes it easy to create intelligent and beautiful documentation for Python projects, written by Georg Brandl and licensed under the BSD license.It was originally created to translate the new Python documentation, but has now been cleaned up in the hope that it will be useful to many other projects. (Of course, this site is also created from reStructuredText sources using Sphinx!) »)

A voir:

http://en.wikipedia.org/wiki/Code_review (« Code reviews can often find and remove common vulnerabilities such as format string exploits, race conditions, memory leaks and buffer overflows, thereby improving software security. Online software repositories based on Subversion with Trac, Mercurial, GIT or others allow groups of individuals to collaboratively review code. Additionally, specific tools for collaborative code review can facilitate the code review process. »)

http://en.wikipedia.org/wiki/Software_distribution (« A software distribution is a bundle of a specific software (or a collection of multiple, even an entire operating system), already compiled and configured. It is generally the closest thing to a turnkey form of a usually GPL or open source source code for a software. It usually takes the form of either rpm, deb, tgz, msi, exe etc. installer and is downloadable from the Internet. These are also known as a Binary distributions…The Python programming language offers a distribution utility called distutils, which requires the creation of a setup.py configuration file »)

http://docs.python.org/dist/dist.html (« Distributing Python Modules »)

http://mail.python.org/mailman/listinfo/doc-sig (« For discussion of both the form and content of Python documentation. The SIG should work towards setting up a Python Documentation Project effort like the Linux Documentation Project. « )

http://fr.wikipedia.org/wiki/Documentation_logicielle (« La documentation logicielle est un texte écrit qui accompagne le logiciel informatique. Elle explique comment le logiciel fonctionne, ou comment on doit l’employer. Le terme peut avoir des significations différentes pour des personnes de différents profils.La documentation constitue une partie importante de l’ingénierie logicielle, qui est trop souvent négligéeDonald Knuth a insisté sur le fait que la documentation peut être un processus très difficile de réflexion après coup et a recommandé la programmation lettrée, qui consiste à écrire la documentation en même temps et en un même lieu que le code source et à l’extraire par des moyens automatiques. »)

http://fr.wikipedia.org/wiki/Gestion_de_configuration (« La gestion de configuration consiste à gérer la description technique d’un système (et de ses divers composants), ainsi qu’à gérer l’ensemble des modifications apportées au cours de l’évolution du système. La gestion de configuration est utilisée pour la gestion de systèmes complexes« )

http://en.wikipedia.org/wiki/Software_deployment (« Software deployment is all of the activities that make a software system available for use. The general deployment process consists of several interrelated activities with possible transitions between them. These activities can occur at the producer site or at the consumer site or both. Because every software system is unique, the precise processes or procedures within each activity can hardly be defined. Therefore, « deployment » should be interpreted as a general process that has to be customized according to specific requirements or characteristics. A brief description of each activity will be presented later. »)

Posted in Distribution de logiciel, Documentation, Génie logiciel, Gestion de projet, Gestion de version, python, Revue de code | Tagué: | Leave a Comment »

Mono : C#, IKVM, Java, Ironpython, Boo, OLPC, Silverlight/Moonlight, etc…

Posted by patrick sur janvier 11, 2008

Mono est encore bien méconnu aussi voici quelques liens intéressants pour en savoir plus:

http://en.wikipedia.org/wiki/Mono_(software) (« Mono is a project led by Novell (formerly by Ximian) to create an Ecma standard compliant .NETC# compiler and a Common Language Runtime. Mono can be run on Linux, BSD, UNIX, Mac OS X, Solaris and Windows operating systems…Mono’s current version is 1.2.6 (as of Dec 12, 2007). This version provides the core API of the .NET Framework as well as partial support for C# 2.0 and Visual Basic.NET and some C# 3.0 support« )

http://www.mono-project.com/Main_Page (« Le site du projet Mono. Mono provides the necessary software to develop and run .NET client and server applications on Linux, Solaris, Mac OS X, Windows, and Unix. Sponsored by Novell (http://www.novell.com), the Mono open source project has an active and enthusiastic contributing community and is positioned to become the leading choice for development of Linux applicationsFeatures

http://www.mono-project.com/Moma (« The Mono Migration Analyzer (MoMA) tool helps you identify issues you may have when porting your .Net application to Mono. It helps pinpoint platform specific calls (P/Invoke) and areas that are not yet supported by the Mono project. While MoMA can help show potential issues, there are many complex factors that cannot be covered by a simple tool. MoMA may fail to point out areas that will cause problems, and may point out areas which will not actually be an issue. »)

http://www.mono-project.com/Languages (« Multiple languages can be used with the Mono platform. The Mono project provides some compilers (C#, Basic, JScript) and there are both open source and commercial compilers that can be used with Mono…Mono-compatible compilers

  • C#: The main C# compiler of the Mono Project is mcs. We also have gmcs, which includes previews of several new C# features, including generics and nullable types. In the future, ‘gmcs’ will become the default compiler.
  • Java: Java applications can run in Mono, see the Java page for more details.
  • Boo: The Boo (http://boo.codehaus.org/Home) language is a scripting language similar to Python designed for the .NET Framework. For details on the particular language features see the Boo Language Features (http://boo.codehaus.org/Language+Features) page.
  • Python There are two possible choices here: PythonNet and IronPython.
    • PythonNet: Brian Lloyd (mailto:brian@No.Spam.zope.com) wrote a bridge to link the Python runtime with the .NET runtime. More information on the PS.NET project can be found here (http://pythonnet.sourceforge.net/). This uses the real Python engine and provides a bridge between the Python world and the .NET world to interoperate.
    • IronPython: is Jim Hugunin’s compiler for Python, it is a complete implementation of Python from scratch that compiles Python code into native CIL. More information is available on the IronPython site
  • GCC: In 2006, the Gcc4cil project was publicly announced. For now it supports the C language but it could be extended to support more gcc front ends..
  • Ruby Ruby.Net (http://www.plas.fit.qut.edu.au/rubynet/) from Queensland University. The compiler can be used to statically compile a Ruby source file into a verifiable .NET v2.0 assembly or it can be used to directly execute a Ruby source file (compile, load and execute). IronRuby (http://www.wilcob.com/Wilco/IronRuby.aspx) from Wilco Bauwer, includes an interactive Ruby Console and works with Mono.
  • …etc »)

http://www.mono-project.com/Java (« Mono is able to run Java code side-by-side with .NET as well as having Java and .NET object interoperate with each other. This is done with IKVM (http://www.ikvm.net) the Java VM implementation that runs on top of .NET and Mono. IKVM was developed by Jeroen Frijters for the .NET Framework. Zoltan Varga ported it to Mono and fixed the Mono runtime to support the features required by IKVM. The class library runtime for IKVM is the GNU Classpath which is rapidly advancing but not entirely complete. IKVM is fully supported by Mono and its part of the standard Mono package distribution. As it stands today, it is able to run popular applications like Eclipse and DerbyThere are two possible ways of using IKVM: one is to use it as a Just-in-Time compiler which translates the Java bytecodes into .NET Intermediate Language as it goes. But this means that at runtime you are compiling things twice: the Java-to-CIL JIT and the CIL-to-Native JITExposing .NET Libraries to Java Now, Gtk# is a .NET assembly (this is the ECMA lingo for « library »), and Java does not know anything about this. It is necessary first to generate some stubs for these classes to let the Java compiler knows about the types defined in the C# world. This is done using the netexp.exe program from IKVM, like this:

$ mono netexp.exe /mono/lib/mscorlib.dll
$mono netexp.exe /mono/lib/mono/gtk-sharp/gtk-sharp.dll
$ mono netexp.exe  /mono/lib/mono/gtk-sharp/glib-sharp.dll
$ mono netexp.exe /mono/lib/mono/gtk-sharp/atk-sharp.dll

The above commands basically « imports » all of the types and their definitions into something suitable for Java to consume, the result is:

$ ls *.jar
atk-sharp.jar  glib-sharp.jar  gtk-sharp.jar  mscorlib.jar

The netexp.exe program will import all of the types into the « cli » namespace. So if you had a class called « Gtk.Window », it will be exposed to Java as « cli.Gtk.Window »…How Complete is Mono/IKVM ? Mono and IKVM depend on the GNU Classpath, so it is as complete as open source Java, but you get the added advantage of accessing any Mono/.NET libraries as well. Today Mono/IKVM can run large applications like Eclipse, Jython and JBoss. Screenshot of Eclipse running on Mono . »)

http://tirania.org/blog/texts/gtkjava.html (« So today I figured it would be an interesting excercise to write a small Gtk# application with Java. To do this, I used IKVM the Java VM implementation that runs on top of .NET and Mono. There are two possible ways of using IKVM: one is to use it as a Just-in-Time compiler which translates the Java bytecodes into .NET Intermediate Language as it goes. But this means that at runtime you are compiling things twice: the Java-to-CIL JIT and the CIL-to-Native JIT. Gtk# is really a bad name. Because Gtk# is not limited to C#, any programming language in the .NET framework can use it today and because it covers more than only the Gtk API, it covers various other components of the GNOME Development Platform.« )

http://www.mono-project.com/Companies_Using_Mono

  • Novell (http://www.novell.com): Novell uses Mono for both client and server applications:
  • Otee (http://www.otee.dk): Their Unity 3D game modeling tool uses Mono so customers can build cross-platform video games. Their engineers have also spoken about their Mono use on the Mono mailing list. Read the Otee success story here.
  • Medsphere (http://medsphere.com): The Medsphere OpenVista software allows physicians to access complete patient health information at the point of care, and it runs on both Windows and Linux, thanks to Mono. Medsphere customers now have the flexibility to choose the OS that is right for them, without worrying about application availability.
  • Quantifi Solutions: (http://quantifisolutions.com) Specialists in complex financial instruments, Quantifi Solutions use Mono to mix C# and C++ code in their finance modeling software. Their financial models are refined constantly, so they need code that is easy to manage and easy to maintain, while still running fast enough to keep up with the markets. One of the Quantifi engineers discusses their implementation here in the mailing list archives
  • Versora : Windows-to-Linux migration specialists Versora used Mono and C# to produce a cross-platform tool that helps companies move system and application setttings and user data. Read their success story here.
  • Mainsoft uses Mono for their Grasshopper product which allows ASP.NET applications to be deployed on J2EE servers.
  • Gaia (http://ajaxwidgets.com) a company that offers Ajax-based ASP.NET controls under a dual-licensing scheme: GPL or proprietary. Learn more from them here .
  • Wikipedia (http://wikipedia.org): WikiPedia uses Mono for its search facilities. The indexing and the actual searching is done by Mono-based applications.

http://tirania.org/blog/archive/2008/Jan-04.html (« Les motivation de Miguel De Icaza pour développer mono et silverlight:…I have been using Linux as my main desktop operating system since 1992 and endured every missing feature, every broken driver, every broken X setup and every missing application since I started. I did so because it was free software, and I had decided that I wanted to run my entire system with free software. I felt that dog fooding Linux and improving Linux on a day-to-day basis would help improve this OS as opposed to improving a proprietary OS…From my perspective, it is crucial for Linux to have good support for Silverlight because I do not want Linux on the desktop to become a second class citizen ever again. Robert, you are asking those of us that use FOSS operating systems to « take one for the team » by not endorsing Silverlight, but yet, you are not willing to live among us. If you are going to preach, preach by exampleThe core of the debate is whether Microsoft will succeed in establishing Silverlight as a RIA platform or not. You believe that without Moonlight they would not have a chance of success, and I believe that they would have regardless of us. In fact, I believe strongly that it is part of Microsoft becoming more open and adapting itself to the multitude of shifts in this industry (open sourcing IronPython, IronRuby, the DLR, the JS library for ASP.NET, the MS-PL, the MS-RL, opening up their code, and so on)….Now, regardless of the strategic discussion about endorsing Silverlight, there are technicalities about Silverlight that make it a fascinating platform. I personally want to write cross platform web applications using C#, Boo, Python and Ruby. And that matters to me, and matters to others. And I have loved Silverlight since it embedded the CLR runtime. Nothing new there, you can read the gory details of my fascination from back then…What prevents anyone from taking the Moonlight source code, embracing it, extending it, innovate with it, prototype with it, and enter the same cycle that Linux, or web browsers have entered? Or someone turning it into a standard? Nothing. The only thing preventing it is lack of imagination. « )

http://wiki.laptop.org/go/Mono (« Sugar activities are usually written in Python using the Python Activity API. This page documents how it is possible to use Mono to write a Sugar activity. With Mono, you can use any underlying language like C# or Boo. »)

http://tquerci.blogspot.com/2008/01/mono-on-olpc-one-laptop-per-child.html (« After a lot of time I’m finally releasing the first version of Sugar assembly to « sugarize » a Mono application. This assembly is needed to integrate a Mono application with the Sugar environment so it can run on an OLPC device. This version allows creating applications that use GTK# forms created programmatically or by a Glade resource fileI have ported two applications to the OLPC device: GBrainy and MonkeysMemory. The first application is an interesting application to « improve » your brain, while the second one is a simple « Memory » game that I wrote to play with my children.« )

http://www.mono-project.com/Software

http://www.mono-project.com/Libraries

GRAPHIQUES

  • Gtk# (« This toolkit is a .NET binding for the Gtk+ toolkit. It is in active development, and there are various applications in the Mono world that use it (Monodoc, Monocov, Mono’s Debugger and various smaller applications, a more complete list is available on the Gtk# Wiki. Platforms: Unix, Windows, GPE, MacOS X (using the X server)« ).

Web Frameworks

CHARTING

  • NPlot (http://netcontrols.org/nplot/wiki/) is a free charting library for .NET and supports various kinds of graphic modes. It boasts an elegant and flexible API. NPlot includes controls for Windows.Forms, ASP.NET and a class for creating Bitmaps. A GTK# control is also available.
  • ZedGraph (http://zedgraph.org) ZedGraph is a set of classes, written in C#, for creating 2D line and bar graphs of arbitrary datasets. The classes provide a high degree of flexibility — almost every aspect of the graph can be user-modified. At the same time, usage of the classes is kept simple by providing default values for all of the graph attributes. The classes include code for choosing appropriate scale ranges and step sizes based on the range of data values being plotted.

SYSTEM PROGRAMMING

  • Mono.Addins is a generic framework for creating extensible applications, and for creating libraries which extend those applications. This framework is derived from the add-in engine used by MonoDevelop, although it has been completely rewritten and improved in many ways to make it more generic and easier to use. The MonoDevelop add-in engine was an improvement over the SharpDevelop engine, which took many ideas from the Eclipse add-in engine. Mono.Addins has been designed to be useful for a wide range of applications: from simple applications with small extensibility needs, to complex applications (such as MonoDevelop itself) which need support for large add-in structures.

http://www.mono-project.com/Moonlight («  Silverlight 1.1 (http://silverlight.net) is a new development technology for the Web created by Microsoft based on the CLR that augments it with a 2D retained graphics system and media playback engine and ships a subset of the standard .NET libraries. Currently the Moonlight project supports both Silverlight 1.0 (canvas + browser-based scripting) as well as 1.1 applications (canvas + ECMA CLI powered execution engine). Building an open source implementation on top of Mono is an obvious choice as Mono has most of the technologies required to implement it but is missing a few components. In this page we will track the work required and the design decisions involved in creating an open source version of itCurrently Moonlight is not packaged as it is still under heavy development. In the future we will provide package repositories for all major distributions through the OpenSUSE Build Service (http://build.opensuse.org) as well as a Mozilla extension that will make the installation experience similar to the one that Windows and MacOS users have today. « )

Posted in 2008, Distribution de logiciel, java, logiciel libre, openSUSE build service, packaging, python, Web Frameworks | Tagué: , , , | 1 Comment »