"…mais ce serait peut-être l'une des plus grandes opportunités manquées de notre époque si le logiciel libre ne libérait rien d'autre que du code…"

Nouvelles de python pour la semaine du 14 au 20 janvier 2008

Posted by patrick sur janvier 21, 2008

En vrac, quelques nouvelles de python

http://highscalability.com/youtube-architecture (« – encore sur la liste Tutor, on apprend que YouTube emploie Python: YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google?

Information Sources

  • Google Video


  • Apache
  • Python
  • Linux (SuSe)
  • MySQL
  • psyco, a dynamic python->C compiler
  • lighttpd for video instead of Apache

    What’s Inside?

    The Stats

  • Supports the delivery of over 100 million videos per day.
  • Founded 2/2005
  • 3/2006 30 million video views/day
  • 7/2006 100 million video views/day
  • 2 sysadmins, 2 scalability software architects
  • 2 feature developers, 2 network engineers, 1 DBA
  • Web Servers

  • NetScalar is used for load balancing and caching static content.
  • Run Apache with mod_fast_cgi.
  • Requests are routed for handling by a Python application server.
  • Application server talks to various databases and other informations sources to get all the data and formats the html page.
  • Can usually scale web tier by adding more machines.
  • The Python web code is usually NOT the bottleneck, it spends most of its time blocked on RPCs.
  • Python allows rapid flexible development and deployment. This is critical given the competition they face.
  • Usually less than 100 ms page service times.
  • Use psyco, a dynamic python->C compiler that uses a JIT compiler approach to optimize inner loops.
  • For high CPU intensive activities like encryption, they use C extensions.
  • Some pre-generated cached HTML for expensive to render blocks.
  • Row level caching in the database.
  • Fully formed Python objects are cached.
  • Some data are calculated and sent to each application so the values are cached in local memory. This is an underused strategy. The fastest cache is in your application server and it doesn’t take much time to send precalculated data to all your servers. Just have an agent that watches for changes, precalculates, and sends….)
  • =============================
    http://www.oreillynet.com/onlamp/blog/2008/01/pymotw_threading.html (« Doug Hellman’s Python Module of the Week is all about threading. »)

    http://www.python.org/dev/peps/pep-0369/ (« This PEP proposes enhancements for the import machinery to add post import hooks. It is intended primarily to support the wider use of abstract base classes that is expected in Python 3.0.The PEP originally started as a combined PEP for lazy imports and post import hooks. After some discussion on the python-dev mailing list the PEP was parted in two separate PEPs « )

    http://www.python.org/dev/peps/pep-3119/ (… Abstract This is a proposal to add Abstract Base Class (ABC) support to Python 3000. It proposes:

    • A way to overload isinstance() and issubclass().
    • A new module abc which serves as an « ABC support framework ». It defines a metaclass for use with ABCs and a decorator that can be used to define abstract methods.
    • Specific ABCs for containers and iterators, to be added to the collections module.

    Much of the thinking that went into the proposal is not about the specific mechanism of ABCs, as contrasted with Interfaces or Generic Functions (GFs), but about clarifying philosophical issues like « what makes a set », « what makes a mapping » and « what makes a sequence ». There’s also a companion PEP 3141, which defines ABCs for numeric types. Acknowledgements Talin wrote the Rationale below [1] as well as most of the section on ABCs vs. Interfaces. For that alone he deserves co-authorship. The rest of the PEP uses « I » referring to the first author. In the domain of object-oriented programming, the usage patterns for interacting with an object can be divided into two basic categories, which are ‘invocation’ and ‘inspection’. Invocation means interacting with an object by invoking its methods. Usually this is combined with polymorphism, so that invoking a given method may run different code depending on the type of an object. Inspection means the ability for external code (outside of the object’s methods) to examine the type or properties of that object, and make decisions on how to treat that object based on that informationone of the criticisms of inspection by classic OOP theorists is the lack of formalisms and the ad hoc nature of what is being inspected. In a language such as Python, in which almost any aspect of an object can be reflected and directly accessed by external code, there are many different ways to test whether an object conforms to a particular protocol or not. For example, if asking ‘is this object a mutable sequence container?’, one can look for a base class of ‘list’, or one can look for a method named ‘__getitem__’. But note that although these tests may seem obvious, neither of them are correct, as one generates false negatives, and the other false positivesThis PEP proposes a particular strategy for organizing these tests known as Abstract Base Classes, or ABC. ABCs are simply Python classes that are added into an object’s inheritance tree to signal certain features of that object to an external inspector. Tests are done using isinstance(), and the presence of a particular ABC means that the test has passed…Specification The specification follows the categories listed in the abstract:

    • A way to overload isinstance() and issubclass().
    • A new module abc which serves as an « ABC support framework ». It defines a metaclass for use with ABCs and a decorator that can be used to define abstract methods.
    • Specific ABCs for containers and iterators, to be added to the collections module. »)

    http://www.webfaction.com/demos/django (« How to install a real-life Django application on our servers. Watch Demo« )

    http://www.vulnerabilite.com/securite-open-source-audit-faille-coverity-actualite-20080110223332.html (« Doté d’une enveloppe initiale de 300 000 dollars, le programme Open Source Hardening Project a été lancé en mars 2006 avec pour mécène le département américain de la sécurité intérieure ( DHS ). Ce programme dont la conduite a été confiée à l’Université de Standford et à la société Coverity, a pour objectif de passer au crible le code de plusieurs logiciels open source (écrits en C et C++) parmi les plus populaires afin d’identifier les failles présentes, et ainsi contribuer au renforcement de leur sécurité... La société basée à San Francisco a par ailleurs défini plusieurs niveaux dans le processus de correction de bugs, soit 3 au total et a annoncé mardi que 11 projets open source ont atteint le niveau le plus élevé ( Rung 2 ). En raison des efforts fournis pour assurer de façon proactive l’intégrité et la sécurité des applications développées, Coverity indique que les entreprises et particuliers peuvent choisir avec encore plus de confiance les logiciels open source suivants : Amanda, NTP, OpenPAM, OpenVPN, Overdose, Perl, PHP, Postfix, Python, Samba et TCL... Si tous les logiciels contrôles ne peuvent pas encore prétendre au Rung 2, certains le pourront dans les prochains mois mais en attendant, Firefox ou encore Apache restent cantonnés au Rung 1 (86 projets), voire au Rung 0 (173 projets) comme nmap ce qui signifie que les bugs détectés n’ont pas encore été corrigés« )

    http://holdenweb.blogspot.com/2008/01/resolver-released.html (« In yet another piece of good news for Python fans, Resolver Systems have released their first product. It’s a spreadsheet that you can manipulate in Python, and it’s received a lot of interest from the financial communities on both sides of the Atlantic. [You would not believe how much of the world’s financial dealing is controlled and managed by spreadsheets; it’s really quite scary]. Resolver One is written in IronPython by a team which includes the Fuzzyman (occasionally also known as Michael Foord), of Voidspace Techie Blog fame, and it’s currently the largest product developed in that languages, with a total codebase (including tests) of over 100 kloc. The company is bravely making the product available (though not, I believe, as open source) free for non-commercial use.« )

    Laisser un commentaire

    Entrez vos coordonnées ci-dessous ou cliquez sur une icône pour vous connecter:

    Logo WordPress.com

    Vous commentez à l'aide de votre compte WordPress.com. Déconnexion / Changer )

    Image Twitter

    Vous commentez à l'aide de votre compte Twitter. Déconnexion / Changer )

    Photo Facebook

    Vous commentez à l'aide de votre compte Facebook. Déconnexion / Changer )

    Photo Google+

    Vous commentez à l'aide de votre compte Google+. Déconnexion / Changer )

    Connexion à %s

    %d blogueurs aiment cette page :