Python C API/ABI compatibility report

UPDATE: There is now an official thread on python-dev.

Upstream Tracker is an open source (GPL) tool that allows to track API/ABI changes between releases of C/C++ libraries. I asked Andrey Ponomarenko, who is the main maintainer of the project, to add Python to the list and here is the result:

http://upstream-tracker.org/versions/python.html

Hope you like it.

ANN: xtrace 0.5 - indented function trace in Xdebug format

Xdebug is a PHP tool that allows to trace how PHP code is executed. Today I release a tool called xtrace into public (and more specifically into public domain), which allows to get the same (or at least very identical) output, but for Python.

Few years ago inability to trace function calls in Python came as a showstopper. I decided to write a familiar tool for Python. More than that - I wanted to integrate it into Spyder. But a year ago xtrace itself faced with showstopper. The showstopper was the behavior of execfile function, that I could not get right at that time, because of the docs, of my expectations and poor knowledge of English. Maybe there is a flaw in my cognitive abilities, but I tried to get at this problem several times and failed. Until recently some hackers from ZenSecurity team brought a concept known an pyjail to my attention. The challenge to prove that pyjail concept is impossible allowed me to concentrate on gory details of execfile works and knowing that documentation is totally confusing for my, I found the time to set my own experiments. You can read them at the link I've given above as well as some analysis why documentation that actually includes all the details can be bad and confusing.

The xtrace was basically broken for three years, starting from the version 0.2 - the day I put execfile() call from root to the xtrace module to the main() function. This placement changed the execfile() behavior, and while trying to debug that I also run into confusing dynamic behavior of dictionary returned by locals(). Opened can of worms made those parasites to completely consume my brain, causing much anger and frustration to be spilled around execfile() and locals() concepts over into Python lists. It is kind of relief now that I can name all the problems, analyse them and look back as enlightened. Being jobless I had a plenty of time to investigate, but I really don't want anyone to enter that state of confusing and helplessness that I had a year ago.

Hopefully, my experience with xtrace will clear the confusion for those who will try to use exec type abilities of Python for developing their own tools. Maybe it will result in a better Python API in the future, with better documentation and position-independent behavior.

I am interested to know the feedback that you can leave in xtrace tracker, such as if the output really matches PHP behavior, if it is accepted by PHP tools and how it behaves in different scopes of Python. It is interesting to convert it to Spyder plugin and see the usage in other tools, but I realize that I may not have time for that. The next focus for me is to add an easy API to xtrace to enable people to write they own tracers more easily. Focus on UX and everything else will come.

Program config as a DNA strand

This is a technical followup to the post about mind-altering programming languages, which concentrates, iterates and extends on the abstract DNA part.

Do you know how DNA looks like from the point of view of Python programmer?
DNA is a list. Of genes, and fillers in between.

    [ GENE, GENE, GENE, None, GENE, STUFF, GENE, .... ]

By the analogy with software, one GENE is one option. Here None or STUFF is something that is not recognized and not used in the process right now, but may be used later even in a different place for a different purpose. This leads to very interesting properties of configuration described as a DNA strand.

Property One: Everything is Optional and Configurable

Usually it is a tough choice what to include into your program config, because an option a day keeps users away. But with DNA config it doesn't matter - you can encode everything - every bit of flexibility you wanted, every feature-creeper, forward-thinking matter. Don't bother yourself with restrictions - include everything.

Property Two: Indefinite Logical Groups and Namespaces

To make sense out of tons of information in a DNA strand, you need to concentrate only on the parts that are relevant to you. This is done by "masking" (or marking) parts of the DNA code according to your current task. When you apply a mask, all GENEs of DNA that are not used are cleared. You can have multiple masks for different programs. An awesome feature-creeper would be to figure out mask automatically during the program run (program explicitly marks sequences what it reads and what it skips).

Property Three: Non-linear Option Identification

Because DNA is a list, you may think that you need to know option position to access it. This is not necessary. Index in a DNA is the most straightforward way to lookup an option value, and mask is a tool to help explore and identify its position. But the true power comes from the fact that a option can be identified by just analyzing content. Option can be a single GENE in specific format, or it can be identified as a sequence of GENEs that pass some validation logic. This makes is possible to write generic configuration analyzers, which can read these sequences to detect configuration problems, patterns and meaningful values to produce nice visualization out of them.

Property Four: Application Types

In real world™ DNA is packed in chromosomes. Humans have 46. Potatoes possess 48. Zombies are somewhere in between. In a virtual world of coding it makes sense to use one chromosome per application type. For example, every Python web framework has a lot of common parts, common features and minor details in these features, so a variety of such frameworks can be encoded with a single DNA strand.

Exercise

It may sound too abstract, but try to think how to build a shared DNA strand for two or more mini web application frameworks in Python. Just remember that DNA code includes both hardcoded application features and modifiable configuration options. And as usual look at BoxCar 2D for an inspiration.

Formatting API Anti-Pattern

I was meditating over subset of core Python API that's dedicated to self-inspection in a running script. This API consists of inspect + traceback + sys.exc_*  + magical locals() and ideally should give you a total understanding of Python state at a given point in your program. But that doesn't happen, because things are complicated, and here is why..

I wanted to get a chain of callers for the function I was debugging. I don't like pdb at all - it is so outdated for 2013 that deserves a separate GSoC project.  I prefer to just insert debug statements into the code and after several weeks and attempts I came up with a helper to get the name of the caller that I used to log and debug such complex projects as SCons and Spyder. I am not as smart as everybody else, but even I didn't expect to spend weeks researching this basic stuff.

I think the reason historically is that people tried to produce the meaningful data using functional approach without taking due attention on the data quality itself. Is the data complete, sufficient and accessible? You can not say that from the manual. The manual (see inspect) encourages you to hack your way in the debris of various bits of data that elusively fail to create a complete picture - the puzzle just doesn't solve, but you can't see this, because you don't know what is the structure of the Python state at a given point in your program and what to look for. The API encourages you to find functions that join and analyse the bits, you fail and try again and in the end you come up with a function filled with uncertainties, that just does some formatting magic.

"Formatting API Anti-Pattern" may not be the best term for that, but it is the API (data structures, functions and other stuff) so complex and incomplete that you can not see this, and instead of that forced to write formatting functions just to make sense of this incomplete data. We are so used to the fact that "API provides abilities" that completely miss the point that it also "provides limitation", and the data in API can be incomplete, insufficient (incomplete for certain tasks) and not accessible (code becomes very complicated to handle).

How to fight this? Think about data first. What data should be available for inspection at a given point in your program when this program is paused? Can you see this data in the manual? If not, then you're dealing with "Formatting API".

User Pattern: Button Status

Introduction into User Patterns

Many of you know about Design Patterns of software development, invented to make complicated solutions look less complicated. The problem with Design Patterns is that they describe solutions to coding problems - not for user problem. People abusing Design Patterns tend to align the functionality of application to patterns, sacrificing features users wanted. The application that looks well-designed in the eyes of developers, in fact makes even simple features hard to implement or just takes too much time. As time passes, users become dissatisfied and eventually lose all interest in the product.

People tried to approach the problem from the other end with User Interface Design guidelines. This often made developers unhappy, because their Design Patterns don't explain how to work with web and Ajax application where "business logic" becomes shared between client and server. Developers wanted to see how different frameworks approach this stuff and to make the development process more intuitive. For this reason there should be a way to see how some typical features/functionality are implemented in different frameworks and this should be more detailed than a wiki or a blog in 15 minutes.

So, let me introduce User Patterns. User Pattern is a very abstract story, missing details about design decisions, but detailed enough to provide implementation and see how _simple_, _extensible_ and _maintainable_ this implementation is.

Button Status User Pattern

User story: I want a page with a button and a status field displayed. When I click the button, the status should be updated.

Description: There are five levels in the quest for the implementation of this pattern.
Level 1. The status is blank - it is fetched/calculated only when the button is pressed.
Level 2. The status is initially available - and it is refreshed/calculated when the button is pressed.
Level 3. When refresh/recalculation fails, the error value is displayed in status field.
Level 4. When refresh/recalculation fails, the error value is displayed in status field along with user friendly message in separate field
Level 5. The status value is stored in application database - when the button is pressed, it is recalculated, saved and only when displayed to the user
Level 6. When recalculation/save or fetch operation fails, user should be presented with friendly message explaining the error and further action (i.e. report problem link)

I'm looking forward to create samples for AppEngine webapp, classic Django and Django-nonrel. Feel free to share your snippets.

URL parameters in action method of HTML form are lost for GET request

Surprisingly how experienced web-developers may miss some basic nuances of form processing that exist for many-many years. One of them is the fact that URL parameters added to action attribute of <form> element are lost for GET requests.

Example:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Query string from action attribute is killed for GET request</title>
</head>
<body>

  <form name="getSearchForm" action="?missed=one" method="get">
    <input type="text" name="q" value="s" />
    <button class="searchButton" type="submit">GET</button>
  </form>

  <form name="postSearchForm" action="?missed=one" method="post">
    <input type="text" name="q" value="s" />
    <button class="searchButton" type="submit">POST</button>
  </form>

</body>
</html> 
The parameter "missed=one" is missing from URL after clicking a button on a form with GET send method. It is present for POST form though.

Google Translate for Trac timeline comments

JavaScript that translates changeset comments in Trac timeline using Google Language API. How to use it with Trac should be well-described at this Trac-Hacks page. Here I just want to state that this code can be reused for other .html pages.

This code is used at http://farmanager.rainforce.org/timeline

Just do not forget to add jQuery and the following line to your header.

<script type="text/javascript" src="http://www.google.com/jsapi"></script>

Here goes the main code:

// Translate changeset comments in Trac timeline
// Using Google Language API version 1

// Code in public domain
// techtonik // rainforce.org


// wrap the code to use $ as jQuery function name
(function($){


// setup language parameters

var srclang = "ru";
var tgtlang = "en";

var gtrans_btn_text = "Translate russian comments";

var srclang_a_z = Array();
srclang_a_z["en"] = /[a-z]+/i;
srclang_a_z["ru"] = /[Р°-С_]+/i;


// load Google API

google.load("language", "1");


// make this anonymous function executed by jQuery when document is loaded
jQuery(document).ready(function() {

    // add "powered by Google" logo into footer
    $('<p id="googleattr" class="left"></p>').insertAfter("#footer p.left:last")
    google.language.getBranding('googleattr');

    // add trac_timeline_translate() button to timeline menu
    var gtrans_btn = $("<input type='button'/>")
          .attr({
              id: "gtrans_btn",
              value:gtrans_btn_text
          })
          .click(function() {
              trac_timeline_translate();
              gtrans_btn.attr("disabled", "disables");
              return false;
          });

    $("#prefs").append("<hr/>");
    $("#prefs")
      .append(
         $("<div align='center'></div>").append(gtrans_btn)
      )
})


function trac_timeline_translate() {
    // translate changeset messages
    g_trans_nodes( $("dd.changeset") );
}


function g_trans_nodes(nodes) {
  // translate nodes one by one
  // todo: group many small nodes into one packet < 5000 bytes

  var transTexts = Array();
  var transNodes = Array();

  for (i=0; i<nodes.size();i++) {
      var text = nodes[i].innerHTML;

      if (text.match(srclang_a_z[srclang])) {
          transTexts.push(text);
          transNodes.push(nodes[i]);
      }
  }

  for (i=0; i<transTexts.length;i++) {
      function makeTransNodeClosure(target_node) {
          // closure to save node parameter to be accessible
          // in callback function
          var node = target_node; // local variable
          var trans_callback = function (result) {
              node.innerHTML = result.translation;
          }
          return trans_callback; // return function
      }
      
      google.language.translate(transTexts[i], srclang, tgtlang, makeTransNodeClosure(transNodes[i]));
    }
}


})(jQuery);