Why Git suxx

$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.

$ git checkout https://github.com/gratipay/grtp.co/pull/178
 Checkout aborted
 There's no file https:\github.com\gratipay\grtp.co\pull\178 at HEAD

$ git switch https://github.com/gratipay/grtp.co/pull/178
git: 'switch' is not a git command. See 'git --help'.

Solution

Download https://github.com/tj/git-extras/blob/master/bin/git-pr to your git --exec-path.

Python Usability Bugs: Formatting argparse subcommands

Suppose you want to build a tool with a simple interface:

usage: sub <command>

commands:

  status -  show status
  list   -  print list

Python proposes to use argparse module. And if you follow documentation, the best you can get will be this output:

usage: sub {status,list} ...

positional arguments:
  {status,list}
    status       show status
    list         print list

And it you implement proper formatting, your code will look like this:

import argparse
import sys

class CustomHelpFormatter(argparse.HelpFormatter):
  def _format_action(self, action):
    if type(action) == argparse._SubParsersAction:
      # inject new class variable for subcommand formatting
      subactions = action._get_subactions()
      invocations = [self._format_action_invocation(a) for a in subactions]
      self._subcommand_max_length = max(len(i) for i in invocations)

    if type(action) == argparse._SubParsersAction._ChoicesPseudoAction:
      # format subcommand help line
      subcommand = self._format_action_invocation(action) # type: str
      width = self._subcommand_max_length
      help_text = ""
      if action.help:
          help_text = self._expand_help(action)
      return "  {:{width}} -  {}\n".format(subcommand, help_text, width=width)

    elif type(action) == argparse._SubParsersAction:
      # process subcommand help section
      msg = '\n'
      for subaction in action._get_subactions():
          msg += self._format_action(subaction)
      return msg
    else:
      return super(CustomHelpFormatter, self)._format_action(action)


def check():
  print("status")
  return 0

parser = argparse.ArgumentParser(usage="sub <command>", add_help=False,
             formatter_class=CustomHelpFormatter)

subparser = parser.add_subparsers(dest="cmd")
subparser.add_parser('status', help='show status')
subparser.add_parser('list', help='print list')

# custom help messge
parser._positionals.title = "commands"

# hack to show help when no arguments supplied
if len(sys.argv) == 1:
  parser.print_help()
  sys.exit(0)

args = parser.parse_args()

if args.cmd == 'list':
  print('list')
elif args.cmd == 'status':
  sys.exit(check())

Here you may see the failure of OOP (object oriented programming). The proper answer to this formatting problem is just to define a data structure for command line help in JSON or similar format and let people dump and process it with templates. Once option definition is parsed, the information there is static, so there is no need in those intertwined recursive method calls. So just do it in 2 pass - get dataset and render template.

Python Usability Bugs: subprocess.Popen executable

subprocess.Popen seems to be designed as a "swiss army knife" of managing external processes, and while the task is pretty hard to solve in cross-platform way, it seems the people who have contributed to it did manage to achieve that. But it still came with some drawbacks and complications. Let's study one of these that I think is a top one from usability point of view, because it confuses people a lot.

I've got a simple program that prints its name and own arguments (forgive me for Windows code, as I was debugging the issue on Windows, but this works the same on Linux too). The program is written in Go to get single executable, because subprocess has special handling for child Python processes (another usability bug for another time).

>argi.exe 1 2 3 4
prog: E:\argi.exe
args: [1 2 3 4]

Let's execute it with subprocess.Popen, and for that I almost always look up the official documentation for Popen prototype:

subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)

Quite scary, right? But let's skip confusing part and quickly figure out something out of it (because time is scarce). Looks like this should do the trick:

import subprocess

args = "1 2 3 4".split()
p = subprocess.Popen(args, executable="argi.exe")
p.communicate()

After saving this code to "subs.py" and running it, you'd probably expect something like this :

> python subs.py
prog: E:\argi.exe
args: [1 2 3 4]

And... you won't get this. What you get is this:

> python subs.py
prog: 1
args: [2 3 4]

And that's kind of crazy - not only the executable was renamed, but the first argument was lost, and it appears that this is actually a documented behavior. So let's define Python Usability Bug as something that is documented but not expected (by most folks who is going to read the code). The trick to get code do what is expected is never use executable argument to subprocess.Popen:

import subprocess

args = "1 2 3 4".split()
args.insert(0, "argi.exe")
p = subprocess.Popen(args)
p.communicate()

>python suby.py
prog: argi.exe
args: [1 2 3 4]

The explanation for former "misbehavior" is that executable is a hack that allows to rename program when running subprocess. It should be named substitute, or - even better - altname to work as an alternative name to pass to child process (instead of providing alternative executable for the former name). To make subprocess.Popen even more intuitive, the args argument should have been named command.

From the high level design point of view, the drawbacks of this function is that it *does way too much*, its arguments are not always intuitive - it takes *a lot of time to grok official docs*, and I need to read it *every time*, because there are too many little important details of Popen behavior (have anybody tried to create its state machine?), so over the last 5 years I still discover various problems with it. Today I just wanted to save you some hours that I've wasted myself while debugging pymake on Windows.

That's it for now. Bonus points to update this post with link when I get more time / mana for it:

• [ ] people who have contributed to it
• [ ] it came with drawbacks
• [ ] have anybody tried to create its state machine?
• [ ] subprocess has special handling for child Python processes

ANN: hexdump 3.2 - packaged and pretty executable

https://pypi.python.org/pypi/hexdump

The new version doesn't add much to the API, but completes a significant research about executable .zip packages with proof of concept distribution that is both an installable Python package that you can upload to PyPI and an executable .zip archive.

The benefit is that you may drop .zip package downloaded from PyPI into your source code repository and invoke it from your helper scripts as python hexdump-3.2.zip Keeping dependencies together with application source code in repository may seem like a bad idea, but we find it useful at Gratipay to setup project from checkout in offline mode with no internet connection.

The solution required three modifications to distutils setup.py (see sources):

• force sdist command to produce .zip file
• strip extra dir created at PyPI package root
• add __main__.py to make .zip executable

If you think that it is cool and should be promoted further, feel free to test Gratipay to send me 0.01+, and I'll get the signal to spend more time on reversing native distutils/packaging gotchas.

Shipping Python tools in executable .zip packages

UP201501: Found out how to force setup.py create .zip instead of .tar.gz

This is about how to make Python source packages executable. As a side effect, this also explains, how to run invoke's tasks.py with a copy of invoke .zip shipped with your source code, but without installing it.

You know, Python can execute .zip files. As simply as:

$ wget https://pypi.python.org/packages/source/h/hexdump/hexdump-3.1.zip
$ python hexdump-3.1.zip
/usr/bin/python: can't find '__main__.py' in 'hexdump-3.1.zip'

Well, you need to place '__main__.py' into the root. It will then import what it needs and do something. The only problem with .zip source packages is that their structure (if created with standard Python tools) includes one additional directory at the top:

`-- hexdump-3.1
    |-- PKG-INFO
    |-- README.txt
    |-- hexdump.py
    |-- hexfile.bin
    `-- setup.py

So, even if you place `__main__.py` into the root, it won't be able to import file from `hexdump-3.1` subdirectory.. unless you prepare for it. Here is the solution:

import os
import sys

# add package .zip to python lookup path
__dir__ = os.path.dirname(__file__)
path = os.path.join(__dir__, 'hexdump-3.1')
sys.path.insert(0, path)

import hexdump
msg = hexdump.dehex("48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21")
print(msg)

Now if you run it, it will print the obvious.

$ python hexdump-3.1.zip
Hello, World!

In Python 3, the result will be wrapped into b'' string. But that's it!

The next hexdump version will likely to ship as executable .zip package

Where you may need it?

It may be useful for scripts automation if you're "vendorizing" dependencies (ship them with your code). It started with need to fix a broken link on Gratipay site. The Gratipay codebase is in public domain clear of any NDA, CLA and all other BS, and that makes it great for people to learn, reuse and enhance. I wish Python internal projects possessed these properties, but PSF politics is another topic.

Gratipay inside site uses make. It doesn't need powerful build tool, such as scons, so I tried to replace it with some simple task automation utility to be more cross-platform. I chose invoke, because of my previous experience with fabric for remote control, and because I saw previous attempts in another Gratipay repository. The necessary condition for new tool is that user experience should not degrade for make users, so main invoke's tasks.py script needed to be self-executable.

I completed the proof of concept by making invoke .zip package executable with the method above, placed it into vendor/ directory, and tuned tasks.py to reexecute itself with invoke .zip This also solved a potential problem with currently unstable invoke API.

tasks.py was modified as following:

import sys
sys.path.insert(0, 'vendor/invoke-0.9.0.zip')

from invoke import task, run

# ... tasks go here

if __name__ == '__main__':
  import sys
  from invoke import cli
  cli.main()

Bonus points

There is one more thing, which is better explained with hexdump example. hexdump package ships `hexfile.bin` data file used for testing. hexdump.py looks for it in its directory (__file__ dirname), and this lookup fails when hexdump.py is in archive. I'd say Python could provide some kind of an API to deal with "virtual import filesystem" (with watchers to track who and when modifies sys.path), but I digress. If you modify the `__main__.py` from the first chapter above to run tests - just call hexdump.runtest() - the execution will finally fail with the following error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "hexdump-3.1.zip/__main__.py", line 13, in
  File "hexdump-3.1.zip/hexdump-3.1/hexdump.py", line 311, in runtest
IOError: [Errno 20] Not a directory: '/root/hexdump-3.1.zip/hexdump-3.1/hexfile.bin'

(yes, I am hexdumping under root). The problem above was explained by Radomir, and I feel like the post is already tool long to write about it myself.

More bonus points

In theory it is possible to hack setup.py to produce executable source .zip package. By default, setup.py sdist command on Linux creates tar.gz files. So, first it needs to be forced to create .zip file. This can be done by overriding default --formats option:

# Override sdist to always produce .zip archive
from distutils.command.sdist import sdist as _sdist
class sdistzip(_sdist):
    def initialize_options(self):
        _sdist.initialize_options(self)
        self.formats = 'zip'

setup(
    ...
    cmdclass={'sdist': sdistzip},
)

I posted relevant documentation links to this StackOverflow question. The rest - how to inject __main__.py into the root of packed .zip - is yet to be covered, and I am not ready to research it just yet. Some hints may be provided by these answers.

Usability conclusion

I am actually quite happy with the ability to execute python .zip packages (which gave a lot of motivation to write this post), because previously I had to care about how to tell people that a tool is a single script, which can be downloaded from repository or from some dedicated download site (I mean Google Code, which does not support this anymore). I also don't trust my own server for downloads, because one day I found some "benzonasos" running there and I don't even have a slightest idea how it got there.

But now it becomes possible to just download and run the stuff from PyPI to test how it works without messing with creating and activating virtualenvs, and installation of package inside. Of course, if your .zip package has dependencies, they still need to be present on your system, but that's still a time saver. Oh, and that means that my tools can now consist of several modules inside.