Python Usability Bugs: subprocess.Popen executable

subprocess.Popen seems to be designed as a "swiss army knife" of managing external processes, and while the task is pretty hard to solve in cross-platform way, it seems the people who have contributed to it did manage to achieve that. But it still came with some drawbacks and complications. Let's study one of these that I think is a top one from usability point of view, because it confuses people a lot.

I've got a simple program that prints its name and own arguments (forgive me for Windows code, as I was debugging the issue on Windows, but this works the same on Linux too). The program is written in Go to get single executable, because subprocess has special handling for child Python processes (another usability bug for another time).

>argi.exe 1 2 3 4
prog: E:\argi.exe
args: [1 2 3 4]

Let's execute it with subprocess.Popen, and for that I almost always look up the official documentation for Popen prototype:

subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)

Quite scary, right? But let's skip confusing part and quickly figure out something out of it (because time is scarce). Looks like this should do the trick:

import subprocess

args = "1 2 3 4".split()
p = subprocess.Popen(args, executable="argi.exe")
p.communicate()

After saving this code to "subs.py" and running it, you'd probably expect something like this :

> python subs.py
prog: E:\argi.exe
args: [1 2 3 4]

And... you won't get this. What you get is this:

> python subs.py
prog: 1
args: [2 3 4]

And that's kind of crazy - not only the executable was renamed, but the first argument was lost, and it appears that this is actually a documented behavior. So let's define Python Usability Bug as something that is documented but not expected (by most folks who is going to read the code). The trick to get code do what is expected is never use executable argument to subprocess.Popen:

import subprocess

args = "1 2 3 4".split()
args.insert(0, "argi.exe")
p = subprocess.Popen(args)
p.communicate()

>python suby.py
prog: argi.exe
args: [1 2 3 4]

The explanation for former "misbehavior" is that executable is a hack that allows to rename program when running subprocess. It should be named substitute, or - even better - altname to work as an alternative name to pass to child process (instead of providing alternative executable for the former name). To make subprocess.Popen even more intuitive, the args argument should have been named command.

From the high level design point of view, the drawbacks of this function is that it *does way too much*, its arguments are not always intuitive - it takes *a lot of time to grok official docs*, and I need to read it *every time*, because there are too many little important details of Popen behavior (have anybody tried to create its state machine?), so over the last 5 years I still discover various problems with it. Today I just wanted to save you some hours that I've wasted myself while debugging pymake on Windows.

That's it for now. Bonus points to update this post with link when I get more time / mana for it:

• [ ] people who have contributed to it
• [ ] it came with drawbacks
• [ ] have anybody tried to create its state machine?
• [ ] subprocess has special handling for child Python processes

Subprocessing. Why it won't be truly crossplatfrom

Every language and application stops to be crossplatform when it comes to executing other applications. Differences in a way various operating systems execute applications is so inconvenient that even new Python Subprocess API is not complete to keep you isolated from them.

Suppose you have an executable script.py in current directory that you want to execute. You would definitely need to execute the script through shell, because only shell knows which program should process it.

Windows

subprocess.Popen("script.py",shell=True)


POSIX

subprocess.Popen("./script.py",shell=True)


So you still need to keep in mind that to execute script from current directory you need to prefix the script with ./ under POSIX.

Subprocessing in Python

It may sound strange, but Python with all its low-level roots and crossplatform portability still doesn't have pythonic interface for working with child processes. Libraries that were born to do things unix-way usually fail under windows. Let's take a look into history of subprocessing at

Evil os.popen()

POSIX os.popen() call is at holy war with windows - one of the reasons there are almost no crossplatform Python scripts that wrap and control 3rd party processes. For example, study this os.popen() call reported in this thread:

import os,sys

out = os.popen("nslookup", "w")
out.write("google.com\n")
out.close()
On POSIX machine it will call nslookup, send a string on its input and nslookup will print the result on console:

> Server: 10.0.80.11
Address: 10.0.80.11#53

Non-authoritative answer:
Name: google.com
Address: 74.125.45.100
Name: google.com
Address: 74.125.67.100
Name: google.com
Address: 74.125.127.100
>
Under windows nslookup won't output anything on screen, but it will process the input correctly and that can be proved by redirecting output to a file:

import os,sys

out = os.popen("nslookup > output.txt", "w")
out.write("google.com\n")
out.close()
The workaround suggested in Python bug #1366 and used in Mercurial doesn't seems to work with Python 2.5.4 on XP. According to Mercurial patch the following code should give output on console:

import os,sys

out = os.popen('"nslookup 2> NUL:"', "w")
out.write("google.com\n")
out.close()
but it doesn't. It still executes though, but it is not what you want if the main thing your code does is output pagination. To work around this inherently evil feature of os.popen() smart guys invented subprocess module.

import subprocess

out = subprocess.Popen("nslookup", stdin=subprocess.PIPE)
out.stdin.write("google.com\n")
out.stdin.close()

According to Python bugtracker, subprocess is capable to guard you against malicious os.popen() in many different ways finding natural ways to deal with evilness #3144, evilness #602245, #...

..to be continued