An ideal options / arguments parsing library for Python

One of primary areas of Python applications is helper scripts and system utilities that are often called from command line. Even programming in Python itself is not comfortable without interactive mode at hand. That's why argument parsing libraries are important to some degree to many Python developers.

There once was getopt library, but it was awkward compared to newer optparse module, but even optparse nowadays is not enough for some who is willing to replace it with argparse. optparse is a kind of "stable minimal API" that is not easy to push a change to. As a person who has a lot of Python command tools at hand I often feel that I need patched version of optparse, but I can't afford managing dependencies or inserting a bulk of code into 5 minute script that should be as minimal and clear as possible for easy maintenance. So I wouldn't mind against more featured argparse library.

However, argparse can only be useful in standard library if it will be:

1. Minimalistic. Like optparse won't take much brainspace to avoid referencing documentation for 80% of the stuff
2. Easily extensible
3. Contains copy/paste recipes for popular usages
   

On the first look argparse is not minimalistic - even in constructor there are a lot of params that won't be used in 80% of cases and that could be easily moved into configuration methods.

Is it easily extensible? It allows to specify formatter class, conflict_handler class, but it seems like you have very limited control over what will be matched as option and what as an argument. There are two options to constructor to define options prefix - prefix_chars and fromfile_prefix_chars, but they are awkward. I wonder if I can provide /longarg on windows in addition to --longarg leaving also /s and -s options? It could be better done with the same custom class, which could be provided in recipes or as an optional import.

There are no recipes in argparse. That's bad, because that means the library is designed as a comprehensive solution to all command-line problems, but it still can't help me make the majority of my scripts more convenient without too much overhead.

argparse is not a huge leap from optparse as the latter once was for getopt in my opinion. For any argument parsing library to become a quality leap from optparse, it should allow an easy way to build svn like command line interfaces with the following features:

1. calling without arguments shows tool name, version and usage info
2. usage info contains either command list, options list or both
3. "tool.py help " is equivalent to "tool.py -h"
4. remove ugly default optparse help for options with arguments
   
   

   -f FILE, --file=FILE  write report to FILE

   SVN is fine

   -r [--revision] ARG      : ARG (some commands also take ARG1:ARG2 range)

   or Mercurial 
   

   -l --logfile       read commit message from  
   

5. cross-platform optional paging
   
   

This is a functionality of an ideal argument parsing library. Feel free to add your own user story.

Subprocessing. Why it won't be truly crossplatfrom

Every language and application stops to be crossplatform when it comes to executing other applications. Differences in a way various operating systems execute applications is so inconvenient that even new Python Subprocess API is not complete to keep you isolated from them.

Suppose you have an executable script.py in current directory that you want to execute. You would definitely need to execute the script through shell, because only shell knows which program should process it.

Windows

subprocess.Popen("script.py",shell=True)


POSIX

subprocess.Popen("./script.py",shell=True)


So you still need to keep in mind that to execute script from current directory you need to prefix the script with ./ under POSIX.

Lost network interfaces after moving Debian virtualbox to another host

Every time Debian virtual machine migrates to a new host, it loses its network interfaces. Debian guest detects new virtual hardware and assigns new aliases losing IP address and other settings. The easiest way is to rename new interface back to original way by editing hard to find config file /etc/udev/rules.d/z25_persistent-net.rules (works for etch)

sudo vim /etc/udev/rules.d/z25_persistent-net.rules
Do not forget to check actual settings for network interfaces with

sudo ifconfig -a
and settings that are applied on each restart in

sudo vim /etc/network/interfaces
Instead of renaming network interfaces in Debian it might be easier to record MAC numbers in Notes for virtual machine and setup virtual network interfaces in host accordingly prior to starting imported machine.

Note about using "App paths" for installers

App paths is recommended (or better say - preferred) way to register application in Windows since Windows XP Service Pack 1 (SP1). I've got a box with Windows XP Home Edition SP3 and I would like to say that from user/application experience it is a bad recommendation and there are two reasons why App Paths is evil.

• cmd environment doesn't understand App Paths


• App Paths seems to work only if application installed for All Users, i.e. App Paths keys created in HKEY_CURRENT_USER are ignored by the system and only those added to HKEY_LOCAL_MACHINE work.

I was surprised to see Inkscape.exe and Python.exe entries under HKCU. Python didn't work from Start->Run dialog (Win-R) until I moved it to HKLM. Inkscape had entries in both HKCU App Paths and in HKLM, so it worked right away, but the set of fields in both was different, so I assumed it stores some valuable user-specific information there. Frankly speaking I was even more surprised to find out that Python works under Far Manager even if its App Paths setting located in HKCU, but it is because Far checks this key itself explicitly. I do not know if it is good assuming that system itself doesn't check this key.

Check if a program is executable in windows batch file

When writing windows batch files it is sometimes necessary to check if a program is installed on user system before going further. The following snippet checks if a program is executable from .bat file, because it is present in current directory of somewhere along %PATH%.

@echo off
svn >nul 2>nul
if %ERRORLEVEL% neq 9009 goto continue
echo Subversion "svn" command not found. Exiting..
exit 1

:continue

>nul 2>nul system suffix redirects output from program (if exists) or from system to nowhere to keep users calm. >nul works for usual output, 2>nul redirects error output. Just make sure that the command you test exits immediately and doesn't hang asking user for input, because user won't see the prompt.

Note about App Paths in batches

Sometimes you'll be surprised to see that your program is executable from your file manager, but fails to run from batch file. This is true, for example, for Far Manager and default installation of Python 2.5. This happens because application is not present in %PATH%, but instead added to registry key App Paths. Batch files are executed by cmd.exe which doesn't check registry (i.e. it uses CreateProcess() system call). File managers usually capable to run such programs by using higher level function ShellExecute(). Look here for more info if you're interested to know how ShellExecute() uses App Paths.

Seems like there is no elegant workaround to silently detect from batch file if your application is available on target system if it installed using App Paths key (like Python). There is start command that can be used to execute commands with ShellExecute(), but when command is not found start displays popup dialog with error message, which is impossible to hide. If you are forced to deal with such application you can create additional/custom .vbs script that calls ShellExecute() and processes errors.

Subprocessing in Python

It may sound strange, but Python with all its low-level roots and crossplatform portability still doesn't have pythonic interface for working with child processes. Libraries that were born to do things unix-way usually fail under windows. Let's take a look into history of subprocessing at

Evil os.popen()

POSIX os.popen() call is at holy war with windows - one of the reasons there are almost no crossplatform Python scripts that wrap and control 3rd party processes. For example, study this os.popen() call reported in this thread:

import os,sys

out = os.popen("nslookup", "w")
out.write("google.com\n")
out.close()
On POSIX machine it will call nslookup, send a string on its input and nslookup will print the result on console:

> Server: 10.0.80.11
Address: 10.0.80.11#53

Non-authoritative answer:
Name: google.com
Address: 74.125.45.100
Name: google.com
Address: 74.125.67.100
Name: google.com
Address: 74.125.127.100
>
Under windows nslookup won't output anything on screen, but it will process the input correctly and that can be proved by redirecting output to a file:

import os,sys

out = os.popen("nslookup > output.txt", "w")
out.write("google.com\n")
out.close()
The workaround suggested in Python bug #1366 and used in Mercurial doesn't seems to work with Python 2.5.4 on XP. According to Mercurial patch the following code should give output on console:

import os,sys

out = os.popen('"nslookup 2> NUL:"', "w")
out.write("google.com\n")
out.close()
but it doesn't. It still executes though, but it is not what you want if the main thing your code does is output pagination. To work around this inherently evil feature of os.popen() smart guys invented subprocess module.

import subprocess

out = subprocess.Popen("nslookup", stdin=subprocess.PIPE)
out.stdin.write("google.com\n")
out.stdin.close()

According to Python bugtracker, subprocess is capable to guard you against malicious os.popen() in many different ways finding natural ways to deal with evilness #3144, evilness #602245, #...

..to be continued

Code syntax hightlight for Blogger

Just add this code inside your template element:

<link href='http://rainforce.org/google/code/prettify/prettify.css' rel='stylesheet' type='text/css'/>
<script src='http://rainforce.org/google/code/prettify/prettify.js' type='text/javascript'></script>
and change <body> tag to be

<body onload="prettyPrint()">
In fact all above is well explained at Google Code Prettify site. Site of the JavaScript library used to colorize <pre> and <code> blocks that have class="prettyprint" attribute. I.e. wrap your code into following HTML block to make it colored:

<code class="prettyprint">
...
</code>
The source of this page contains CSS definition with colors for this blog, but you may copy/paste it from the block below:

<style type='text/css'>
/* desert scheme ported from vim to google prettify */
code.prettyprint { display: block; padding: 2px; border: 1px solid #888; background-color: #333; }
.str { color: #ffa0a0; } /* string - pink */
.kwd { color: #f0e68c; font-weight: bold; }
.com { color: #87ceeb; } /* comment - skyblue */
.typ { color: #98fb98; } /* type - lightgreen */
.lit { color: #cd5c5c; } /* literal - darkred */
.pun { color: #fff; } /* punctuation */
.pln { color: #fff; } /* plaintext */
.tag { color: #f0e68c; font-weight: bold; } /* html/xml tag - lightyellow*/
.atn { color: #bdb76b; font-weight: bold; } /* attribute name - khaki*/
.atv { color: #ffa0a0; } /* attribute value - pink */
.dec { color: #98fb98; } /* decimal - lightgreen */
</style>

Query to get MySQL table type

The following command can be used to get the type (engine) of MySQL table from current database in select query:

SELECT table_name, engine FROM information_schema.tables WHERE table_name="your_table" AND table_schema=DATABASE();