Working with complex issues

Some issues should be cut into chewable chunks and linked together into a dependency tree. Each chunk should be accompanied by "digestion recipe" that includes tools, skills and necessary ingredients. If the issue is very complicated, the chunks may be split in pieces that take no longer than one day to get the chunk, analyze it and solve. The work on the issue can then be spread over the looong time.

This requires tools. Tools to minimize waste of time on getting all necessary stuff to start work, save work, send it in one day and wait for it to be approved. Learning these tools should also be quick to get the task done.

Life is short, so time savers are critical. Cheatsheet template with already filled address of contacts, repositories etc. can greatly reduce the time to get the thing done. Once template is filled, it is the a cheatsheet that can be commented with new information for this specific task. These comments can then be incorporated back to original template or into a new, for more advanced usage. If making a reusable template is just one hour, and solution for the task is one day - a template one day and the task the other is better.

Repacking library.zip from py2exe

intro

py2exe tool converts Python scripts to standalone .exe distributives. It isolates all required Python modules together with Python interpreter itself and wraps them into single library.zip file. This way application is not affected by Python modules that may be already installed on user system. Unfortunately, this also means that you can't add new modules to your standalone Python program.

For example, you need to add Hg-Git extension to Mercurial installed as standalone program. You can specify path to extension in Mercurial.ini, but Hg-Git depends on Dulwich module, which is not present in library.zip Attempt to use this extension will fail. The same problem is with converting Bazaar repositories using convert extension that is present in library.zip, but additionally requires installed bzr module.

solution

The script below helps to add Python modules to library.zip file made with py2exe. Latest version should be available at this bitbucket repository. The following command unpacks library.zip in current directory and makes library_unpacked.zip that you can edit with your favorite archiver:

python relibzip.py unpack

After you've finished, issue:

python relibzip.py pack

and script will create library_packed.zip from extracted resources. Copy this file over library.zip and you all set.

Source code (ugly, but works):

"""
pack/unpack library.zip created by py2exe to standard .zip archive

library.zip created with py2exe can not always be processed by standard
archivers. this script removes extra chunks added by py2exe and puts
them back when requested

MIT license, by techtonik // gmail.com
"""

import sys
import os
from optparse import OptionParser
import struct


PYTHONDLL = "<pythondll>"
PDNAME = "pythondll"
ZLIBPYD = "<zlib.pyd>"
ZDNAME = "zlibpyd"

UNPACKED = "library_unpacked.zip"
PACKED = "library_packed.zip"


def unpack(filename):
   f = open(filename, "rb")
   # looking for PYTHONDLL name
   pdname = f.read(len(PYTHONDLL))
   if pdname != PYTHONDLL:
       if pdname[:2] == "PK":
           sys.exit("Seems to be normal .zip archive, not unpacking")
       else:
           sys.exit("Unknown archive format")

   def save_section(secname, fname):
       print "Extracting %s section to %s" % (secname, fname)
       fsize = struct.unpack("i", f.read(4))[0]
       fpd = open(fname, "wb")
       fpd.write(f.read(fsize))
       fpd.close()
   save_section(PYTHONDLL, PDNAME)

   buf = ""
   zdname = f.read(len(ZLIBPYD))
   if zdname != ZLIBPYD:
       if zdname[:2] == "PK":
           print "No zlib.pyd section"
           buf = zdname
       else:
           sys.exit("Unknown archive format")
   else:
       save_section(ZLIBPYD, ZDNAME)
  
   flib = open(UNPACKED, "wb")
   flib.write(buf)
   flib.write(f.read())
   flib.close()
   f.close()
   print "Done. Unpacked .zip contents is available at %s" % UNPACKED
   sys.exit(0)


def pack(tofname):

   if not os.path.exists(PDNAME):
       sys.exit("%s section file %s is not found. Exiting" % (PYTHONDLL, PDNAME))
   if not os.path.exists(UNPACKED):
       sys.exit("Unpacked version %s is not found. Exiting" % UNPACKED)

   f = open(tofname, "wb")

   def write_section(secname, fname):
       print "Writing %s section from %s" % (secname,fname)
       f.write(PYTHONDLL)
       fsize = os.stat(PDNAME).st_size
       f.write(struct.pack("i", fsize))

       fpd = open(fname, "rb")
       f.write(fpd.read())
       fpd.close()
       print "Removing section file %s" % fname
       os.remove(fname)
   write_section(PYTHONDLL, PDNAME)

   # check for optional ZLIBPYD section
   if not os.path.exists(ZDNAME):
       print "No %s section file %s. Skipping" % (ZLIBPYD, ZDNAME)
   else:
       write_section(ZLIBPYD, ZDNAME)

   fzip = open(UNPACKED, "rb")
   f.write(fzip.read())
   fzip.close()

   f.close()
   print "Done. Packed .zip contents is available at %s" % tofname
   sys.exit(0)


parser = OptionParser(usage="usage: %prog ",
   description="update library.zip created by py2exe utility")
opt,arg = parser.parse_args()
  
if arg and arg[0] == 'unpack':
   unpack("library.zip")
elif arg and arg[0] == 'pack':
   pack(PACKED)
else:
   sys.exit(parser.format_help())

An ideal options / arguments parsing library for Python

One of primary areas of Python applications is helper scripts and system utilities that are often called from command line. Even programming in Python itself is not comfortable without interactive mode at hand. That's why argument parsing libraries are important to some degree to many Python developers.

There once was getopt library, but it was awkward compared to newer optparse module, but even optparse nowadays is not enough for some who is willing to replace it with argparse. optparse is a kind of "stable minimal API" that is not easy to push a change to. As a person who has a lot of Python command tools at hand I often feel that I need patched version of optparse, but I can't afford managing dependencies or inserting a bulk of code into 5 minute script that should be as minimal and clear as possible for easy maintenance. So I wouldn't mind against more featured argparse library.

However, argparse can only be useful in standard library if it will be:

1. Minimalistic. Like optparse won't take much brainspace to avoid referencing documentation for 80% of the stuff
2. Easily extensible
3. Contains copy/paste recipes for popular usages
   

On the first look argparse is not minimalistic - even in constructor there are a lot of params that won't be used in 80% of cases and that could be easily moved into configuration methods.

Is it easily extensible? It allows to specify formatter class, conflict_handler class, but it seems like you have very limited control over what will be matched as option and what as an argument. There are two options to constructor to define options prefix - prefix_chars and fromfile_prefix_chars, but they are awkward. I wonder if I can provide /longarg on windows in addition to --longarg leaving also /s and -s options? It could be better done with the same custom class, which could be provided in recipes or as an optional import.

There are no recipes in argparse. That's bad, because that means the library is designed as a comprehensive solution to all command-line problems, but it still can't help me make the majority of my scripts more convenient without too much overhead.

argparse is not a huge leap from optparse as the latter once was for getopt in my opinion. For any argument parsing library to become a quality leap from optparse, it should allow an easy way to build svn like command line interfaces with the following features:

1. calling without arguments shows tool name, version and usage info
2. usage info contains either command list, options list or both
3. "tool.py help " is equivalent to "tool.py -h"
4. remove ugly default optparse help for options with arguments
   
   

   -f FILE, --file=FILE  write report to FILE

   SVN is fine

   -r [--revision] ARG      : ARG (some commands also take ARG1:ARG2 range)

   or Mercurial 
   

   -l --logfile       read commit message from  
   

5. cross-platform optional paging
   
   

This is a functionality of an ideal argument parsing library. Feel free to add your own user story.

Subprocessing. Why it won't be truly crossplatfrom

Every language and application stops to be crossplatform when it comes to executing other applications. Differences in a way various operating systems execute applications is so inconvenient that even new Python Subprocess API is not complete to keep you isolated from them.

Suppose you have an executable script.py in current directory that you want to execute. You would definitely need to execute the script through shell, because only shell knows which program should process it.

Windows

subprocess.Popen("script.py",shell=True)


POSIX

subprocess.Popen("./script.py",shell=True)


So you still need to keep in mind that to execute script from current directory you need to prefix the script with ./ under POSIX.

Lost network interfaces after moving Debian virtualbox to another host

Every time Debian virtual machine migrates to a new host, it loses its network interfaces. Debian guest detects new virtual hardware and assigns new aliases losing IP address and other settings. The easiest way is to rename new interface back to original way by editing hard to find config file /etc/udev/rules.d/z25_persistent-net.rules (works for etch)

sudo vim /etc/udev/rules.d/z25_persistent-net.rules
Do not forget to check actual settings for network interfaces with

sudo ifconfig -a
and settings that are applied on each restart in

sudo vim /etc/network/interfaces
Instead of renaming network interfaces in Debian it might be easier to record MAC numbers in Notes for virtual machine and setup virtual network interfaces in host accordingly prior to starting imported machine.

Note about using "App paths" for installers

App paths is recommended (or better say - preferred) way to register application in Windows since Windows XP Service Pack 1 (SP1). I've got a box with Windows XP Home Edition SP3 and I would like to say that from user/application experience it is a bad recommendation and there are two reasons why App Paths is evil.

• cmd environment doesn't understand App Paths


• App Paths seems to work only if application installed for All Users, i.e. App Paths keys created in HKEY_CURRENT_USER are ignored by the system and only those added to HKEY_LOCAL_MACHINE work.

I was surprised to see Inkscape.exe and Python.exe entries under HKCU. Python didn't work from Start->Run dialog (Win-R) until I moved it to HKLM. Inkscape had entries in both HKCU App Paths and in HKLM, so it worked right away, but the set of fields in both was different, so I assumed it stores some valuable user-specific information there. Frankly speaking I was even more surprised to find out that Python works under Far Manager even if its App Paths setting located in HKCU, but it is because Far checks this key itself explicitly. I do not know if it is good assuming that system itself doesn't check this key.

Check if a program is executable in windows batch file

When writing windows batch files it is sometimes necessary to check if a program is installed on user system before going further. The following snippet checks if a program is executable from .bat file, because it is present in current directory of somewhere along %PATH%.

@echo off
svn >nul 2>nul
if %ERRORLEVEL% neq 9009 goto continue
echo Subversion "svn" command not found. Exiting..
exit 1

:continue

>nul 2>nul system suffix redirects output from program (if exists) or from system to nowhere to keep users calm. >nul works for usual output, 2>nul redirects error output. Just make sure that the command you test exits immediately and doesn't hang asking user for input, because user won't see the prompt.

Note about App Paths in batches

Sometimes you'll be surprised to see that your program is executable from your file manager, but fails to run from batch file. This is true, for example, for Far Manager and default installation of Python 2.5. This happens because application is not present in %PATH%, but instead added to registry key App Paths. Batch files are executed by cmd.exe which doesn't check registry (i.e. it uses CreateProcess() system call). File managers usually capable to run such programs by using higher level function ShellExecute(). Look here for more info if you're interested to know how ShellExecute() uses App Paths.

Seems like there is no elegant workaround to silently detect from batch file if your application is available on target system if it installed using App Paths key (like Python). There is start command that can be used to execute commands with ShellExecute(), but when command is not found start displays popup dialog with error message, which is impossible to hide. If you are forced to deal with such application you can create additional/custom .vbs script that calls ShellExecute() and processes errors.