Friday, January 29, 2010

Working with complex issues

Some issues should be cut into chewable chunks and linked together into a dependency tree. Each chunk should be accompanied by "digestion recipe" that includes tools, skills and necessary ingredients. If the issue is very complicated, the chunks may be split in pieces that take no longer than one day to get the chunk, analyze it and solve. The work on the issue can then be spread over the looong time.

This requires tools. Tools to minimize waste of time on getting all necessary stuff to start work, save work, send it in one day and wait for it to be approved. Learning these tools should also be quick to get the task done.

Life is short, so time savers are critical. Cheatsheet template with already filled address of contacts, repositories etc. can greatly reduce the time to get the thing done. Once template is filled, it is the a cheatsheet that can be commented with new information for this specific task. These comments can then be incorporated back to original template or into a new, for more advanced usage. If making a reusable template is just one hour, and solution for the task is one day - a template one day and the task the other is better.

Monday, January 25, 2010

Repacking from py2exe

py2exe tool converts Python scripts to standalone .exe distributives. It isolates all required Python modules together with Python interpreter itself and wraps them into single file. This way application is not affected by Python modules that may be already installed on user system. Unfortunately, this also means that you can't add new modules to your standalone Python program.

For example, you need to add Hg-Git extension to Mercurial installed as standalone program. You can specify path to extension in Mercurial.ini, but Hg-Git depends on Dulwich module, which is not present in Attempt to use this extension will fail. The same problem is with converting Bazaar repositories using convert extension that is present in, but additionally requires installed bzr module.

The script below helps to add Python modules to file made with py2exe. Latest version should be available at this bitbucket repository. The following command unpacks in current directory and makes that you can edit with your favorite archiver:

python unpack
After you've finished, issue:

python pack
and script will create from extracted resources. Copy this file over and you all set.

Source code (ugly, but works):

pack/unpack created by py2exe to standard .zip archive created with py2exe can not always be processed by standard
archivers. this script removes extra chunks added by py2exe and puts
them back when requested

MIT license, by techtonik //

import sys
import os
from optparse import OptionParser
import struct

PYTHONDLL = "<pythondll>"
PDNAME = "pythondll"
ZLIBPYD = "<zlib.pyd>"
ZDNAME = "zlibpyd"


def unpack(filename):
   f = open(filename, "rb")
   # looking for PYTHONDLL name
   pdname =
   if pdname != PYTHONDLL:
       if pdname[:2] == "PK":
           sys.exit("Seems to be normal .zip archive, not unpacking")
           sys.exit("Unknown archive format")

   def save_section(secname, fname):
       print "Extracting %s section to %s" % (secname, fname)
       fsize = struct.unpack("i",[0]
       fpd = open(fname, "wb")
   save_section(PYTHONDLL, PDNAME)

   buf = ""
   zdname =
   if zdname != ZLIBPYD:
       if zdname[:2] == "PK":
           print "No zlib.pyd section"
           buf = zdname
           sys.exit("Unknown archive format")
       save_section(ZLIBPYD, ZDNAME)
   flib = open(UNPACKED, "wb")
   print "Done. Unpacked .zip contents is available at %s" % UNPACKED

def pack(tofname):

   if not os.path.exists(PDNAME):
       sys.exit("%s section file %s is not found. Exiting" % (PYTHONDLL, PDNAME))
   if not os.path.exists(UNPACKED):
       sys.exit("Unpacked version %s is not found. Exiting" % UNPACKED)

   f = open(tofname, "wb")

   def write_section(secname, fname):
       print "Writing %s section from %s" % (secname,fname)
       fsize = os.stat(PDNAME).st_size
       f.write(struct.pack("i", fsize))

       fpd = open(fname, "rb")
       print "Removing section file %s" % fname
   write_section(PYTHONDLL, PDNAME)

   # check for optional ZLIBPYD section
   if not os.path.exists(ZDNAME):
       print "No %s section file %s. Skipping" % (ZLIBPYD, ZDNAME)
       write_section(ZLIBPYD, ZDNAME)

   fzip = open(UNPACKED, "rb")

   print "Done. Packed .zip contents is available at %s" % tofname

parser = OptionParser(usage="usage: %prog ",
   description="update created by py2exe utility")
opt,arg = parser.parse_args()
if arg and arg[0] == 'unpack':
elif arg and arg[0] == 'pack':