Friday, January 29, 2010

Working with complex issues

Some issues should be cut into chewable chunks and linked together into a dependency tree. Each chunk should be accompanied by "digestion recipe" that includes tools, skills and necessary ingredients. If the issue is very complicated, the chunks may be split in pieces that take no longer than one day to get the chunk, analyze it and solve. The work on the issue can then be spread over the looong time.

This requires tools. Tools to minimize waste of time on getting all necessary stuff to start work, save work, send it in one day and wait for it to be approved. Learning these tools should also be quick to get the task done.

Life is short, so time savers are critical. Cheatsheet template with already filled address of contacts, repositories etc. can greatly reduce the time to get the thing done. Once template is filled, it is the a cheatsheet that can be commented with new information for this specific task. These comments can then be incorporated back to original template or into a new, for more advanced usage. If making a reusable template is just one hour, and solution for the task is one day - a template one day and the task the other is better.

Monday, January 25, 2010

Repacking library.zip from py2exe



intro
py2exe tool converts Python scripts to standalone .exe distributives. It isolates all required Python modules together with Python interpreter itself and wraps them into single library.zip file. This way application is not affected by Python modules that may be already installed on user system. Unfortunately, this also means that you can't add new modules to your standalone Python program.

For example, you need to add Hg-Git extension to Mercurial installed as standalone program. You can specify path to extension in Mercurial.ini, but Hg-Git depends on Dulwich module, which is not present in library.zip Attempt to use this extension will fail. The same problem is with converting Bazaar repositories using convert extension that is present in library.zip, but additionally requires installed bzr module.

solution
The script below helps to add Python modules to library.zip file made with py2exe. Latest version should be available at this bitbucket repository. The following command unpacks library.zip in current directory and makes library_unpacked.zip that you can edit with your favorite archiver:

python relibzip.py unpack
After you've finished, issue:

python relibzip.py pack
and script will create library_packed.zip from extracted resources. Copy this file over library.zip and you all set.

Source code (ugly, but works):

"""
pack/unpack library.zip created by py2exe to standard .zip archive

library.zip created with py2exe can not always be processed by standard
archivers. this script removes extra chunks added by py2exe and puts
them back when requested

MIT license, by techtonik // gmail.com
"""

import sys
import os
from optparse import OptionParser
import struct


PYTHONDLL = "<pythondll>"
PDNAME = "pythondll"
ZLIBPYD = "<zlib.pyd>"
ZDNAME = "zlibpyd"

UNPACKED = "library_unpacked.zip"
PACKED = "library_packed.zip"


def unpack(filename):
   f = open(filename, "rb")
   # looking for PYTHONDLL name
   pdname = f.read(len(PYTHONDLL))
   if pdname != PYTHONDLL:
       if pdname[:2] == "PK":
           sys.exit("Seems to be normal .zip archive, not unpacking")
       else:
           sys.exit("Unknown archive format")

   def save_section(secname, fname):
       print "Extracting %s section to %s" % (secname, fname)
       fsize = struct.unpack("i", f.read(4))[0]
       fpd = open(fname, "wb")
       fpd.write(f.read(fsize))
       fpd.close()
   save_section(PYTHONDLL, PDNAME)

   buf = ""
   zdname = f.read(len(ZLIBPYD))
   if zdname != ZLIBPYD:
       if zdname[:2] == "PK":
           print "No zlib.pyd section"
           buf = zdname
       else:
           sys.exit("Unknown archive format")
   else:
       save_section(ZLIBPYD, ZDNAME)
  
   flib = open(UNPACKED, "wb")
   flib.write(buf)
   flib.write(f.read())
   flib.close()
   f.close()
   print "Done. Unpacked .zip contents is available at %s" % UNPACKED
   sys.exit(0)


def pack(tofname):

   if not os.path.exists(PDNAME):
       sys.exit("%s section file %s is not found. Exiting" % (PYTHONDLL, PDNAME))
   if not os.path.exists(UNPACKED):
       sys.exit("Unpacked version %s is not found. Exiting" % UNPACKED)

   f = open(tofname, "wb")

   def write_section(secname, fname):
       print "Writing %s section from %s" % (secname,fname)
       f.write(PYTHONDLL)
       fsize = os.stat(PDNAME).st_size
       f.write(struct.pack("i", fsize))

       fpd = open(fname, "rb")
       f.write(fpd.read())
       fpd.close()
       print "Removing section file %s" % fname
       os.remove(fname)
   write_section(PYTHONDLL, PDNAME)

   # check for optional ZLIBPYD section
   if not os.path.exists(ZDNAME):
       print "No %s section file %s. Skipping" % (ZLIBPYD, ZDNAME)
   else:
       write_section(ZLIBPYD, ZDNAME)

   fzip = open(UNPACKED, "rb")
   f.write(fzip.read())
   fzip.close()

   f.close()
   print "Done. Packed .zip contents is available at %s" % tofname
   sys.exit(0)


parser = OptionParser(usage="usage: %prog ",
   description="update library.zip created by py2exe utility")
opt,arg = parser.parse_args()
  
if arg and arg[0] == 'unpack':
   unpack("library.zip")
elif arg and arg[0] == 'pack':
   pack(PACKED)
else:
   sys.exit(parser.format_help())