another day another vice another roll of the dice

Thursday, May 06, 2010

Sphinx PDF with rst2pdf

I deliberately omit word LaT*X in my post to avoid missing people who add '-LaT*X' in search queries. Yes, it is possible to generate PDF with Sphinx without LaT*X in cross-platform way. Yes, on Windows too. You will need only rst2pdf. Actually integration with Sphinx is well described in rst2pdf manual (text and PDF), but people find it hard to find this information, so I'll quote checklist here:

install rst2pdf
register rst2pdf in your conf.py Sphinx config
extensions = ['sphinx.ext.autodoc','rst2pdf.pdfbuilder']
run
sphinx-build -bpdf sourcedir outdir

I hope it was helpful. Actually, check the manual - it has some useful options for conf.py and it's more up-to-date.

Friday, April 23, 2010

Why Far Manager and Vim are awesome

Far Manager and Vim are both console tools. When non-computer folks see blue panels of Far on my screen they are usually surprised like "DOS? But why?". These folks still remember DOS windows (DOS Windows, huh), but if some fourteen years adult tells me something about DOS it will be my turn to be surprised.

Far Manager is essentially a Windows console Swiss Army Knife. Much like Vim is for Unix. While GUI tools are awesome, they are not as responsive as console ones, not deterministic - I know it sounds horrible, but it just means that at any given moment you can't be sure what a given combination of keys will do. It depends on the focus and this focus is not always clearly visible. Interaction using mouse is slow (if you're not a hardcore gamer, of course). But I actually written this post to say that:

You don't need to press "ENTER" for most used commands and that is awesome!

In Far you may setup shortcuts as easy as pressing F2 and then Ins. Every action is reachable as a series of key presses. You don't need to put you hand off the keyboard to reach the mouse. Forget about mouse - you will need it only for copy/pasting. You may easily automate repeated key pressing by using Ctrl-. then typing the keys you like and then hitting Ctrl-. again. You'll be prompted for a key combination that will invoke the keys you've just written. Repeated edits, replacements, file renames, copies - a lot of things can be automated using this shortcut called Macro.

Another from many Far features you may find awesome is output redirection. By using edit:< prefix on the command line you may redirect the output to embedded editor and navigate around it as you like. With Colorer plugin you will even get syntax highlight for the output.

Enjoy!

Wednesday, April 14, 2010

Porting Python applications from Unix to Windows

os.open

Note the difference between open and os.open and ensure that all os.open calls have os.O_BINARY flag for Windows.

See also Stani and Nadia talk about cross platform application development and distribution

Wednesday, February 03, 2010

URL parameters in action method of HTML form are lost for GET request

Surprisingly how experienced web-developers may miss some basic nuances of form processing that exist for many-many years. One of them is the fact that URL parameters added to action attribute of <form> element are lost for GET requests.

Example:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Query string from action attribute is killed for GET request</title>
</head>
<body>

  <form name="getSearchForm" action="?missed=one" method="get">
    <input type="text" name="q" value="s" />
    <button class="searchButton" type="submit">GET</button>
  </form>

  <form name="postSearchForm" action="?missed=one" method="post">
    <input type="text" name="q" value="s" />
    <button class="searchButton" type="submit">POST</button>
  </form>

</body>
</html>

The parameter "missed=one" is missing from URL after clicking a button on a form with GET send method. It is present for POST form though.

Friday, January 29, 2010

Working with complex issues

Some issues should be cut into chewable chunks and linked together into a dependency tree. Each chunk should be accompanied by "digestion recipe" that includes tools, skills and necessary ingredients. If the issue is very complicated, the chunks may be split in pieces that take no longer than one day to get the chunk, analyze it and solve. The work on the issue can then be spread over the looong time.

This requires tools. Tools to minimize waste of time on getting all necessary stuff to start work, save work, send it in one day and wait for it to be approved. Learning these tools should also be quick to get the task done.

Life is short, so time savers are critical. Cheatsheet template with already filled address of contacts, repositories etc. can greatly reduce the time to get the thing done. Once template is filled, it is the a cheatsheet that can be commented with new information for this specific task. These comments can then be incorporated back to original template or into a new, for more advanced usage. If making a reusable template is just one hour, and solution for the task is one day - a template one day and the task the other is better.

Monday, January 25, 2010

Repacking library.zip from py2exe

intro

py2exe tool converts Python scripts to standalone .exe distributives. It isolates all required Python modules together with Python interpreter itself and wraps them into single library.zip file. This way application is not affected by Python modules that may be already installed on user system. Unfortunately, this also means that you can't add new modules to your standalone Python program.

For example, you need to add Hg-Git extension to Mercurial installed as standalone program. You can specify path to extension in Mercurial.ini, but Hg-Git depends on Dulwich module, which is not present in library.zip Attempt to use this extension will fail. The same problem is with converting Bazaar repositories using convert extension that is present in library.zip, but additionally requires installed bzr module.

solution

The script below helps to add Python modules to library.zip file made with py2exe. Latest version should be available at this bitbucket repository. The following command unpacks library.zip in current directory and makes library_unpacked.zip that you can edit with your favorite archiver:

python relibzip.py unpack

After you've finished, issue:

python relibzip.py pack

and script will create library_packed.zip from extracted resources. Copy this file over library.zip and you all set.

Source code (ugly, but works):


"""
pack/unpack library.zip created by py2exe to standard .zip archive

library.zip created with py2exe can not always be processed by standard
archivers. this script removes extra chunks added by py2exe and puts
them back when requested

MIT license, by techtonik // gmail.com
"""

import sys
import os
from optparse import OptionParser
import struct


PYTHONDLL = "<pythondll>"
PDNAME = "pythondll"
ZLIBPYD = "<zlib.pyd>"
ZDNAME = "zlibpyd"

UNPACKED = "library_unpacked.zip"
PACKED = "library_packed.zip"


def unpack(filename):
   f = open(filename, "rb")
   # looking for PYTHONDLL name
   pdname = f.read(len(PYTHONDLL))
   if pdname != PYTHONDLL:
       if pdname[:2] == "PK":
           sys.exit("Seems to be normal .zip archive, not unpacking")
       else:
           sys.exit("Unknown archive format")

   def save_section(secname, fname):
       print "Extracting %s section to %s" % (secname, fname)
       fsize = struct.unpack("i", f.read(4))[0]
       fpd = open(fname, "wb")
       fpd.write(f.read(fsize))
       fpd.close()
   save_section(PYTHONDLL, PDNAME)

   buf = ""
   zdname = f.read(len(ZLIBPYD))
   if zdname != ZLIBPYD:
       if zdname[:2] == "PK":
           print "No zlib.pyd section"
           buf = zdname
       else:
           sys.exit("Unknown archive format")
   else:
       save_section(ZLIBPYD, ZDNAME)
  
   flib = open(UNPACKED, "wb")
   flib.write(buf)
   flib.write(f.read())
   flib.close()
   f.close()
   print "Done. Unpacked .zip contents is available at %s" % UNPACKED
   sys.exit(0)


def pack(tofname):

   if not os.path.exists(PDNAME):
       sys.exit("%s section file %s is not found. Exiting" % (PYTHONDLL, PDNAME))
   if not os.path.exists(UNPACKED):
       sys.exit("Unpacked version %s is not found. Exiting" % UNPACKED)

   f = open(tofname, "wb")

   def write_section(secname, fname):
       print "Writing %s section from %s" % (secname,fname)
       f.write(PYTHONDLL)
       fsize = os.stat(PDNAME).st_size
       f.write(struct.pack("i", fsize))

       fpd = open(fname, "rb")
       f.write(fpd.read())
       fpd.close()
       print "Removing section file %s" % fname
       os.remove(fname)
   write_section(PYTHONDLL, PDNAME)

   # check for optional ZLIBPYD section
   if not os.path.exists(ZDNAME):
       print "No %s section file %s. Skipping" % (ZLIBPYD, ZDNAME)
   else:
       write_section(ZLIBPYD, ZDNAME)

   fzip = open(UNPACKED, "rb")
   f.write(fzip.read())
   fzip.close()

   f.close()
   print "Done. Packed .zip contents is available at %s" % tofname
   sys.exit(0)


parser = OptionParser(usage="usage: %prog ",
   description="update library.zip created by py2exe utility")
opt,arg = parser.parse_args()
  
if arg and arg[0] == 'unpack':
   unpack("library.zip")
elif arg and arg[0] == 'pack':
   pack(PACKED)
else:
   sys.exit(parser.format_help())

Tuesday, December 08, 2009

An ideal options / arguments parsing library for Python

One of primary areas of Python applications is helper scripts and system utilities that are often called from command line. Even programming in Python itself is not comfortable without interactive mode at hand. That's why argument parsing libraries are important to some degree to many Python developers.

There once was getopt library, but it was awkward compared to newer optparse module, but even optparse nowadays is not enough for some who is willing to replace it with argparse. optparse is a kind of "stable minimal API" that is not easy to push a change to. As a person who has a lot of Python command tools at hand I often feel that I need patched version of optparse, but I can't afford managing dependencies or inserting a bulk of code into 5 minute script that should be as minimal and clear as possible for easy maintenance. So I wouldn't mind against more featured argparse library.

However, argparse can only be useful in standard library if it will be:

Minimalistic. Like optparse won't take much brainspace to avoid referencing documentation for 80% of the stuff
Easily extensible
Contains copy/paste recipes for popular usages

On the first look argparse is not minimalistic - even in constructor there are a lot of params that won't be used in 80% of cases and that could be easily moved into configuration methods.

Is it easily extensible? It allows to specify formatter class, conflict_handler class, but it seems like you have very limited control over what will be matched as option and what as an argument. There are two options to constructor to define options prefix - prefix_chars and fromfile_prefix_chars, but they are awkward. I wonder if I can provide /longarg on windows in addition to --longarg leaving also /s and -s options? It could be better done with the same custom class, which could be provided in recipes or as an optional import.

There are no recipes in argparse. That's bad, because that means the library is designed as a comprehensive solution to all command-line problems, but it still can't help me make the majority of my scripts more convenient without too much overhead.

argparse is not a huge leap from optparse as the latter once was for getopt in my opinion. For any argument parsing library to become a quality leap from optparse, it should allow an easy way to build svn like command line interfaces with the following features:

calling without arguments shows tool name, version and usage info
usage info contains either command list, options list or both
"tool.py help " is equivalent to "tool.py -h"

remove ugly default optparse help for options with arguments

-f FILE, --file=FILE  write report to FILE

SVN is fine

-r [--revision] ARG      : ARG (some commands also take ARG1:ARG2 range)

or Mercurial

-l --logfile       read commit message from

cross-platform optional paging

This is a functionality of an ideal argument parsing library. Feel free to add your own user story.