Chrome logo was awesome

This is a Chrome 11 logo.

This one is the old variant.

Finding unused files in a SCons project with Process Monitor

New technologies are born and die, but one things remains in your project - their files. Quite often you have no idea about where are they used, and attempt to remove them may lead to serious consequences.

Fortunately, if your project is managed by fine grained build system such as SCons, if your build scripts are not globbing too much, there are chances you can find files that are not participating in the builds.

Here is how to do this on Windows using Process Monitor tool that intercepts all system calls including file access.

While build systems are usually common for C/C++ and Java projects, it is possible to add fine-grained file usage control for any project. For example, SCons itself is written entirely in Python, it could run directly from the source checkout or build distributives from checkout. But instead, it uses build procedure to copy all necessary files from checkout into separate directory and do stuff from there.

Thanks to that it is possible to see which files are no more actual. While it is possible to compare checkout source tree and copied directory trees, I'll go through the hells of monitoring system file access in a source tree during the build process using Process Monitor (FileMon in the past). Linux should have similar tools too - let me know how are they called.

The process is the following:

1. Start Process Monitor
2. Stop incoming event flood by (un)clicking Capture (Ctrl-E) button
3. Open Filter (Ctrl-L) dialog to add some filters


SCons build is started by bootstrap.py script from a root of SCons source checkout. The script is executed by python executable, so I add python.exe process name to the filter. I know that bootstrap.py copies files from src/ subdirectory, so it is the directory I need to monitor, so I add this dir to filters too.

4. Go Tools -> File Summary...


There is a list of paths catched by Process Monitor when listening to system calls. They are already filtered, but additional filters can be applied using bottom left button to make information even more useful.

5. Export to CSV using Save...

Exported CSV is not very useful without some postprocessing. I used the following a script to compare the list of paths in CSV to actual src/ directory contents. This gives me names of files that were not touched during build at all.

SRCDIR = "C:\\p\\python\\scons\\src"
CSVLIST = 'accessed_bootstrap_files.CSV'

import csv
import os

reader = csv.reader(open(CSVLIST))
header = reader.next()
pathidx = header.index("Path")
pathset = set([row[pathidx] for row in reader])

#for row in pathset:
#  print row

fileset = set()
for root, dirs, files in os.walk(SRCDIR):
  fileset.update( [os.path.join(root, f) for f in files] )
  if '.svn' in dirs:
    dirs.remove('.svn')  # don't visit .svn directories

if len(pathset & fileset) == 0:
  print 'Error: File sets do not intersect at all'

print "Files not found in source directory tree:"
for f in (pathset - fileset):
  if not os.path.isdir(f):
    print f

print
print "Untouched files in source directory tree:"
for f in sorted(fileset - pathset):
  if not os.path.isdir(f):
    print f

I've found a few interesting things about SCons. Core tests are mixed with source files in repository checkout. They are not copied during bootstrap build. There are also few setup.py files, post-install script and announcement that don't participate in the build.

Here is the output of the above script:

Files not found in source directory tree:
<Total>

Untouched files in source directory tree:
C:\p\python\scons\src\.aeignore
C:\p\python\scons\src\Announce.txt
C:\p\python\scons\src\engine\.aeignore
C:\p\python\scons\src\engine\SCons\.aeignore
C:\p\python\scons\src\engine\SCons\ActionTests.py
C:\p\python\scons\src\engine\SCons\BuilderTests.py
C:\p\python\scons\src\engine\SCons\CacheDirTests.py
C:\p\python\scons\src\engine\SCons\DefaultsTests.py
C:\p\python\scons\src\engine\SCons\EnvironmentTests.py
C:\p\python\scons\src\engine\SCons\ErrorsTests.py
C:\p\python\scons\src\engine\SCons\ExecutorTests.py
C:\p\python\scons\src\engine\SCons\JobTests.py
C:\p\python\scons\src\engine\SCons\MemoizeTests.py
C:\p\python\scons\src\engine\SCons\Node\.aeignore
C:\p\python\scons\src\engine\SCons\Node\AliasTests.py
C:\p\python\scons\src\engine\SCons\Node\FSTests.py
C:\p\python\scons\src\engine\SCons\Node\NodeTests.py
C:\p\python\scons\src\engine\SCons\Node\PythonTests.py
C:\p\python\scons\src\engine\SCons\Optik\.aeignore
C:\p\python\scons\src\engine\SCons\PathListTests.py
C:\p\python\scons\src\engine\SCons\Platform\.aeignore
C:\p\python\scons\src\engine\SCons\Platform\PlatformTests.py
C:\p\python\scons\src\engine\SCons\SConfTests.py
C:\p\python\scons\src\engine\SCons\SConsignTests.py
C:\p\python\scons\src\engine\SCons\Scanner\.aeignore
C:\p\python\scons\src\engine\SCons\Scanner\CTests.py
C:\p\python\scons\src\engine\SCons\Scanner\DirTests.py
C:\p\python\scons\src\engine\SCons\Scanner\FortranTests.py
C:\p\python\scons\src\engine\SCons\Scanner\IDLTests.py
C:\p\python\scons\src\engine\SCons\Scanner\LaTeXTests.py
C:\p\python\scons\src\engine\SCons\Scanner\ProgTests.py
C:\p\python\scons\src\engine\SCons\Scanner\RCTests.py
C:\p\python\scons\src\engine\SCons\Scanner\ScannerTests.py
C:\p\python\scons\src\engine\SCons\Script\.aeignore
C:\p\python\scons\src\engine\SCons\Script\MainTests.py
C:\p\python\scons\src\engine\SCons\Script\SConscriptTests.py
C:\p\python\scons\src\engine\SCons\SubstTests.py
C:\p\python\scons\src\engine\SCons\TaskmasterTests.py
C:\p\python\scons\src\engine\SCons\Tool\.aeignore
C:\p\python\scons\src\engine\SCons\Tool\JavaCommonTests.py
C:\p\python\scons\src\engine\SCons\Tool\PharLapCommonTests.py
C:\p\python\scons\src\engine\SCons\Tool\ToolTests.py
C:\p\python\scons\src\engine\SCons\Tool\f03.xml
C:\p\python\scons\src\engine\SCons\Tool\msvsTests.py
C:\p\python\scons\src\engine\SCons\UtilTests.py
C:\p\python\scons\src\engine\SCons\Variables\BoolVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\EnumVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\ListVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\PackageVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\PathVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\VariablesTests.py
C:\p\python\scons\src\engine\SCons\WarningsTests.py
C:\p\python\scons\src\engine\SCons\cppTests.py
C:\p\python\scons\src\engine\setup.cfg
C:\p\python\scons\src\engine\setup.py
C:\p\python\scons\src\script\.aeignore
C:\p\python\scons\src\script\scons-post-install.py
C:\p\python\scons\src\script\setup.cfg
C:\p\python\scons\src\script\setup.py
C:\p\python\scons\src\test_aegistests.py
C:\p\python\scons\src\test_files.py
C:\p\python\scons\src\test_interrupts.py
C:\p\python\scons\src\test_pychecker.py
C:\p\python\scons\src\test_setup.py
C:\p\python\scons\src\test_strings.py


Hope this helps clean up your projects too.

P.S. I wish there was a Python script replacement for Process Monitor, or at least that it could be controlled from command line.

Asynchronous input from Windows console

too long didn't read - https://bitbucket.org/techtonik/async-console-input - public domain/MIT example for Windows implemented in pure Python (ctypes)

What is asynchronous input?

Given: The program need to terminate immediately when a subprocess exits or user hits 'q'. It should not eat 100% CPU time.

Blocking synchronous input:

import sys, subprocess
from msvcrt import getch, kbhit

p = subprocess.Popen([r"notepad"], shell=True)

while True:
  char = getch()
  if char == 'q':
    sys.exit('terminated by user')
  if p.poll() != None:
    sys.exit('terminated by child exit')

This one doesn't work (doesn't exits immediately when child process terminates), because getch() blocks execution key is pressed, and even if child process exits, it won't be detected until some key is pressed.

Non-blocking synchronous input (polling):

import sys, subprocess
from msvcrt import getch, kbhit

p = subprocess.Popen([r"notepad"], shell=True)

while True:
  while kbhit():
    if getch() == 'q':
      sys.exit('terminated by user')
  if p.poll() != None:
    sys.exit('terminated by child exit')

This works ok, but constant polling uses 100% of CPU resources. It is possible to insert time.sleep() instructions to reduce the carbon footprint, but these crutches will greatly slow down the console when you add z-type to console to spend your time until the background process is finished.

Asynchronous console input on Windows with Python

Asynchronous input allows your program to receive notification from operating system when an input is available. This means that you define events your program needs to react (not necessary console events), send this list to operating system and put your program into wait mode. When system sees this event, your wait function returns and it's possible to inspect/filter event to a greater detail.

Windows provides WaitForMultipleObjects wait function, and in Linux I believe it is select call. If you've tried to understand how Twisted works, but couldn't - will it make more clear if I say that twisted reactor is a wait function? What you do when you code with twisted is just configuring events to react, making chains of them so that one event reacts to another event. This allow to build very interesting, complex and de-coupled asynchronous applications in a couple of days.

O.k. Getting back from Twisted to the asynchronous console input on Windows. Below is the full source code that uses WaitForMultipleObjects. I am afraid that's minimal complete example possible to build with ctypes using Windows API. Tested with Python 2.5

Non-blocking asynchronous input from console:
https://bitbucket.org/techtonik/async-console-input

"""
Example of non-blocking asynchronous console input using
Windows API calls in Python.  This can become handy for
async console tools such as IRC client.

Public domain or MIT license
by anatoly techtonik 


Notes:
  1. WaitForMultipleObjects is used to listen for the
     signals from process and stdin handles
  2. When handle is signalled it remains in this state
     until reset
  3. msvcrt.* keyboard functions don't clear signalled
     state from stdin handle, that's why console API
     functions are used to clear the input buffer
     instead of kbhit()/getch() loop
"""


import ctypes
import ctypes.wintypes
import subprocess

# open notepad in separate process and monitor its execution
# at the same time asynchronously processing events from
# standard input without wasting 100% CPU on looping

# OpenProcess desired access flag
#   "the right to use the object for synchronization. This
#    enables a thread to wait until the object is in the
#    signaled state"
SYNCHRONIZE=0x00100000L
# Constant to get stdin handle with GetStdHandle() call
STD_INPUT_HANDLE = -10
# Constant for infinite timeout in WaitForMultipleObjects()
INFINITE = -1

# --- processing input structures -------------------------
# INPUT_RECORD structure
#  events:
EVENTIDS = dict(
  FOCUS_EVENT = 0x0010,
  KEY_EVENT = 0x0001,      # only key event is handled
  MENU_EVENT = 0x0008,
  MOUSE_EVENT = 0x0002,
  WINDOW_BUFFER_SIZE_EVENT = 0x0004)
EVENTS = dict(zip(EVENTIDS.values(), EVENTIDS.keys()))
#  records:
class _uChar(ctypes.Union):
  _fields_ = [('UnicodeChar', ctypes.wintypes.WCHAR),
              ('AsciiChar', ctypes.wintypes.c_char)]
class KEY_EVENT_RECORD(ctypes.Structure):
  _fields_ = [
    ('keyDown', ctypes.wintypes.BOOL),
    ('repeatCount', ctypes.wintypes.WORD),
    ('virtualKeyCode', ctypes.wintypes.WORD),
    ('virtualScanCode', ctypes.wintypes.WORD),
    ('char', _uChar),
    ('controlKeyState', ctypes.wintypes.DWORD)]
class _Event(ctypes.Union):
  _fields_ = [('keyEvent', KEY_EVENT_RECORD)]
  #  MOUSE_EVENT_RECORD        MouseEvent;
  #  WINDOW_BUFFER_SIZE_RECORD WindowBufferSizeEvent;
  #  MENU_EVENT_RECORD         MenuEvent;
  #  FOCUS_EVENT_RECORD        FocusEvent;
class INPUT_RECORD(ctypes.Structure):
  _fields_ = [
    ('eventType', ctypes.wintypes.WORD),
    ('event', _Event)]
# --- /processing input structures ------------------------



np = subprocess.Popen([r"notepad"],
       stdin=subprocess.PIPE,
       stdout=subprocess.PIPE,
       stderr=subprocess.PIPE,
       shell=True)
# OpenProcess returns handle that can be used in wait functions
# params: desiredAccess, inheritHandle, processId
nph = ctypes.windll.kernel32.OpenProcess(SYNCHRONIZE, False, np.pid)
print("Started Notepad with pid=%s, handle=%s" % (np.pid, nph))

ch = ctypes.windll.kernel32.GetStdHandle(STD_INPUT_HANDLE)

handles = [ch, nph]

ctypes.windll.kernel32.FlushConsoleInputBuffer(ch)
eventnum = ctypes.wintypes.DWORD()
eventread = ctypes.wintypes.DWORD()
inbuf = (INPUT_RECORD * 1)()

print "[q]uit, [s]top console processing, launch bro[w]ser"
stopflag = False
while not stopflag and nph in handles:
    print "Waiting for handles %s.." % handles

    HandleArrayType = ctypes.wintypes.HANDLE * len(handles)
    handle_array = HandleArrayType(*handles)
    # params: count, handles, waitAll, milliseconds
    ret = ctypes.windll.kernel32.WaitForMultipleObjects(
            len(handle_array), handle_array, False, INFINITE)

    if handles[ret] == ch:
        """
        # msvcrt won't work, because it doesn't reset
        # signalled state of stdin handle
        import msvcrt
        while msvcrt.kbhit():
            print "key!"
            print msvcrt.getch()
        continue
        """
        # --- processing input ---------------------------
        ctypes.windll.kernel32.GetNumberOfConsoleInputEvents(
          ch, ctypes.byref(eventnum))
        for i in range(eventnum.value):
          # params: handler, buffer, length, eventsnum
          ctypes.windll.kernel32.ReadConsoleInputW(
            ch, ctypes.byref(inbuf), 2, ctypes.byref(eventread))
          if EVENTS[inbuf[0].eventType] != 'KEY_EVENT':
            print EVENTS[inbuf[0].eventType]
            pass
          else:
            keyEvent = inbuf[0].event.keyEvent
            if not keyEvent.keyDown:
              continue
            char = keyEvent.char.UnicodeChar.lower()
            #print char, keyEvent
            if char == 'q':
              print('[q] key pressed. Exiting..')
              stopflag = True
            elif char == 's':
              handles.remove(ch)
            elif char == 'w':
              import webbrowser
              webbrowser.open('http://techtonik.rainforce.org')
            #print char
        # --- /processing input --------------------------

    elif handles[ret] == nph:
        print("Notepad is closed. Exiting..")
        handles.remove(nph)
    else:
        print("Warning: Unknown return value '%s'" % ret)

ctypes.windll.kernel32.FlushConsoleInputBuffer(ch)
ctypes.windll.kernel32.CloseHandle(nph)
print "Done."

Where can it be useful?

Writing first Windows console IRC client in Python and make it cross-platform? Network log viewers with keyboard shortcuts? Real-time roguelikes? Matrix sniffer screensaver? I don't know - its your time. Hope I saved you some though.

Enjoy! If you want to enhance this stuff - feel free to join ever empty https://groups.google.com/forum/#!forum/rainforce for public discussion.

User Pattern: Button Status

Introduction into User Patterns

Many of you know about Design Patterns of software development, invented to make complicated solutions look less complicated. The problem with Design Patterns is that they describe solutions to coding problems - not for user problem. People abusing Design Patterns tend to align the functionality of application to patterns, sacrificing features users wanted. The application that looks well-designed in the eyes of developers, in fact makes even simple features hard to implement or just takes too much time. As time passes, users become dissatisfied and eventually lose all interest in the product.

People tried to approach the problem from the other end with User Interface Design guidelines. This often made developers unhappy, because their Design Patterns don't explain how to work with web and Ajax application where "business logic" becomes shared between client and server. Developers wanted to see how different frameworks approach this stuff and to make the development process more intuitive. For this reason there should be a way to see how some typical features/functionality are implemented in different frameworks and this should be more detailed than a wiki or a blog in 15 minutes.

So, let me introduce User Patterns. User Pattern is a very abstract story, missing details about design decisions, but detailed enough to provide implementation and see how _simple_, _extensible_ and _maintainable_ this implementation is.

Button Status User Pattern

User story: I want a page with a button and a status field displayed. When I click the button, the status should be updated.

Description: There are five levels in the quest for the implementation of this pattern.
Level 1. The status is blank - it is fetched/calculated only when the button is pressed.
Level 2. The status is initially available - and it is refreshed/calculated when the button is pressed.
Level 3. When refresh/recalculation fails, the error value is displayed in status field.
Level 4. When refresh/recalculation fails, the error value is displayed in status field along with user friendly message in separate field
Level 5. The status value is stored in application database - when the button is pressed, it is recalculated, saved and only when displayed to the user
Level 6. When recalculation/save or fetch operation fails, user should be presented with friendly message explaining the error and further action (i.e. report problem link)

I'm looking forward to create samples for AppEngine webapp, classic Django and Django-nonrel. Feel free to share your snippets.

[PySide] Python for Qt version 1.0.0~beta4 "I have altered the deal" released

I can't see this post on planet.python.org and can't resist to share the good news about new PySide release.

The PySide team is happy to announce the fourth beta release of PySide:
Python for Qt. New versions of some of the PySide toolchain components
apiextractor, generatorrunner, shiboken, libpyside, pyside-tools have been
released as well.

Like the others, this is a source code release only; we hope our community
packagers will be providing provide binary packages shortly. To acquire the
source code packages, refer to our download wiki page [1] or pull the relevant
tagged versions from our git repositories [2].

Major changes since 1.0.0~beta3
===============================

This is a bug fix release. Since beta3, a total of 47 high-priority
bugs have been fixed. See the list of fixed bugs at the end of this message.

Path towards 1.0 release
========================

There are still some outstanding bugs in our Bugzilla [3]. To have these
fixed, we plan to do other beta in two weeks. The beta cycle will continue
until we have all P2 bugs fixed.


About PySide
============

PySide is the Nokia-sponsored Python Qt bindings project, providing
access to not only the complete Qt 4.7 framework but also Qt Mobility,
as well as to generator tools for rapidly generating bindings for any
Qt-based libraries.

The PySide project is developed in the open, with all facilities you'd
expect from any modern OSS project such as all code in a git repository
[2], an open Bugzilla [5] for reporting bugs, and an open design
process [4]. We welcome any contribution without requiring a transfer of
copyright.

List of bugs fixed
==================

624     button click emit doesn't work
484     Error compiling QtContacts 1.1 (problems with const QList)
498     powerStateChanged-SIGNAL not emitted!
509     Can't use Shiboken when both Debug and Released are installed.
528     Connecting to SIGNAL positionUpdated fails
552     Segmentation fault when using QUiLoader and QTabWidget
553     A warning against using QUILoader is needed in the documentation
560     Lack of QtCore.Signal documentation
582     Python slots don't get called when they have a custom decorator
589     Crash related to QGraphicsProxyWidget and QVariant
592     shiboken.dll produces a segmentation fault when reloading a PySide module
608     Photoviewer example missing license boilerplates and shebang lines
609     Python site-packages path cannot be customized
610     QWidgetItemV2 not exposed to Python
626     Problem building PySide on OS X (qabstractfileengine_wrapper.cpp:
No such file or directory)
406     Unable to send instant messages using QMessageService
458     Doesn't build with QtMobility 1.1.0~beta2
487     Support QContactDetailFieldDefinition.setAllowableTypes
497     Miising __lt__ operators in QtMobility::QGeoMapObject
499     QFeedbackHapticsInterface and QFeedbackFileInterface are broken
511     QPainter doesn't respect Qt.NoPen
522     example/threads/mandelbrot.py crashes on exit
523     QWidget.winId() returns PyCObject (expected unsigned long)
530     Importing division from future breaks QPoint division
531     sessionProperty "ConnectInBackground" does not work
539     MCC and MNC interchanged
541     QTableWidget.itemAt(row, col) always returns item at 0, 0.
550     Can't call PySide slot from QtScript when the args are a list of anything.
556     QGraphicsScene.addItem performs very poorly when the scene has >10000 items
562     pyside-uic does not generate some layers properties
568     List insertion time grows with list size
574     In docs of QUuid there's documentation for a function called
"operator QString"
575     Strange behaviour of QTextEdit.ExtraSelection().cursor
584     python pickle module can't treat QByteArray object of PySide
591     QtCore.QRect has no attribute "getRect()" in Windows binary
611     enum values lack a tp_print implementation
614     FAil to register 2 objects in the same address
619     never automatically delete a QWidget that has no parent widget and
is visible
620     QAbstractItemModel.createIndex(int,int,PyObject*) does not
increment refcount
621     QGLWidget.bindTexture(QString) does not bind the texture correctly
622     PPA pyside is broken on Ubuntu 10.10
623     QGLWidget.bindTexture(QPixmap, GLenum, GLenum) is missing
625     QFileDialog return a tuple instead of a unicode
628     pyside-uic can't effect "headerVisible" attribute for QTreeView
and QTreeWidget
232     [FTBFS] Fails to build on hurd-i386 (Test "lock" hangs for more
than 191 minutes)
255     Test qtscripttools_debugger segfaults on ia64
298     Contact subtype not correctly set



References
==========

[1] http://developer.qt.nokia.com/wiki/PySideDownloads
[2] http://qt.gitorious.org/pyside
[3] http://bugs.openbossa.org/
[4] http://www.pyside.org/docs/pseps/psep-0001.html

Thanks
PySide team.

Python GUIniverse

I am on the PySide of the Force.

Injecting SVN bindings into Mercurial on Windows

If you need to add Subversion bindings for your Mercurial, which was installed using ordinary installer (and not TortoiseHg), then the following script can help to put all files in place.

https://bitbucket.org/techtonik/py2exe-relibzip/src/f15f8d032b5b/reinject.py

Unpack svn bindings, add all *.py files from svn/ and libsvn/ directories into the root of library.zip archive *with* these directories. Then call script with path to unpacked libsvn/. The script will generate some *.dll and *.py files in current directory. Put *.py into library.zip:/libsvn/ and *.dll into Mercurial installation directory. You may need to also copy libsvn_swig_py-1.dll into Mercurial dir.

The script can be tuned to insert C extensions to any py2exe distribution compiled with bundle_files=3