Saturday, November 12, 2011
The exciting thing that this project uses patched version of python-vnc-viewer written by Chris Liechti that I've uploaded to Google Code, because it had to be updated to work with modern versions of required libraries. It is nice to see this effort brought up something cool today.
Sunday, August 28, 2011
See detailed info here.
$ df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 75M 4.0K 75M 1% /run/shm
*RAM disk* is a term from the past when DOS was alive and information was stored on disks instead of internet. If you created image of some disk, it was possible to load it into memory. Memory disks were useful to load software from Live CDs. Usually software needs some space to write data during boot sequence, and RAM is the fastest way to setup one.
Filesystem space in memory can be extremely useful today too. For example, to run tests without reducing resource of SSD. While the idea is not new, there was no incentive to explore it until I've run upon tmpfs reference in Ubuntu Wiki.
For example, to get 2Gb of space for files in RAM, edit /etc/fstab to add the following line:
/var/ramspace is now the place to store your files in memory.
tmpfs /var/ramspace tmpfs defaults,size=2048M 0 0
Sunday, July 03, 2011
""" Implementation of Raw Audio Socket server spec in pure Python http://code.google.com/p/rainforce/wiki/RawAudioSocket """ import sys #-- CHAPTER 1: CONTINUOUS SOUND PLAYBACK WITH WINDOWS WINMM LIBRARY -- # # Based on tutorial "Playing Audio in Windows using waveOut Interface" # by David Overton import ctypes from ctypes import wintypes # 1. Open Sound Device # --- define necessary data structures from mmsystem.h HWAVEOUT = wintypes.HANDLE WAVE_FORMAT_PCM = 0x1 WAVE_MAPPER = -1 CALLBACK_NULL = 0 MMSYSERR_NOERROR = 0 class WAVEFORMATEX(ctypes.Structure): _fields_ = [ ('wFormatTag', wintypes.WORD), # 0x0001 WAVE_FORMAT_PCM. PCM audio # 0xFFFE The format is specified in the WAVEFORMATEXTENSIBLE.SubFormat # Other values are in mmreg.h ('nChannels', wintypes.WORD), ('SamplesPerSec', wintypes.DWORD), ('AvgBytesPerSec', wintypes.DWORD), # for WAVE_FORMAT_PCM is the product of nSamplesPerSec and nBlockAlign ('nBlockAlign', wintypes.WORD), # for WAVE_FORMAT_PCM is the product of nChannels and wBitsPerSample # divided by 8 (bits per byte) ('wBitsPerSample', wintypes.WORD), # for WAVE_FORMAT_PCM should be equal to 8 or 16 ('cbSize', wintypes.WORD)] # extra format information size, should be 0 # --- /define # Data must be processes in pieces that are multiple of # nBlockAlign bytes of data at a time. Written and read # data from a device must always start at the beginning # of a block. Playback of PCM data can not be started in # the middle of a sample on a non-block-aligned boundary. hwaveout = HWAVEOUT() wavefx = WAVEFORMATEX( WAVE_FORMAT_PCM, 2, # nChannels 44100, # SamplesPerSec 705600,# AvgBytesPerSec = 44100 SamplesPerSec * 16 wBitsPerSample 4, # nBlockAlign = 2 nChannels * 16 wBitsPerSample / 8 bits per byte 16, # wBitsPerSample 0 ) # Open default wave device ret = ctypes.windll.winmm.waveOutOpen( ctypes.byref(hwaveout), # buffer to receive a handle identifying # the open waveform-audio output device WAVE_MAPPER, # constant to point to default wave device ctypes.byref(wavefx), # identifier for data format sent for device 0, # DWORD_PTR dwCallback - callback mechanizm 0, # DWORD_PTR dwCallbackInstance - user instance data for callback CALLBACK_NULL # DWORD fdwOpen - flag for opening the device ) if ret != MMSYSERR_NOERROR: sys.exit('Error opening default waveform audio device (WAVE_MAPPER)') print "Default Wave Audio output device is opened successfully" # 2. Write Audio Blocks to Device # --- define necessary data structures PVOID = wintypes.HANDLE WAVERR_BASE = 32 WAVERR_STILLPLAYING = WAVERR_BASE + 1 class WAVEHDR(ctypes.Structure): _fields_ = [ ('lpData', wintypes.LPSTR), # pointer to waveform buffer ('dwBufferLength', wintypes.DWORD), # in bytes ('dwBytesRecorded', wintypes.DWORD), # when used in input ('dwUser', wintypes.DWORD), # user data ('dwFlags', wintypes.DWORD), ('dwLoops', wintypes.DWORD), # times to loop, for output buffers only ('lpNext', PVOID), # reserved, struct wavehdr_tag *lpNext ('reserved', wintypes.DWORD)] # reserved # The lpData, dwBufferLength, and dwFlags members must be set before calling # the waveInPrepareHeader or waveOutPrepareHeader function. (For either # function, the dwFlags member must be set to zero.) # --- /define class AudioWriter(object): def __init__(self, hwaveout): self.hwaveout = hwaveout self.wavehdr = WAVEHDR() def write(self, data): self.wavehdr.dwBufferLength = len(data) self.wavehdr.lpData = data # Prepare block for playback if ctypes.windll.winmm.waveOutPrepareHeader( self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr) ) != MMSYSERR_NOERROR: sys.exit('Error: waveOutPrepareHeader failed') # Write block, returns immediately unless a synchronous driver is # used (not often) if ctypes.windll.winmm.waveOutWrite( self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr) ) != MMSYSERR_NOERROR: sys.exit('Error: waveOutWrite failed') # [ ] calculate sleep delay based on sample length # iii [ ] Measure CPU usage spike during wait without delay import time time.sleep(1) # Wait until playback is finished while True: # unpreparing the header fails until the block is played ret = ctypes.windll.winmm.waveOutUnprepareHeader( self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr) ) if ret == WAVERR_STILLPLAYING: import time time.sleep(1) continue if ret != MMSYSERR_NOERROR: sys.exit('Error: waveOutUnprepareHeader failed with code 0x%x' % ret) break # [ ] it's no good to read all the PCM data into memory at once data = open('95672__Corsica_S__frequency_change_approved.raw', 'rb').read() aw = AudioWriter(hwaveout) aw.write(data) # x. Close Sound Device ctypes.windll.winmm.waveOutClose(hwaveout) print "Default Wave Audio output device is closed" #-- /CHAPTER 1 --
Thursday, May 19, 2011
$ svn up U bitten\slave.py G bitten\tests\queue.py Conflict discovered in 'bitten/queue.py'. Select: (p) postpone, (df) diff-full, (e) edit, (mc) mine-conflict, (tc) theirs-conflict, (s) show all options:
Should you select "theirs conflict" or "mine confict" to fix your code? In situation when you feel that incoming change is right you may want to select "mine conflict" as the change that conflicts and should be removed as a result. But the correct answer it to select "theirs conflict" as the change to be incorporated in your code. Subversion developers think that you're are not removing conflicts, but incorporating them into your code. If they could think of user action - it is removing or resolving conflict in high-level terminology, or selecting correct change in low-level description.
Let's see the help:
Conflict discovered in 'bitten/slave.py'. Select: (p) postpone, (df) diff-full, (e) edit, (mc) mine-conflict, (tc) theirs-conflict, (s) show all options: s (e) edit - change merged file in an editor (df) diff-full - show all changes made to merged file (r) resolved - accept merged version of file (dc) display-conflict - show all conflicts (ignoring merged version) (mc) mine-conflict - accept my version for all conflicts (same) (tc) theirs-conflict - accept their version for all conflicts (same) (mf) mine-full - accept my version of entire file (even non-conflicts) (tf) theirs-full - accept their version of entire file (same) (p) postpone - mark the conflict to be resolved later (l) launch - launch external tool to resolve conflict (s) show all - show this list
Frankly, I can't understand how (mc) is different from (mf) if both versions accept all changes. (same) marker doesn't add much sense either. But even if help doesn't describe anything - these options were still added for a reason, and if we meditate a bit on them starting from the bottom comment to (mf) help, it becomes clear that:
1. (mf) throws away any incoming changes for the given file even if some of them were merged successfully
2. that means (mc) probably keeps the changes that were merged successfully
3. that means (df) shows more changes than (dc) output
Recommendations? Introduce `change` and `conflict` terminology earlier, i.e. show info like `3 changes, 2 conflicts`. It will also allow incremental conflict resolution, like if you (e)dited the counter is changed to `4 changes, 1 conflict` etc. Final recommendation if to replace `(ignoring merged version)` with `(ignoring merged changes)` string. There are surely more ways to improve this.
Friday, April 29, 2011
Wednesday, April 06, 2011
Fortunately, if your project is managed by fine grained build system such as SCons, if your build scripts are not globbing too much, there are chances you can find files that are not participating in the builds.
Here is how to do this on Windows using Process Monitor tool that intercepts all system calls including file access.
While build systems are usually common for C/C++ and Java projects, it is possible to add fine-grained file usage control for any project. For example, SCons itself is written entirely in Python, it could run directly from the source checkout or build distributives from checkout. But instead, it uses build procedure to copy all necessary files from checkout into separate directory and do stuff from there.
Thanks to that it is possible to see which files are no more actual. While it is possible to compare checkout source tree and copied directory trees, I'll go through the hells of monitoring system file access in a source tree during the build process using Process Monitor (FileMon in the past). Linux should have similar tools too - let me know how are they called.
The process is the following:
- Start Process Monitor
- Stop incoming event flood by (un)clicking Capture (Ctrl-E) button
- Open Filter (Ctrl-L) dialog to add some filters
- Go Tools -> File Summary...
- Export to CSV using Save...
SCons build is started by
bootstrap.py script from a root of SCons source checkout. The script is executed by python executable, so I add
python.exe process name to the filter. I know that
bootstrap.py copies files from
src/ subdirectory, so it is the directory I need to monitor, so I add this dir to filters too.
There is a list of paths catched by Process Monitor when listening to system calls. They are already filtered, but additional filters can be applied using bottom left button to make information even more useful.
Exported CSV is not very useful without some postprocessing. I used the following a script to compare the list of paths in CSV to actual
src/directory contents. This gives me names of files that were not touched during build at all.
I've found a few interesting things about SCons. Core tests are mixed with source files in repository checkout. They are not copied during bootstrap build. There are also few
SRCDIR = "C:\\p\\python\\scons\\src" CSVLIST = 'accessed_bootstrap_files.CSV' import csv import os reader = csv.reader(open(CSVLIST)) header = reader.next() pathidx = header.index("Path") pathset = set([row[pathidx] for row in reader]) #for row in pathset: # print row fileset = set() for root, dirs, files in os.walk(SRCDIR): fileset.update( [os.path.join(root, f) for f in files] ) if '.svn' in dirs: dirs.remove('.svn') # don't visit .svn directories if len(pathset & fileset) == 0: print 'Error: File sets do not intersect at all' print "Files not found in source directory tree:" for f in (pathset - fileset): if not os.path.isdir(f): print f print print "Untouched files in source directory tree:" for f in sorted(fileset - pathset): if not os.path.isdir(f): print f
setup.pyfiles, post-install script and announcement that don't participate in the build.
Here is the output of the above script:
Files not found in source directory tree:
Untouched files in source directory tree:
Hope this helps clean up your projects too.
P.S. I wish there was a Python script replacement for Process Monitor, or at least that it could be controlled from command line.
Tuesday, March 29, 2011
What is asynchronous input?
Given: The program need to terminate immediately when a subprocess exits or user hits 'q'. It should not eat 100% CPU time.
Blocking synchronous input:
import sys, subprocess
from msvcrt import getch, kbhit
p = subprocess.Popen([r"notepad"], shell=True)
char = getch()
if char == 'q':
sys.exit('terminated by user')
if p.poll() != None:
sys.exit('terminated by child exit')
This one doesn't work (doesn't exits immediately when child process terminates), because getch() blocks execution key is pressed, and even if child process exits, it won't be detected until some key is pressed.
Non-blocking synchronous input (polling):
import sys, subprocess
from msvcrt import getch, kbhit
p = subprocess.Popen([r"notepad"], shell=True)
if getch() == 'q':
sys.exit('terminated by user')
if p.poll() != None:
sys.exit('terminated by child exit')
This works ok, but constant polling uses 100% of CPU resources. It is possible to insert
time.sleep()instructions to reduce the carbon footprint, but these crutches will greatly slow down the console when you add z-type to console to spend your time until the background process is finished.
Asynchronous console input on Windows with Python
Asynchronous input allows your program to receive notification from operating system when an input is available. This means that you define events your program needs to react (not necessary console events), send this list to operating system and put your program into
waitmode. When system sees this event, your wait function returns and it's possible to inspect/filter event to a greater detail.
Windows provides WaitForMultipleObjects wait function, and in Linux I believe it is select call. If you've tried to understand how Twisted works, but couldn't - will it make more clear if I say that twisted reactor is a wait function? What you do when you code with twisted is just configuring events to react, making chains of them so that one event reacts to another event. This allow to build very interesting, complex and de-coupled asynchronous applications in a couple of days.
O.k. Getting back from Twisted to the asynchronous console input on Windows. Below is the full source code that uses WaitForMultipleObjects. I am afraid that's minimal complete example possible to build with ctypes using Windows API. Tested with Python 2.5
Non-blocking asynchronous input from console:
Example of non-blocking asynchronous console input using
Windows API calls in Python. This can become handy for
async console tools such as IRC client.
Public domain or MIT license
by anatoly techtonik
1. WaitForMultipleObjects is used to listen for the
signals from process and stdin handles
2. When handle is signalled it remains in this state
3. msvcrt.* keyboard functions don't clear signalled
state from stdin handle, that's why console API
functions are used to clear the input buffer
instead of kbhit()/getch() loop
# open notepad in separate process and monitor its execution
# at the same time asynchronously processing events from
# standard input without wasting 100% CPU on looping
# OpenProcess desired access flag
# "the right to use the object for synchronization. This
# enables a thread to wait until the object is in the
# signaled state"
# Constant to get stdin handle with GetStdHandle() call
STD_INPUT_HANDLE = -10
# Constant for infinite timeout in WaitForMultipleObjects()
INFINITE = -1
# --- processing input structures -------------------------
# INPUT_RECORD structure
EVENTIDS = dict(
FOCUS_EVENT = 0x0010,
KEY_EVENT = 0x0001, # only key event is handled
MENU_EVENT = 0x0008,
MOUSE_EVENT = 0x0002,
WINDOW_BUFFER_SIZE_EVENT = 0x0004)
EVENTS = dict(zip(EVENTIDS.values(), EVENTIDS.keys()))
_fields_ = [('UnicodeChar', ctypes.wintypes.WCHAR),
_fields_ = [
_fields_ = [('keyEvent', KEY_EVENT_RECORD)]
# MOUSE_EVENT_RECORD MouseEvent;
# WINDOW_BUFFER_SIZE_RECORD WindowBufferSizeEvent;
# MENU_EVENT_RECORD MenuEvent;
# FOCUS_EVENT_RECORD FocusEvent;
_fields_ = [
# --- /processing input structures ------------------------
np = subprocess.Popen([r"notepad"],
# OpenProcess returns handle that can be used in wait functions
# params: desiredAccess, inheritHandle, processId
nph = ctypes.windll.kernel32.OpenProcess(SYNCHRONIZE, False, np.pid)
print("Started Notepad with pid=%s, handle=%s" % (np.pid, nph))
ch = ctypes.windll.kernel32.GetStdHandle(STD_INPUT_HANDLE)
handles = [ch, nph]
eventnum = ctypes.wintypes.DWORD()
eventread = ctypes.wintypes.DWORD()
inbuf = (INPUT_RECORD * 1)()
print "[q]uit, [s]top console processing, launch bro[w]ser"
stopflag = False
while not stopflag and nph in handles:
print "Waiting for handles %s.." % handles
HandleArrayType = ctypes.wintypes.HANDLE * len(handles)
handle_array = HandleArrayType(*handles)
# params: count, handles, waitAll, milliseconds
ret = ctypes.windll.kernel32.WaitForMultipleObjects(
len(handle_array), handle_array, False, INFINITE)
if handles[ret] == ch:
# msvcrt won't work, because it doesn't reset
# signalled state of stdin handle
# --- processing input ---------------------------
for i in range(eventnum.value):
# params: handler, buffer, length, eventsnum
ch, ctypes.byref(inbuf), 2, ctypes.byref(eventread))
if EVENTS[inbuf.eventType] != 'KEY_EVENT':
keyEvent = inbuf.event.keyEvent
if not keyEvent.keyDown:
char = keyEvent.char.UnicodeChar.lower()
#print char, keyEvent
if char == 'q':
print('[q] key pressed. Exiting..')
stopflag = True
elif char == 's':
elif char == 'w':
# --- /processing input --------------------------
elif handles[ret] == nph:
print("Notepad is closed. Exiting..")
print("Warning: Unknown return value '%s'" % ret)
Where can it be useful?
Writing first Windows console IRC client in Python and make it cross-platform? Network log viewers with keyboard shortcuts? Real-time roguelikes? Matrix sniffer screensaver? I don't know - its your time. Hope I saved you some though.
Enjoy! If you want to enhance this stuff - feel free to join ever empty https://groups.google.com/forum/#!forum/rainforce for public discussion.
Tuesday, March 22, 2011
Introduction into User Patterns
Many of you know about Design Patterns of software development, invented to make complicated solutions look less complicated. The problem with Design Patterns is that they describe solutions to coding problems - not for user problem. People abusing Design Patterns tend to align the functionality of application to patterns, sacrificing features users wanted. The application that looks well-designed in the eyes of developers, in fact makes even simple features hard to implement or just takes too much time. As time passes, users become dissatisfied and eventually lose all interest in the product.
People tried to approach the problem from the other end with User Interface Design guidelines. This often made developers unhappy, because their Design Patterns don't explain how to work with web and Ajax application where "business logic" becomes shared between client and server. Developers wanted to see how different frameworks approach this stuff and to make the development process more intuitive. For this reason there should be a way to see how some typical features/functionality are implemented in different frameworks and this should be more detailed than a wiki or a blog in 15 minutes.
So, let me introduce User Patterns. User Pattern is a very abstract story, missing details about design decisions, but detailed enough to provide implementation and see how _simple_, _extensible_ and _maintainable_ this implementation is.
Button Status User Pattern
User story: I want a page with a button and a status field displayed. When I click the button, the status should be updated.
Description: There are five levels in the quest for the implementation of this pattern.
Level 1. The status is blank - it is fetched/calculated only when the button is pressed.
Level 2. The status is initially available - and it is refreshed/calculated when the button is pressed.
Level 3. When refresh/recalculation fails, the error value is displayed in status field.
Level 4. When refresh/recalculation fails, the error value is displayed in status field along with user friendly message in separate field
Level 5. The status value is stored in application database - when the button is pressed, it is recalculated, saved and only when displayed to the user
Level 6. When recalculation/save or fetch operation fails, user should be presented with friendly message explaining the error and further action (i.e. report problem link)
I'm looking forward to create samples for AppEngine webapp, classic Django and Django-nonrel. Feel free to share your snippets.
Friday, January 21, 2011
The PySide team is happy to announce the fourth beta release of PySide: Python for Qt. New versions of some of the PySide toolchain components apiextractor, generatorrunner, shiboken, libpyside, pyside-tools have been released as well. Like the others, this is a source code release only; we hope our community packagers will be providing provide binary packages shortly. To acquire the source code packages, refer to our download wiki page  or pull the relevant tagged versions from our git repositories . Major changes since 1.0.0~beta3 =============================== This is a bug fix release. Since beta3, a total of 47 high-priority bugs have been fixed. See the list of fixed bugs at the end of this message. Path towards 1.0 release ======================== There are still some outstanding bugs in our Bugzilla . To have these fixed, we plan to do other beta in two weeks. The beta cycle will continue until we have all P2 bugs fixed. About PySide ============ PySide is the Nokia-sponsored Python Qt bindings project, providing access to not only the complete Qt 4.7 framework but also Qt Mobility, as well as to generator tools for rapidly generating bindings for any Qt-based libraries. The PySide project is developed in the open, with all facilities you'd expect from any modern OSS project such as all code in a git repository , an open Bugzilla  for reporting bugs, and an open design process . We welcome any contribution without requiring a transfer of copyright. List of bugs fixed ================== 624 button click emit doesn't work 484 Error compiling QtContacts 1.1 (problems with const QList
) 498 powerStateChanged-SIGNAL not emitted! 509 Can't use Shiboken when both Debug and Released are installed. 528 Connecting to SIGNAL positionUpdated fails 552 Segmentation fault when using QUiLoader and QTabWidget 553 A warning against using QUILoader is needed in the documentation 560 Lack of QtCore.Signal documentation 582 Python slots don't get called when they have a custom decorator 589 Crash related to QGraphicsProxyWidget and QVariant 592 shiboken.dll produces a segmentation fault when reloading a PySide module 608 Photoviewer example missing license boilerplates and shebang lines 609 Python site-packages path cannot be customized 610 QWidgetItemV2 not exposed to Python 626 Problem building PySide on OS X (qabstractfileengine_wrapper.cpp: No such file or directory) 406 Unable to send instant messages using QMessageService 458 Doesn't build with QtMobility 1.1.0~beta2 487 Support QContactDetailFieldDefinition.setAllowableTypes 497 Miising __lt__ operators in QtMobility::QGeoMapObject 499 QFeedbackHapticsInterface and QFeedbackFileInterface are broken 511 QPainter doesn't respect Qt.NoPen 522 example/threads/mandelbrot.py crashes on exit 523 QWidget.winId() returns PyCObject (expected unsigned long) 530 Importing division from future breaks QPoint division 531 sessionProperty "ConnectInBackground" does not work 539 MCC and MNC interchanged 541 QTableWidget.itemAt(row, col) always returns item at 0, 0. 550 Can't call PySide slot from QtScript when the args are a list of anything. 556 QGraphicsScene.addItem performs very poorly when the scene has >10000 items 562 pyside-uic does not generate some layers properties 568 List insertion time grows with list size 574 In docs of QUuid there's documentation for a function called "operator QString" 575 Strange behaviour of QTextEdit.ExtraSelection().cursor 584 python pickle module can't treat QByteArray object of PySide 591 QtCore.QRect has no attribute "getRect()" in Windows binary 611 enum values lack a tp_print implementation 614 FAil to register 2 objects in the same address 619 never automatically delete a QWidget that has no parent widget and is visible 620 QAbstractItemModel.createIndex(int,int,PyObject*) does not increment refcount 621 QGLWidget.bindTexture(QString) does not bind the texture correctly 622 PPA pyside is broken on Ubuntu 10.10 623 QGLWidget.bindTexture(QPixmap, GLenum, GLenum) is missing 625 QFileDialog return a tuple instead of a unicode 628 pyside-uic can't effect "headerVisible" attribute for QTreeView and QTreeWidget 232 [FTBFS] Fails to build on hurd-i386 (Test "lock" hangs for more than 191 minutes) 255 Test qtscripttools_debugger segfaults on ia64 298 Contact subtype not correctly set References ==========  http://developer.qt.nokia.com/wiki/PySideDownloads  http://qt.gitorious.org/pyside  http://bugs.openbossa.org/  http://www.pyside.org/docs/pseps/psep-0001.html Thanks PySide team.
Tuesday, January 18, 2011
Saturday, January 08, 2011
Unpack svn bindings, add all *.py files from svn/ and libsvn/ directories into the root of library.zip archive *with* these directories. Then call script with path to unpacked libsvn/. The script will generate some *.dll and *.py files in current directory. Put *.py into library.zip:/libsvn/ and *.dll into Mercurial installation directory. You may need to also copy libsvn_swig_py-1.dll into Mercurial dir.
The script can be tuned to insert C extensions to any py2exe distribution compiled with bundle_files=3