About Environment

In Python applications environment is often an ambiguous term that needs clarification. In general sense `the environment` is system environment with PATH and friends accessible from os.environ within Python. But in Python applications it can mean different things.

In Trac `the environment` is a directory with settings, database and other files related to one Trac site.

In SCons `the environment` is a structure in memory that holds dependency trees, helper functions, builders and other stuff. It is written to disk only for caching.


Quite many other applications have some kind of environment for their own purpose with meanings close to either Trac or SCons, which often confuses newbies or strangers who are not aware about the context. Software development clearly needs more specific terms in English to make people write and read in the same language without those excessive contexts.

Spyder IDE Internals: Highlighting in 2.2.0dev

UPD: Highligher support in code editor have been improved, so its instance can now be traced easily.

The entrypoint to syntax highlighting in Spyder IDE is located in CodeEditor widget at spyderlib/widgets/sourcecode/codeeditor.py

CodeEditor widget is basically file content in one of main tabs. The whole stack of tabs is called EditorStack. So editors are grouped in editor stacks, each editor renders one file, and each editor has its own highligher created from assigned self.highlighter_class

Highlighers are implemented with Qt's QSyntaxHighlighter. No pygments, nothing like this, so can't say if Qt is faster, but it should be. Default self.highligher_class for every code editor is TextSH. The actual instance is created by set_language() method called from setup_editor(). If setup_editor() is not called, the highlighter can be unset, but frankly I don't know what's the purpose of using such editor.

if set, syntax highlighter (self.highlighter) is responsible for:

• coloring raw text data inside editor on load
• coloring text data when editor is cloned
• updating document highlight on line edits
• providing color palette (scheme) for the editor
• providing data for Outliner

self.highlighter is not responsible for:

• background highlight for current line
• background highlight for search / current line occurrences

Enjoy hacking.

Multidimensional programming

From time to time I reverse some piece of open source code just to understand how it works. The biggest problem with that is the amount of things I need to juggle in my head until they find a place on the canvas of reversed blueprint. If there are too many - I give up or put the project on the back burner. This was with Stackless and Twisted, but I really glad to finally get to them.

Quite often is starts with some bug that seems easy to fix and I go for it. In ideal world it shouldn't take more than 15 minutes for studying the code to gain understanding what should be changed, but it is also important to have a confidence that the change won't break anything.

I won't tell you how to deal with that complexity, but I'd like to share an idea that I've got from Large Hadron Collider. =) Let me tell you that story...


I always had troubles trying to squeeze more than 3 dimensions into my head. I could imagine a dot moving along X or Y axes easily, could imagine it moving along Z axis, but everything more than that caused a confusion.

I could have my brain exploded when some time ago a friend of mine tried to explain what's going on in LHC. He said that physics and mathematicians are trying to figure out how many dimensions our universe has. They assume that our universe has more than 4 dimensions (3 coordinates and 1 time value). In fact they argue that the truth is somewhere between 9 and 26!

I am not a fan of scientific theories - my neurons are pretty calm to that matter, but when you just need to understand, because the person in front of you is knowledgeable and tries all the best to explain  - here is when you start to feel colliding brain cells under your skull. To the honor of my friend, he didn't try to build-up suspense (as I would probably do before explaining properly) and managed to draw a clear picture in my mind. A standard plot with two axes - X and Y, and a dot.

"Look at this dot on X/Y plane. This dot has coordinates in two dimensions - X and Y", - he explained. "These are only two you can see here. But this dot is from the real world, so it also has Z coordinate", - he added putting a label "Z=0.1" next to the dot. "This dot also have speed", - he drew "V=0.1 m/s". "Now we can see values for all 4 traditional dimensions of our dot, but we can add more. There are many things that we can classify our dot. It probably has color.", - he drew a color box and a hex value. "It has temperature", - a new label "t = 25С" appeared in the column. "There are 6 already, and you can add your own.". I immediately imagined a dot travelling through canvas of "Stars!" game with all those labels nearby that constantly change their values as the dot moved. "Wow! Now I see", - that was a nice feeling - I must admit I can be pretty dumb sometimes. =) It didn't make a science fan out of me, but did make an important short-circuit in the depths of my head, which popped up a few months later..

A few months later.

As it usually happens in software development you need at least some basic design before setting down to code, and with time design phase completely faded from our process into "who made it first - wins" motto. It is hard to argue with that, so I didn't. But as a result after some time, the stability of our releases dropped. People were not communicating, and started to forget about some aspects of our system that could fire at any time in completely unexpected places. Even though all our code undergoes reviews, the reviewers tend to forget about those aspect as well. The process went out of control - we couldn't keep all the aspects in our heads when planning for the next feature to be released, and I found myself in same state I was when trying to understand the complexity of multidimensional string theories. That lead me to the idea that we need to control the amount of aspects that we need to keep in mind when coding and restructure our architecture to keep those aspects at minimum. This will help us to regain sense of confidence into what we are doing, and save some money from the bills from the nearest bar on the name of our release manager.


So, the `multidimensional programming` concept means that at any point of your code there are multiple things that you should be aware of. These things are the 'dimensions', and the more you have - the more complex your application is. Basically, these are the things that can be broken by any change at this place. Good application architecture is orthogonal - you can work at only one dimension at a time without thinking too much about all others. But you need to know all of them anyway to gain a sense of confidence.


For example, a recent change in Spyder IDE requires me to rename some file. This breaks the code, which I grep and fix - that's one dimension. But it will also likely to break a translation for the strings in that file, because the string is now at a different place. I imagine that nobody will be interested to check and translate the same stuff over and over again, so that's one more thing I'd like to avoid, so I should keep that in mind.

Another example are web applications. You need to keep in mind 'user privileges', type of HTTP request ('ajax', ...) and response ('json', ...) required. You need to make sure `critical errors` are handled and reported, and `static files` are correctly served by web server. You need to save incomplete `data between requests`, and cleanup it where possible. Make sure there is sufficient `XSRF protection` and `browser compatibility`. There are a lot more to it, and so far these have nothing to do with the logic of your web application. Frameworks help to deal with that, but inside they are still multidimensional. If framework is not flexible for you - that probably means it tries to keep some dimensions orthogonal, and there could be a good reason for that.


Maybe that's not much, but at least now you can argument that Large Hadron Collider experiments have much in common with software engineering, and when somebody asks about your job - you can proudly state that you're on par with scientists with their string theory, but in your own big enterprise application universe. =)

Rietveld architecture: AppEngine/Django request processing

Just a quick note/reminder of the request handling flow in AppEngine environment for mixed AE/Django application such a Rietveld . Hopefully it provides a good entrypoint to understand how AppEngine works. I use Rietveld as an example, because this project is basically born to show how to run Django on AE.

Rietveld is a Django application that is run by AppEngine. Let's leave all Django stuff aside and learn how AppEngine loads and initializes Python applications first.


Import and execution in Python web apps

In PHP when your code is executed, it is read and interpreted (executed) from start to finish every time a new request arrives. In Python the code is read only once (imported), executed and on subsequent requests only the part that handles request is invoked over and over. The catch is that first request is always different in Python.


What happens when application is uploaded to AppEngine?


Step 1. Standard AppEngine application loading and initialization sequence

• AppEngine reads app.yaml to understand how to load application (which version of Python it requires and which URLs are handled by which Python scripts)

• AppEngine initializes application by creating an instance for it

• Then it looks at URL and executes script that shoud process this URL according to app.yaml

This stuff is actual for every AppEngine application.


Example: --- app.yaml from Rietveld project --->

It is the entrypoint to understand any AppEngine app. If you want to know what is called when you request an URL - first thing to do is to look there.

Step 2. Python code to fine-tune AppEngine params and configure request handler

All requests in Rietveld (except static files) are handled by main.py. It does the following:

• Imports appengine_config.py that in turn:
• Initializes and tunes Appstats tool
• Chooses version of Django to use (1.2 currently)
• Configures Django to read settings.py with Rietveld specific parameters
• Adds logger for all exceptions
• Removes Django's DB rollback event handler (because Rietveld doesn't use DB layer of Django)
• Creates request handler using Django
• Passes handler to AE's run_wsgi_app() util to give Django control to process request

Django didn't fire at this point, and nothing magical happened. 

Step 3. Request handling magic

Request handling starts with run_wsgi_app() - this magical function implicitly imports appengine_config.py to read its own settings behind the scenes and then gives control to Django handler created earlier.

Django reads its settings.py mentioned earlier and processes options before executing anything application/request specific:

• Configures middlewares - that's important, because they provide such things as user object in request:
• django.middleware.common.CommonMiddleware  - doesn't seem to be used (docs)
• django.middleware.http.ConditionalGetMiddleware - not sure why it is needed
• codereview.middleware.AddUserToRequestMiddleware - this one also fetches user-specific parameters from Account record
• codereview.middleware.PropagateExceptionMiddleware - logs and rewrites exceptions to be more user-friendly
• Sets urls.py to be the ROOT_URLCONF - mapping between URLs and handler functions in views.py
• Enables django.core.context_processors.request which adds `request` object to templates
• Configures template loaders
• Configures file uploads
• Configures URL to generate path to static files as `/static/`
• Rietveld own constants like incoming email address are also defined here

After all above is done, Django handler starts processing the request:

• It looks into urls.py to find what function should process requested URL
• urls.py is a redirect to codereview/urls.py with actual mapping, so it reads the latter as well
• Finds associated function name and calls this function from views.py

And that's basically the entrypoint that you need to start hacking Rietveld/Django and AppEngine.

Python based VNC client in browser

What I really like in open source is when people reuse existing stuff to bring up something new. Today Arkaitz Jimenez brought up a way to connect to VNC server with browser using hybrid solution from Python, JavaScript, gevent and WebSockets.

The exciting thing that this project uses patched version of python-vnc-viewer written by Chris Liechti that I've uploaded to Google Code, because it had to be updated to work with modern versions of required libraries. It is nice to see this effort brought up something cool today.

How to make RAM disk in Linux

UPDATE (2014-08-27): Exactly three years later I discovered that Linux already comes with RAM disk enabled by default, mounted as `/dev/shm` (which points to `/run/shm` on Debian/Ubuntu):

$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            75M  4.0K   75M   1% /run/shm

See detailed info here.

*RAM disk* is a term from the past when DOS was alive and information was stored on disks instead of internet. If you created image of some disk, it was possible to load it into memory. Memory disks were useful to load software from Live CDs. Usually software needs some space to write data during boot sequence, and RAM is the fastest way to setup one.

Filesystem space in memory can be extremely useful today too. For example, to run tests without reducing resource of SSD. While the idea is not new, there was no incentive to explore it until I've run upon tmpfs reference in Ubuntu Wiki.

For example, to get 2Gb of space for files in RAM, edit /etc/fstab to add the following line:

tmpfs     /var/ramspace       tmpfs     defaults,size=2048M     0     0

/var/ramspace is now the place to store your files in memory.

Audio output on Windows with pure Python + ctypes

I want to make a simple walkie-talkie for a local LAN to have fun with my friends. I've created Raw Audio Socket spec to keep it really simple. We have a lot of different operating systems, so it would be nice to have a single implementation that can be run cross-platform. I chose Python and started with getting the sound playing on Windows.

Python is known for its easy integration with C libraries and, indeed, there is a lot of Python bindings calling audio libraries written in C from Python. However, it requires the library to be compiled for your specific platform. I chose to avoid dependencies on any C code and instead call Windows WinMM Multimedia API  directly with ctypes module.

Thanks to an excellent tutorial by David Overton, here is the pure Python proof of concept that plays standard CD Audio PCM 44.1kHz 16bit Stereo sample from external file to the default sound device.

"""
Implementation of Raw Audio Socket server spec in pure Python
http://code.google.com/p/rainforce/wiki/RawAudioSocket

"""

import sys

#-- CHAPTER 1: CONTINUOUS SOUND PLAYBACK WITH WINDOWS WINMM LIBRARY --
#
# Based on tutorial "Playing Audio in Windows using waveOut Interface"
# by David Overton

import ctypes
from ctypes import wintypes


# 1. Open Sound Device

# --- define necessary data structures from mmsystem.h
HWAVEOUT = wintypes.HANDLE
WAVE_FORMAT_PCM = 0x1
WAVE_MAPPER = -1
CALLBACK_NULL = 0
MMSYSERR_NOERROR = 0

class WAVEFORMATEX(ctypes.Structure):
  _fields_ = [
    ('wFormatTag',  wintypes.WORD),
      # 0x0001 WAVE_FORMAT_PCM. PCM audio
      # 0xFFFE The format is specified in the WAVEFORMATEXTENSIBLE.SubFormat
      # Other values are in mmreg.h 
    ('nChannels',   wintypes.WORD),
    ('SamplesPerSec',  wintypes.DWORD),
    ('AvgBytesPerSec', wintypes.DWORD),
      # for WAVE_FORMAT_PCM is the product of nSamplesPerSec and nBlockAlign
    ('nBlockAlign', wintypes.WORD),
      # for WAVE_FORMAT_PCM is the product of nChannels and wBitsPerSample
      # divided by 8 (bits per byte)
    ('wBitsPerSample', wintypes.WORD),
      # for WAVE_FORMAT_PCM should be equal to 8 or 16
    ('cbSize',      wintypes.WORD)]
      # extra format information size, should be 0
# --- /define

# Data must be processes in pieces that are multiple of
# nBlockAlign bytes of data at a time. Written and read
# data from a device must always start at the beginning
# of a block. Playback of PCM data can not be started in
# the middle of a sample on a non-block-aligned boundary.

hwaveout = HWAVEOUT()
wavefx = WAVEFORMATEX(
  WAVE_FORMAT_PCM,
  2,     # nChannels
  44100, # SamplesPerSec
  705600,# AvgBytesPerSec = 44100 SamplesPerSec * 16 wBitsPerSample
  4,     # nBlockAlign = 2 nChannels * 16 wBitsPerSample / 8 bits per byte
  16,    # wBitsPerSample
  0
)

# Open default wave device
ret = ctypes.windll.winmm.waveOutOpen(
  ctypes.byref(hwaveout), # buffer to receive a handle identifying
                          # the open waveform-audio output device
  WAVE_MAPPER,            # constant to point to default wave device
  ctypes.byref(wavefx),   # identifier for data format sent for device
  0, # DWORD_PTR dwCallback - callback mechanizm
  0, # DWORD_PTR dwCallbackInstance - user instance data for callback
  CALLBACK_NULL # DWORD fdwOpen - flag for opening the device
)

if ret != MMSYSERR_NOERROR:
  sys.exit('Error opening default waveform audio device (WAVE_MAPPER)')

print "Default Wave Audio output device is opened successfully"


# 2. Write Audio Blocks to Device

# --- define necessary data structures
PVOID = wintypes.HANDLE
WAVERR_BASE = 32
WAVERR_STILLPLAYING = WAVERR_BASE + 1
class WAVEHDR(ctypes.Structure):
  _fields_ = [
    ('lpData', wintypes.LPSTR), # pointer to waveform buffer
    ('dwBufferLength', wintypes.DWORD),  # in bytes
    ('dwBytesRecorded', wintypes.DWORD), # when used in input
    ('dwUser', wintypes.DWORD),          # user data
    ('dwFlags', wintypes.DWORD),
    ('dwLoops', wintypes.DWORD),  # times to loop, for output buffers only
    ('lpNext', PVOID),            # reserved, struct wavehdr_tag *lpNext
    ('reserved', wintypes.DWORD)] # reserved
# The lpData, dwBufferLength, and dwFlags members must be set before calling
# the waveInPrepareHeader or waveOutPrepareHeader function. (For either
# function, the dwFlags member must be set to zero.)
# --- /define

class AudioWriter(object):
  def __init__(self, hwaveout):
    self.hwaveout = hwaveout
    self.wavehdr = WAVEHDR()

  def write(self, data):
    self.wavehdr.dwBufferLength = len(data)
    self.wavehdr.lpData = data
    
    # Prepare block for playback
    if ctypes.windll.winmm.waveOutPrepareHeader(
         self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr)
       ) != MMSYSERR_NOERROR:
      sys.exit('Error: waveOutPrepareHeader failed')

    # Write block, returns immediately unless a synchronous driver is
    # used (not often)
    if ctypes.windll.winmm.waveOutWrite(
         self.hwaveout, ctypes.byref(self.wavehdr), ctypes.sizeof(self.wavehdr)
       ) != MMSYSERR_NOERROR:
      sys.exit('Error: waveOutWrite failed')

    # [ ] calculate sleep delay based on sample length
    # iii [ ] Measure CPU usage spike during wait without delay
    import time
    time.sleep(1)

    # Wait until playback is finished
    while True:
      # unpreparing the header fails until the block is played
      ret = ctypes.windll.winmm.waveOutUnprepareHeader(
              self.hwaveout,
              ctypes.byref(self.wavehdr),
              ctypes.sizeof(self.wavehdr)
            )
      if ret == WAVERR_STILLPLAYING:
        import time
        time.sleep(1)
        continue
      if ret != MMSYSERR_NOERROR:
        sys.exit('Error: waveOutUnprepareHeader failed with code 0x%x' % ret)
      break


# [ ] it's no good to read all the PCM data into memory at once
data = open('95672__Corsica_S__frequency_change_approved.raw', 'rb').read()

aw = AudioWriter(hwaveout)
aw.write(data)


# x. Close Sound Device

ctypes.windll.winmm.waveOutClose(hwaveout)
print "Default Wave Audio output device is closed"

#-- /CHAPTER 1 --

Windows also provides DirectSound API, but it looks too complicated for me ATM. The code above is also available from https://bitbucket.org/techtonik/audiosocket and marked with 0.1 tag. You may expect to find further modifications there.