Writing SCons Plugin: Discovery

UP201410: Add version info and link to Parts concept.

SCons is a build tool written completely in Python, so that you can just put it into your repository, extend for your own purposes and run directly from checkout without installing anything. SCons documentation doesn't use word plugin to refer to extensions. It calls them Tools. If you're not familiar with SCons concepts, check out this wiki page.

Plugin Discovery

SCons tries its best to avoid magical behavior. That's why it ignores paths and options set in system environment and requires that you specify everything explicitly. The same is true for plugins. Example from SCons man page:

env = Environment(tools = ['default', 'foo'], toolpath = ['tooldir'])

This creates build environment, initializes default tools, sets lookup path for plugins to tooldir/ and enables tool named foo, which is located in tooldir/foo.py. foo.py should have two functions - generate(env) and exists(env).

To test that your tool is found correctly, check env['TOOLS']:

print(env['TOOLS'])

If filename is not there - SCons was unable to find it. If it was a problem with contents of foo.py - SCons would fail with exception.

SCons has an automatic tool discovery mechanism - if you don't like to specify toolpath directly - you may place your tool in one of several locations that SCons scans before executing SConstruct. See description of the --site-dir=dir option for details.

Version compatibility and future

Described behavior is true for SCons 2.3.4 and earlier versions. It may be not very intuitive, so things may change in future, especially if we find resources to integrate Parts.

Python C API/ABI compatibility report

UPDATE: There is now an official thread on python-dev.

Upstream Tracker is an open source (GPL) tool that allows to track API/ABI changes between releases of C/C++ libraries. I asked Andrey Ponomarenko, who is the main maintainer of the project, to add Python to the list and here is the result:

http://upstream-tracker.org/versions/python.html

Hope you like it.

ANN: xtrace 0.5 - indented function trace in Xdebug format

Xdebug is a PHP tool that allows to trace how PHP code is executed. Today I release a tool called xtrace into public (and more specifically into public domain), which allows to get the same (or at least very identical) output, but for Python.

Few years ago inability to trace function calls in Python came as a showstopper. I decided to write a familiar tool for Python. More than that - I wanted to integrate it into Spyder. But a year ago xtrace itself faced with showstopper. The showstopper was the behavior of execfile function, that I could not get right at that time, because of the docs, of my expectations and poor knowledge of English. Maybe there is a flaw in my cognitive abilities, but I tried to get at this problem several times and failed. Until recently some hackers from ZenSecurity team brought a concept known an pyjail to my attention. The challenge to prove that pyjail concept is impossible allowed me to concentrate on gory details of execfile works and knowing that documentation is totally confusing for my, I found the time to set my own experiments. You can read them at the link I've given above as well as some analysis why documentation that actually includes all the details can be bad and confusing.

The xtrace was basically broken for three years, starting from the version 0.2 - the day I put execfile() call from root to the xtrace module to the main() function. This placement changed the execfile() behavior, and while trying to debug that I also run into confusing dynamic behavior of dictionary returned by locals(). Opened can of worms made those parasites to completely consume my brain, causing much anger and frustration to be spilled around execfile() and locals() concepts over into Python lists. It is kind of relief now that I can name all the problems, analyse them and look back as enlightened. Being jobless I had a plenty of time to investigate, but I really don't want anyone to enter that state of confusing and helplessness that I had a year ago.

Hopefully, my experience with xtrace will clear the confusion for those who will try to use exec type abilities of Python for developing their own tools. Maybe it will result in a better Python API in the future, with better documentation and position-independent behavior.

I am interested to know the feedback that you can leave in xtrace tracker, such as if the output really matches PHP behavior, if it is accepted by PHP tools and how it behaves in different scopes of Python. It is interesting to convert it to Spyder plugin and see the usage in other tools, but I realize that I may not have time for that. The next focus for me is to add an easy API to xtrace to enable people to write they own tracers more easily. Focus on UX and everything else will come.

ANN: hexdump 2.0 - view/edit your binary with hex tool

https://pypi.python.org/pypi/hexdump

Finally some prod that can be named feature-complete for release. It is cross-platform, meaning it should run the same on Windows (tested), Linux, and OS X. It is Python 2 and Python 3 compatible. And it released into public domain, so that you won't have any problems in reusing it for your commercial and non-commercial hacking.

For those who are unaware of what hexdump is, hexdump is a representation of any binary data in human readable form. This form is good for hacking, inspecting and debugging binary data and protocols, but it is also good for editing such data. I am not pasting the output if the tool to encourage you to play with it yourself.

It can be used as command line tool and as a library. The most simple way is to use it as a tool:

# install
$ python -m pip install hexdump

# dump
$ python -m hexdump binary.bin > dump.txt
...

# restore
$ python -m hexdump --restore dump.txt
...

P.S. I don't mind including `hexdump` as provisional package in Python standard library if anyone will be able to convince PSF to accept public domain, CC0 or MIT licensed code.

Draw a pixel with PySDL2

This is a minimal code to output pixel on the screen using PySDL2.

UPD: (March 2014) Up for PySDL2 0.9.0 (RenderContext renamed to Renderer)

#!/usr/bin/env python
"""
The code is placed into public domain
by anatoly techtonik <techtonik@gmail.com>
"""
import sdl2
import sdl2.ext as lib

lib.init()

window = lib.Window('', size=(300, 100))
window.show()

renderer = lib.Renderer(window)
renderer.draw_point([10,10], lib.Color(255,255,255))
renderer.present()

running = True
while running:
  for e in lib.get_events():
    if e.type == sdl2.SDL_QUIT:
      running = False
      break
    if e.type == sdl2.SDL_KEYDOWN:
      if e.key.keysym.sym == sdl2.SDLK_ESCAPE:
        running = False
        break

Open Source / Free Standards vs ISO/IEC

Intro

While trying to use Galaxy Note 10.1 as a tablet and remote control device for my Windows and Linux stations, I discovered awesome MIT licensed GfxTablet project (draw on your PC via your Android device):

GfxTablet shall make it possible to use your Android device (especially tablets) like a graphics tablet.

It consists of two components:

• the GfxTablet Android app
• the input driver for your PC

The GfxTablet app sends motion and touch events via UDP to a specified host on port 40118.

It was so awesome in its simplicity and protocol that I couldn't resist to build a Python client for it. It didn't take long (well, a day maybe), before I noticed two errors in its protocol. First is that byte order for fields with 2 bytes length was not described and appeared to be big endian (while I assumed the opposite). Second is that one of the fields described as 2 bytes ushort was actually 1 byte size octet. After reading the source, I found the mistakes and edited the protocol description from the web to fix them, which resulted in this (already merged, yay!) pull request.

Standard on Byte Size

I've got an interesting comment on my pull request:

"octet" is more accurate than "byte" because there are systems and programming languages which have a byte size that is not 8 bits

My natural reaction was "No way! That's can't be true.", but Wikipedia said I am wrong. Luckily, it also said that:

The de facto standard of eight bits is a convenient power of two permitting the values 0 through 255 for one byte. The international standard IEC 80000-13 codified this common meaning.

This was the first time I thought that ISO did something right, so I decided to take a look myself at this standard. I found two copies - IEC and ISO. ISO site has a better SEO department, so they've got a better Google position for their shop. Yes, the ISO and IEC are shops - the price to get official size of byte is ISO CHF 154,00 or IEC CHF 150. They really like to have accounts in Swiss banks for some reasons. I'd advice to sell those standard in Bitcoins instead - it is more profitable in a long term.

ISO/IEC as Commercial De-Facto Authorities

Why do we need some organizations like ISO/IEC that place their name and put limitations around access to de-facto standards that more like any other information want to be free? I'd say that our awesome decentralized and independent approach to develop what do you feel and support what do you want is just not widely exposed to those conservative oldschool bureaucrats, who still live in their own world of central authorities that should dictate people what to do.

Don't get me wrong. There are conflicting points when you DO need to set a standard, and an enforcing organization like ISO/IEC is required (enforcing, because market force business to comply no matter how "recommended" the standard is). The costs of dealing with conflicting parties and convincing them is high, and that's why they set price on papers (the calculation if the price is fair is thankfully out of scope for this post). But the thing that bothers me more is that they set price on assessing the facts that are de-facto standards and common knowledge.

The problem here that we separately don't have a tool to say something as whole. The problem here is that if anybody will try to speak for the whole net, the net would resist and that's natural. Because people tend to say too much in one phrase and they are too smart. The reason is to keep the facts short, clear voices from responsibility and make it all countable and strictly out of politics.

What can be fixed here?

Usually people sign petitions. I propose to extend this just for fun. Make a technical statement that countries should agree on (no politics, please), give people an opportunity to support these openly, say if they don't want to support openly and give ability to support in closed manner (?respect privacy), the same way for disapprovals. Disapprovals may carry a reason. Once this data is in place, let people upvote and see what will happen.

The statement - "the byte is 8 bits".

Then build a list of countries that nationally accepted this statement. Then name this initiative somehow - it is important to keep this strictly technical to shot the zombies.

Once a statement reaches some degree of exposure and human votes per country, country can decide to accept it by placing official signed statement online. Over the time, the statements can be combined into free will packs and signed too. This will allow to sync.

Arguments. I am not sure they are needed for de-facto standards, because you reach consensus not by persuasion, but by collecting overall feeling. However, if there are problems with de-facto ways, and people feel there is something wrong, there should have an ability to "opt-in for a change" to upvote/downvote such arguments too. People should not be ashamed to set a value of "my butthurt" meter when voting or proposing counter-arguments, because we are irrational by our design, and technical problems with standards need more feedback than any other area of development on the butthurt effect.

So, simple statements, public voting, open process, feddback, realtime status and summary on nation adoptions.

Who should do this?

I'd be interested in working on this if I had some place to live in of my own. I'd start from contacting guys from Stack Overflow to reuse open source parts of their experience. Quite boring, right? Well, I am not saying that I want and plan to work on this alone. I am just saying that I am not in a position to take a role of coordinator. I just want to says that if you like the idea, maybe even in some crippled variant, found the resources to go, and want to try, feel free to ping me.

One of the tools that was really close and impressed was (now defunct) http://hammerprinciple.com/ which helped me to discover bad things about my favorite version control and programming language without too much butthurt injury. Hopefully, it will strike back again.

Roundup Tracker: Create Issues by Email

There is one thing about bugs.python.org and other Roundup issue tracker instances that is not widely known. It is the fact that you can create new issues and update old ones directly from your email client, without visiting web interface at all.

As much as I hate Debian's email-only tracker, I must admit that having email control feature in addition to web interface can save some time, especially if you constantly forget passwords for different trackers like me.

So, to create new issue, just send email to the address that tracker uses to send mail to you. Well-known addresses of Python trackers:

• report@bugs.python.org - for filling bugs in Python with b.p.o tracker
• metatracker@psf.upfronthosting.co.za - for reporting problems with b.p.o tracker itself

Note that your email needs to be present in tracker database for it to accept your request, so you might need to create your account first.

You can also update existing issues by adding suffixes like [status=closed;resolution=invalid] to the subject field of your replies. I just closed issue19825 to test this method. You can try it too next time you feel uncomfortable about escaping from your mailbox.

This stuff is actually documented in official Roundup docs, but who reads the docs, anyway.