SCons build targets

SCons is awesome. Just saying. If you want to know (or troubleshoot) how SCons selects targets to be built, add this snippet at the end of your SConstruct:

def dump_targets(targets):
  for t in targets:
    if type(t) == str:
      name = t
    else:
      name = t.name
    print("  <" + str(t.__class__.__name__) + "> " + name)

print("[*] Default targets:")
dump_targets(DEFAULT_TARGETS)

print("[*] Command line targets:")
dump_targets(COMMAND_LINE_TARGETS)
print("[*] All build targets:")
dump_targets(BUILD_TARGETS)

For my copy of Wesnoth, 'scons .' produces this output:

[*] Default targets:
  <Alias> wesnoth
  <Alias> wesnothd
[*] Command line targets:
  <str> .
[*] All build targets:
  <str> .

And if you want to know how to specify targets or what do they mean, read the second page of SCons man documentation. Just for convenience I quote it here.

---

scons is normally executed in a top-level directory containing a SConstruct file, optionally specifying as command-line arguments the target file or files to be built.

By default, the command

scons

will build all target files in or below the current directory. Explicit default targets (to be built when no targets are specified on the command line) may be defined the SConscript file(s) using the Default() function, described below.

Even when Default() targets are specified in the SConscript file(s), all target files in or below the current directory may be built by explicitly specifying the current directory (.) as a command-line target:

scons .

Building all target files, including any files outside of the current directory, may be specified by supplying a command-line target of the root directory (on POSIX systems):

scons /

or the path name(s) of the volume(s) in which all the targets should be built (on Windows systems):

scons C:\ D:\

To build only specific targets, supply them as command-line arguments:

scons foo bar

in which case only the specified targets will be built (along with any derived files on which they depend).

Specifying "cleanup" targets in SConscript files is not usually necessary. The -c flag removes all files necessary to build the specified target:

scons -c .

to remove all target files, or:

scons -c build export

to remove target files under build and export. Additional files or directories to remove can be specified using the Clean() function. Conversely, targets that would normally be removed by the -c invocation can be prevented from being removed by using the NoClean() function.

A subset of a hierarchical tree may be built by remaining at the top-level directory (where the SConstruct file lives) and specifying the subdirectory as the target to be built:

scons src/subdir

or by changing directory and invoking scons with the -u option, which traverses up the directory hierarchy until it finds the SConstruct file, and then builds targets relatively to the current subdirectory:

cd src/subdir
scons -u .

Lifehack - import directly from command line

If you're obsessed with UX as I am, and long repetitive commands seem to break your flow, you may appreciate this little trick for invoking Python from command line.

I invoke python from console/terminal to check some behavior that I tend to forget after a day in JavaScript or other programming languages. It is often way faster than reading the docs. It also helps to quickly try new PyPI libraries after installing with pip (saves on mouse clicking in IDE etc.).

Today I found a time saver that makes me happy. This example is typed for Windows, but there should be no problem for any Linux hacker to port it to their favorites. So instead of doing this:

> python [ENTER]
>>> import os [ENTER]
>>> os.name [ENTER]
'nt'

I can just type:

> import os [ENTER]
>>> os.name [ENTER]
'nt'

It should be now easy to understand how it works, so I will complicate it a bit. The trick is this little import.bat script, placed in system PATH:

@echo off
py -2 -c "import sys; print(sys.version)"
py -2 -i -c "import %*"

py here is a py.exe launcher that can call either Python 2 or Python 3 depending on argument. It comes installed with Windows Python versions. I like to know which Python I work with, so I've added version into to the command, and to avoid polluting the global space of new interpreter, the version is printed by separate command.

Shipping Python tools in executable .zip packages

UP201501: Found out how to force setup.py create .zip instead of .tar.gz

This is about how to make Python source packages executable. As a side effect, this also explains, how to run invoke's tasks.py with a copy of invoke .zip shipped with your source code, but without installing it.

You know, Python can execute .zip files. As simply as:

$ wget https://pypi.python.org/packages/source/h/hexdump/hexdump-3.1.zip
$ python hexdump-3.1.zip
/usr/bin/python: can't find '__main__.py' in 'hexdump-3.1.zip'

Well, you need to place '__main__.py' into the root. It will then import what it needs and do something. The only problem with .zip source packages is that their structure (if created with standard Python tools) includes one additional directory at the top:

`-- hexdump-3.1
    |-- PKG-INFO
    |-- README.txt
    |-- hexdump.py
    |-- hexfile.bin
    `-- setup.py

So, even if you place `__main__.py` into the root, it won't be able to import file from `hexdump-3.1` subdirectory.. unless you prepare for it. Here is the solution:

import os
import sys

# add package .zip to python lookup path
__dir__ = os.path.dirname(__file__)
path = os.path.join(__dir__, 'hexdump-3.1')
sys.path.insert(0, path)

import hexdump
msg = hexdump.dehex("48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21")
print(msg)

Now if you run it, it will print the obvious.

$ python hexdump-3.1.zip
Hello, World!

In Python 3, the result will be wrapped into b'' string. But that's it!

The next hexdump version will likely to ship as executable .zip package

Where you may need it?

It may be useful for scripts automation if you're "vendorizing" dependencies (ship them with your code). It started with need to fix a broken link on Gratipay site. The Gratipay codebase is in public domain clear of any NDA, CLA and all other BS, and that makes it great for people to learn, reuse and enhance. I wish Python internal projects possessed these properties, but PSF politics is another topic.

Gratipay inside site uses make. It doesn't need powerful build tool, such as scons, so I tried to replace it with some simple task automation utility to be more cross-platform. I chose invoke, because of my previous experience with fabric for remote control, and because I saw previous attempts in another Gratipay repository. The necessary condition for new tool is that user experience should not degrade for make users, so main invoke's tasks.py script needed to be self-executable.

I completed the proof of concept by making invoke .zip package executable with the method above, placed it into vendor/ directory, and tuned tasks.py to reexecute itself with invoke .zip This also solved a potential problem with currently unstable invoke API.

tasks.py was modified as following:

import sys
sys.path.insert(0, 'vendor/invoke-0.9.0.zip')

from invoke import task, run

# ... tasks go here

if __name__ == '__main__':
  import sys
  from invoke import cli
  cli.main()

Bonus points

There is one more thing, which is better explained with hexdump example. hexdump package ships `hexfile.bin` data file used for testing. hexdump.py looks for it in its directory (__file__ dirname), and this lookup fails when hexdump.py is in archive. I'd say Python could provide some kind of an API to deal with "virtual import filesystem" (with watchers to track who and when modifies sys.path), but I digress. If you modify the `__main__.py` from the first chapter above to run tests - just call hexdump.runtest() - the execution will finally fail with the following error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "hexdump-3.1.zip/__main__.py", line 13, in
  File "hexdump-3.1.zip/hexdump-3.1/hexdump.py", line 311, in runtest
IOError: [Errno 20] Not a directory: '/root/hexdump-3.1.zip/hexdump-3.1/hexfile.bin'

(yes, I am hexdumping under root). The problem above was explained by Radomir, and I feel like the post is already tool long to write about it myself.

More bonus points

In theory it is possible to hack setup.py to produce executable source .zip package. By default, setup.py sdist command on Linux creates tar.gz files. So, first it needs to be forced to create .zip file. This can be done by overriding default --formats option:

# Override sdist to always produce .zip archive
from distutils.command.sdist import sdist as _sdist
class sdistzip(_sdist):
    def initialize_options(self):
        _sdist.initialize_options(self)
        self.formats = 'zip'

setup(
    ...
    cmdclass={'sdist': sdistzip},
)

I posted relevant documentation links to this StackOverflow question. The rest - how to inject __main__.py into the root of packed .zip - is yet to be covered, and I am not ready to research it just yet. Some hints may be provided by these answers.

Usability conclusion

I am actually quite happy with the ability to execute python .zip packages (which gave a lot of motivation to write this post), because previously I had to care about how to tell people that a tool is a single script, which can be downloaded from repository or from some dedicated download site (I mean Google Code, which does not support this anymore). I also don't trust my own server for downloads, because one day I found some "benzonasos" running there and I don't even have a slightest idea how it got there.

But now it becomes possible to just download and run the stuff from PyPI to test how it works without messing with creating and activating virtualenvs, and installation of package inside. Of course, if your .zip package has dependencies, they still need to be present on your system, but that's still a time saver. Oh, and that means that my tools can now consist of several modules inside.

Roundup Tracker: Create Issues by Email

There is one thing about bugs.python.org and other Roundup issue tracker instances that is not widely known. It is the fact that you can create new issues and update old ones directly from your email client, without visiting web interface at all.

As much as I hate Debian's email-only tracker, I must admit that having email control feature in addition to web interface can save some time, especially if you constantly forget passwords for different trackers like me.

So, to create new issue, just send email to the address that tracker uses to send mail to you. Well-known addresses of Python trackers:

• report@bugs.python.org - for filling bugs in Python with b.p.o tracker
• metatracker@psf.upfronthosting.co.za - for reporting problems with b.p.o tracker itself

Note that your email needs to be present in tracker database for it to accept your request, so you might need to create your account first.

You can also update existing issues by adding suffixes like [status=closed;resolution=invalid] to the subject field of your replies. I just closed issue19825 to test this method. You can try it too next time you feel uncomfortable about escaping from your mailbox.

This stuff is actually documented in official Roundup docs, but who reads the docs, anyway.

Command to generate SSH key on Linux

Because ssh-keygen has human unfriendly command line interface without --help option, here is a quick reminder how to generate very secure SSH key valid for one year (52 weeks) with "13.04" comment for its public part:

ssh-keygen -t rsa -b 4096 -V +52w -C 13.04

This key can be used to avoid typing password when pushing your Python code to sites like Bitbucket or GitHub (Google Code is not there yet) or to securely upload your packages to PyPI.

How to make RAM disk in Linux

UPDATE (2014-08-27): Exactly three years later I discovered that Linux already comes with RAM disk enabled by default, mounted as `/dev/shm` (which points to `/run/shm` on Debian/Ubuntu):

$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            75M  4.0K   75M   1% /run/shm

See detailed info here.

*RAM disk* is a term from the past when DOS was alive and information was stored on disks instead of internet. If you created image of some disk, it was possible to load it into memory. Memory disks were useful to load software from Live CDs. Usually software needs some space to write data during boot sequence, and RAM is the fastest way to setup one.

Filesystem space in memory can be extremely useful today too. For example, to run tests without reducing resource of SSD. While the idea is not new, there was no incentive to explore it until I've run upon tmpfs reference in Ubuntu Wiki.

For example, to get 2Gb of space for files in RAM, edit /etc/fstab to add the following line:

tmpfs     /var/ramspace       tmpfs     defaults,size=2048M     0     0

/var/ramspace is now the place to store your files in memory.

Python in Industial Automation

Before celebrating 10th Google birthday I would like to make a gesture of goodwill to fellow comrades from industrial automation field, who over years battled with a very special kind of the most conservative software in world.

Industrial Automation Software

It is the Software that directly puts people lives at stake. A little bug may kill people, inflict tremendous physical damage to buldings, vehicles and equipment. No wonder that Industrial Automation Software is among the most slowly developed technologies in the world. People fear to change things that work if the cost of error is so high. But using outdated software that nobody is capable to maintain carries even greater risks. Just as connected to Internet Windows without updates is brought down by viruses in minutes, the Enterprise is brought down by unmaintained legacy systems that just await the time to explode. Industrial software is ought to be stable, error free and resistant to all kind of threats, but it is not true in most cases. Most industrial software products lack public exposure, and so far I haven't seen any serious industrial product that didn't crash. Industrial software is the classic example of vendor lock-in. It happens just everythere. If you have some hardware sensors in the field and would like to collect signals from them - most likely you would have to buy industrial controller or PLC, which will be on 24/7/365/10 to gather your data. PLC's were mostly used solely for control automation without the need to share data with external systems, but in IT age you would need that data sooner or later. To get it from he PLC you would need to buy expensive OPC server that is capable to poll PLC using their own protocol and provide you with data in OPC format. OPC stood for OLE for Process Control and as you can guess from its name was vendor-locked to Microsoft from the start. It was based on OLE, COM and DCOM - its proprietary technology. To read one byte from your PLC you would have to buy Windows, buy OPC server and pay specialist to write an OPC client to access one byte of your sensor. Even after that some companies dared to include "Lower TCO" slogan in their markting campaigns. There was no alternative those days though. MS SQL is the only promoted "realtime database" even if there was a word "near" before the quotes, but it has support. Now there are plenty of open protocols and standards for data communication, many databases and time-tested solutions from other IT areas, but things are moving slowly in Industrial Automation world where "open protocol" still means that you are free to implement a specification, but need to pay for texts and it is ok. DCOM is slowly replaced by .NET, new consortiums appear just to put their label "approved" on community accepted standards. Industrial Automation is a business with enterprises that are primary sources of capital directly extracted from raw materials, so there is nothing exceptional in constant buzz that is generated by efforts of players to bite off a piece from this cake. But even more don't want to lose their share. The players are huge and so are the problems with Industrial Applications. It takes a lot of efforts to prove something different when it comes against mature and experienced players with a big sum of arguments in their bank accounts.

Industrial Automation applications are meant to be error free and stable, but there are still many errors, flakiness and unstability there. High cost of specialized instruments to debug and troubleshoot problems, low community response, closed protocols, formats and sale-oriented support that is not allowed to give any developers feedback - all these things are on the back side of the modern software development trends. Sooner or later the situation will change and industrial software will undergo the same change that affected internet and made it enjoy the fruits of open source developers collaboration. In the meanwhile let me show an example, how to "Lower your TCO" for the task of accessing data from Siemens S7 controllers.

Introducing Python for Automation World

You can control everything in Python. Starting from virtual game characters and application objects to real robots and trains. Just like you can control them in C. That means that Python is capable to directly use C libraries. Even if they are not really suitable to do such things it gives you the freedom of choice. Pulling slow snake's tail may be more efficient than guarding against fast mosCuitoes aiming at sucking you memory away. Python gives you a finer degree of control focusing your efforts on program logic while in C about 30% of code is dealing with memory allocation and type conversion. Python has more clear syntax than C and that is more important - the same syntax is used to control speedy C libraries.

Controller Data - Corner Stone of Automation

Data is everything. In Automation the primary source of data are sensors - any kind of device that is capable to measure something in common terms. Sensors are usually made to be simple to keep them free from errors, so they are often provide their data using electrical signals. These signals are usually transmitted in rough industrial environment where electrical disturbances are so common that direct connection of such sensors with computers (through COM port, for example) is a reason for frequent PC component replacements. That why specialized devices exist to poll sensor data. These are usually PLCs (programmable logic controllets) and distributed periphery devices that have a higher degree of protection against physical threats than common PC stations. Data inside PLC is the starting point to source of information for are used by industrial panels to visualize and control technical process parameters - link of PLC + visualization panel is commonly referenced as SCADA solution. Access to data hidden inside PLC from a panel is made using proprietary protocols even if the medium is Ethernet. Communication with upper levels require aforementioned Industrial Software provided for a price. But some protocols have alternative open source C libraries or drivers for PLC direct data access and I am going to show how to use such library with Python to run and stop PLC.

Parts of Assembly: libnodave, SWIG and Python

libnodave - communication library for exchanging data with Siemens PLCs released under LGPL license (you are welcomed to support this project if you find it useful). Python is a general-purpose scripting language. SWIG - tool to bind C and Python. Another required thing is C compiler. Because automation designers are mostly windows folks, we will use GCC from MinGW project Minimalistic user interface is done with wxWidgets library, which is also accessible from Python as wxPython library (created with SWIG, by the way) The total cost of all tools is zero. The performance of libnodave solution is still have to be measured. Siemens site contains performance data that can be useful for comparison.

libnodave allows to communicate with PLC with an API described on it's site. To start/stop controller you need to compile and execute the following C program:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include "nodavesimple.h"
#include "openSocket.h"

#ifdef LINUX
#include <unistd.h>
#include <sys/time.h>
#include <fcntl.h>
#define UNIX_STYLE
#endif

#ifdef BCCWIN
#include <time.h>
   void usage(void);
   void wait(void);
#define WIN_STYLE    
#endif

#include <errno.h>

int main(int argc, char **argv) {
   int useProtocol, useSlot;

   daveInterface * di;
   daveConnection * dc;
   _daveOSserialType fds;
  
   useProtocol=daveProtoISOTCP;
   useSlot=2;
  
   fds.rfd=openSocket(102, "192.168.1.244");
   fds.wfd=fds.rfd;
  
   if (fds.rfd>0) {
   di =daveNewInterface(fds,"IF1",0, useProtocol, daveSpeed187k);
   daveSetTimeout(di,5000000);
   dc =daveNewConnection(di,2,0,useSlot);
  
   if (0==daveConnectPLC(dc)) {
       daveStart(dc);
   return 0;
   } else {
       return -2;
   }
   } else {
       return -1;
   }   
}

What I am trying to achieve is the following equivalent in Python:

import nodave
plc = nodave.NoDaveTCPConnect('192.168.1.244')
plc.start()

To make this happen the libnodave library should be made available to Python as a module. Module is imported to make its functions available for Python scripters. To make module for libnodave we need to compile library, generate source for module and compile module itself.

[To be continued..]