SCons build targets

SCons is awesome. Just saying. If you want to know (or troubleshoot) how SCons selects targets to be built, add this snippet at the end of your SConstruct:

def dump_targets(targets):
  for t in targets:
    if type(t) == str:
      name = t
    else:
      name = t.name
    print("  <" + str(t.__class__.__name__) + "> " + name)

print("[*] Default targets:")
dump_targets(DEFAULT_TARGETS)

print("[*] Command line targets:")
dump_targets(COMMAND_LINE_TARGETS)
print("[*] All build targets:")
dump_targets(BUILD_TARGETS)

For my copy of Wesnoth, 'scons .' produces this output:

[*] Default targets:
  <Alias> wesnoth
  <Alias> wesnothd
[*] Command line targets:
  <str> .
[*] All build targets:
  <str> .

And if you want to know how to specify targets or what do they mean, read the second page of SCons man documentation. Just for convenience I quote it here.

---

scons is normally executed in a top-level directory containing a SConstruct file, optionally specifying as command-line arguments the target file or files to be built.

By default, the command

scons

will build all target files in or below the current directory. Explicit default targets (to be built when no targets are specified on the command line) may be defined the SConscript file(s) using the Default() function, described below.

Even when Default() targets are specified in the SConscript file(s), all target files in or below the current directory may be built by explicitly specifying the current directory (.) as a command-line target:

scons .

Building all target files, including any files outside of the current directory, may be specified by supplying a command-line target of the root directory (on POSIX systems):

scons /

or the path name(s) of the volume(s) in which all the targets should be built (on Windows systems):

scons C:\ D:\

To build only specific targets, supply them as command-line arguments:

scons foo bar

in which case only the specified targets will be built (along with any derived files on which they depend).

Specifying "cleanup" targets in SConscript files is not usually necessary. The -c flag removes all files necessary to build the specified target:

scons -c .

to remove all target files, or:

scons -c build export

to remove target files under build and export. Additional files or directories to remove can be specified using the Clean() function. Conversely, targets that would normally be removed by the -c invocation can be prevented from being removed by using the NoClean() function.

A subset of a hierarchical tree may be built by remaining at the top-level directory (where the SConstruct file lives) and specifying the subdirectory as the target to be built:

scons src/subdir

or by changing directory and invoking scons with the -u option, which traverses up the directory hierarchy until it finds the SConstruct file, and then builds targets relatively to the current subdirectory:

cd src/subdir
scons -u .

Shipping Python tools in executable .zip packages

UP201501: Found out how to force setup.py create .zip instead of .tar.gz

This is about how to make Python source packages executable. As a side effect, this also explains, how to run invoke's tasks.py with a copy of invoke .zip shipped with your source code, but without installing it.

You know, Python can execute .zip files. As simply as:

$ wget https://pypi.python.org/packages/source/h/hexdump/hexdump-3.1.zip
$ python hexdump-3.1.zip
/usr/bin/python: can't find '__main__.py' in 'hexdump-3.1.zip'

Well, you need to place '__main__.py' into the root. It will then import what it needs and do something. The only problem with .zip source packages is that their structure (if created with standard Python tools) includes one additional directory at the top:

`-- hexdump-3.1
    |-- PKG-INFO
    |-- README.txt
    |-- hexdump.py
    |-- hexfile.bin
    `-- setup.py

So, even if you place `__main__.py` into the root, it won't be able to import file from `hexdump-3.1` subdirectory.. unless you prepare for it. Here is the solution:

import os
import sys

# add package .zip to python lookup path
__dir__ = os.path.dirname(__file__)
path = os.path.join(__dir__, 'hexdump-3.1')
sys.path.insert(0, path)

import hexdump
msg = hexdump.dehex("48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21")
print(msg)

Now if you run it, it will print the obvious.

$ python hexdump-3.1.zip
Hello, World!

In Python 3, the result will be wrapped into b'' string. But that's it!

The next hexdump version will likely to ship as executable .zip package

Where you may need it?

It may be useful for scripts automation if you're "vendorizing" dependencies (ship them with your code). It started with need to fix a broken link on Gratipay site. The Gratipay codebase is in public domain clear of any NDA, CLA and all other BS, and that makes it great for people to learn, reuse and enhance. I wish Python internal projects possessed these properties, but PSF politics is another topic.

Gratipay inside site uses make. It doesn't need powerful build tool, such as scons, so I tried to replace it with some simple task automation utility to be more cross-platform. I chose invoke, because of my previous experience with fabric for remote control, and because I saw previous attempts in another Gratipay repository. The necessary condition for new tool is that user experience should not degrade for make users, so main invoke's tasks.py script needed to be self-executable.

I completed the proof of concept by making invoke .zip package executable with the method above, placed it into vendor/ directory, and tuned tasks.py to reexecute itself with invoke .zip This also solved a potential problem with currently unstable invoke API.

tasks.py was modified as following:

import sys
sys.path.insert(0, 'vendor/invoke-0.9.0.zip')

from invoke import task, run

# ... tasks go here

if __name__ == '__main__':
  import sys
  from invoke import cli
  cli.main()

Bonus points

There is one more thing, which is better explained with hexdump example. hexdump package ships `hexfile.bin` data file used for testing. hexdump.py looks for it in its directory (__file__ dirname), and this lookup fails when hexdump.py is in archive. I'd say Python could provide some kind of an API to deal with "virtual import filesystem" (with watchers to track who and when modifies sys.path), but I digress. If you modify the `__main__.py` from the first chapter above to run tests - just call hexdump.runtest() - the execution will finally fail with the following error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "hexdump-3.1.zip/__main__.py", line 13, in
  File "hexdump-3.1.zip/hexdump-3.1/hexdump.py", line 311, in runtest
IOError: [Errno 20] Not a directory: '/root/hexdump-3.1.zip/hexdump-3.1/hexfile.bin'

(yes, I am hexdumping under root). The problem above was explained by Radomir, and I feel like the post is already tool long to write about it myself.

More bonus points

In theory it is possible to hack setup.py to produce executable source .zip package. By default, setup.py sdist command on Linux creates tar.gz files. So, first it needs to be forced to create .zip file. This can be done by overriding default --formats option:

# Override sdist to always produce .zip archive
from distutils.command.sdist import sdist as _sdist
class sdistzip(_sdist):
    def initialize_options(self):
        _sdist.initialize_options(self)
        self.formats = 'zip'

setup(
    ...
    cmdclass={'sdist': sdistzip},
)

I posted relevant documentation links to this StackOverflow question. The rest - how to inject __main__.py into the root of packed .zip - is yet to be covered, and I am not ready to research it just yet. Some hints may be provided by these answers.

Usability conclusion

I am actually quite happy with the ability to execute python .zip packages (which gave a lot of motivation to write this post), because previously I had to care about how to tell people that a tool is a single script, which can be downloaded from repository or from some dedicated download site (I mean Google Code, which does not support this anymore). I also don't trust my own server for downloads, because one day I found some "benzonasos" running there and I don't even have a slightest idea how it got there.

But now it becomes possible to just download and run the stuff from PyPI to test how it works without messing with creating and activating virtualenvs, and installation of package inside. Of course, if your .zip package has dependencies, they still need to be present on your system, but that's still a time saver. Oh, and that means that my tools can now consist of several modules inside.

Writing SCons Plugin: Discovery

UP201410: Add version info and link to Parts concept.

SCons is a build tool written completely in Python, so that you can just put it into your repository, extend for your own purposes and run directly from checkout without installing anything. SCons documentation doesn't use word plugin to refer to extensions. It calls them Tools. If you're not familiar with SCons concepts, check out this wiki page.

Plugin Discovery

SCons tries its best to avoid magical behavior. That's why it ignores paths and options set in system environment and requires that you specify everything explicitly. The same is true for plugins. Example from SCons man page:

env = Environment(tools = ['default', 'foo'], toolpath = ['tooldir'])

This creates build environment, initializes default tools, sets lookup path for plugins to tooldir/ and enables tool named foo, which is located in tooldir/foo.py. foo.py should have two functions - generate(env) and exists(env).

To test that your tool is found correctly, check env['TOOLS']:

print(env['TOOLS'])

If filename is not there - SCons was unable to find it. If it was a problem with contents of foo.py - SCons would fail with exception.

SCons has an automatic tool discovery mechanism - if you don't like to specify toolpath directly - you may place your tool in one of several locations that SCons scans before executing SConstruct. See description of the --site-dir=dir option for details.

Version compatibility and future

Described behavior is true for SCons 2.3.4 and earlier versions. It may be not very intuitive, so things may change in future, especially if we find resources to integrate Parts.

Using getopt with optparse (or how to move from getopt gradually)

TL;DR: https://bitbucket.org/techtonik/scons/commits/bcb60b

SCons has a very old and interesting codebase with a lots of outdated and unusual stuff that makes it more difficult to extend. One such thing is getopt library, which is a predessor for Optik library (written by Greg Ward) now better known as optparse.

So I wanted to replace getopt with optparse, but didn't want to change everything in one step, because I didn't have time to check every option. Instead I decided to parse options I needed with optparse and leave everything else to the old getopt engine.

getopt only needs a list of arguments to work. sys.argv[1:] to be exact. This is also the second half of result returned by OptionParser.parse_args() function. The only problem was to teach OptionParser to ignore unknown options and leave them in arguments. Strange thing, but Optik examples included this user story, completely ignored in optparse documentation. To make this long user story short, you need to subclass OptionParser to use getopt with optparse:

# "Pass-through" option parsing -- an OptionParser that ignores
# unknown options and lets them pile up in the leftover argument
# list.  Useful to gradually port getopt to optparse.

from optparse import OptionParser, BadOptionError

class PassThroughOptionParser(OptionParser):
    def _process_long_opt(self, rargs, values):
        try:
            OptionParser._process_long_opt(self, rargs, values)
        except BadOptionError, err:
            self.largs.append(err.opt_str)
    def _process_short_opts(self, rargs, values):
        try:
            OptionParser._process_short_opts(self, rargs, values)
        except BadOptionError, err:
            self.largs.append(err.opt_str)

parser = PassThroughOptionParser(add_help_option=False)
parser.add_option('-a', '--all', action='store_true',
                      help="Run all tests.")
(options, args) = parser.parse_args()

#print "options:", options
#print "args:", args

Now pass args down to the getopt call and you're all set.

P.S. In argparse you can use ArgumentParser.parse_known_args() function.

Update 2013-02: For humane option parsing you should definitely see docopt library.

About Environment

In Python applications environment is often an ambiguous term that needs clarification. In general sense `the environment` is system environment with PATH and friends accessible from os.environ within Python. But in Python applications it can mean different things.

In Trac `the environment` is a directory with settings, database and other files related to one Trac site.

In SCons `the environment` is a structure in memory that holds dependency trees, helper functions, builders and other stuff. It is written to disk only for caching.


Quite many other applications have some kind of environment for their own purpose with meanings close to either Trac or SCons, which often confuses newbies or strangers who are not aware about the context. Software development clearly needs more specific terms in English to make people write and read in the same language without those excessive contexts.

Finding unused files in a SCons project with Process Monitor

New technologies are born and die, but one things remains in your project - their files. Quite often you have no idea about where are they used, and attempt to remove them may lead to serious consequences.

Fortunately, if your project is managed by fine grained build system such as SCons, if your build scripts are not globbing too much, there are chances you can find files that are not participating in the builds.

Here is how to do this on Windows using Process Monitor tool that intercepts all system calls including file access.

While build systems are usually common for C/C++ and Java projects, it is possible to add fine-grained file usage control for any project. For example, SCons itself is written entirely in Python, it could run directly from the source checkout or build distributives from checkout. But instead, it uses build procedure to copy all necessary files from checkout into separate directory and do stuff from there.

Thanks to that it is possible to see which files are no more actual. While it is possible to compare checkout source tree and copied directory trees, I'll go through the hells of monitoring system file access in a source tree during the build process using Process Monitor (FileMon in the past). Linux should have similar tools too - let me know how are they called.

The process is the following:

1. Start Process Monitor
2. Stop incoming event flood by (un)clicking Capture (Ctrl-E) button
3. Open Filter (Ctrl-L) dialog to add some filters


SCons build is started by bootstrap.py script from a root of SCons source checkout. The script is executed by python executable, so I add python.exe process name to the filter. I know that bootstrap.py copies files from src/ subdirectory, so it is the directory I need to monitor, so I add this dir to filters too.

4. Go Tools -> File Summary...


There is a list of paths catched by Process Monitor when listening to system calls. They are already filtered, but additional filters can be applied using bottom left button to make information even more useful.

5. Export to CSV using Save...

Exported CSV is not very useful without some postprocessing. I used the following a script to compare the list of paths in CSV to actual src/ directory contents. This gives me names of files that were not touched during build at all.

SRCDIR = "C:\\p\\python\\scons\\src"
CSVLIST = 'accessed_bootstrap_files.CSV'

import csv
import os

reader = csv.reader(open(CSVLIST))
header = reader.next()
pathidx = header.index("Path")
pathset = set([row[pathidx] for row in reader])

#for row in pathset:
#  print row

fileset = set()
for root, dirs, files in os.walk(SRCDIR):
  fileset.update( [os.path.join(root, f) for f in files] )
  if '.svn' in dirs:
    dirs.remove('.svn')  # don't visit .svn directories

if len(pathset & fileset) == 0:
  print 'Error: File sets do not intersect at all'

print "Files not found in source directory tree:"
for f in (pathset - fileset):
  if not os.path.isdir(f):
    print f

print
print "Untouched files in source directory tree:"
for f in sorted(fileset - pathset):
  if not os.path.isdir(f):
    print f

I've found a few interesting things about SCons. Core tests are mixed with source files in repository checkout. They are not copied during bootstrap build. There are also few setup.py files, post-install script and announcement that don't participate in the build.

Here is the output of the above script:

Files not found in source directory tree:
<Total>

Untouched files in source directory tree:
C:\p\python\scons\src\.aeignore
C:\p\python\scons\src\Announce.txt
C:\p\python\scons\src\engine\.aeignore
C:\p\python\scons\src\engine\SCons\.aeignore
C:\p\python\scons\src\engine\SCons\ActionTests.py
C:\p\python\scons\src\engine\SCons\BuilderTests.py
C:\p\python\scons\src\engine\SCons\CacheDirTests.py
C:\p\python\scons\src\engine\SCons\DefaultsTests.py
C:\p\python\scons\src\engine\SCons\EnvironmentTests.py
C:\p\python\scons\src\engine\SCons\ErrorsTests.py
C:\p\python\scons\src\engine\SCons\ExecutorTests.py
C:\p\python\scons\src\engine\SCons\JobTests.py
C:\p\python\scons\src\engine\SCons\MemoizeTests.py
C:\p\python\scons\src\engine\SCons\Node\.aeignore
C:\p\python\scons\src\engine\SCons\Node\AliasTests.py
C:\p\python\scons\src\engine\SCons\Node\FSTests.py
C:\p\python\scons\src\engine\SCons\Node\NodeTests.py
C:\p\python\scons\src\engine\SCons\Node\PythonTests.py
C:\p\python\scons\src\engine\SCons\Optik\.aeignore
C:\p\python\scons\src\engine\SCons\PathListTests.py
C:\p\python\scons\src\engine\SCons\Platform\.aeignore
C:\p\python\scons\src\engine\SCons\Platform\PlatformTests.py
C:\p\python\scons\src\engine\SCons\SConfTests.py
C:\p\python\scons\src\engine\SCons\SConsignTests.py
C:\p\python\scons\src\engine\SCons\Scanner\.aeignore
C:\p\python\scons\src\engine\SCons\Scanner\CTests.py
C:\p\python\scons\src\engine\SCons\Scanner\DirTests.py
C:\p\python\scons\src\engine\SCons\Scanner\FortranTests.py
C:\p\python\scons\src\engine\SCons\Scanner\IDLTests.py
C:\p\python\scons\src\engine\SCons\Scanner\LaTeXTests.py
C:\p\python\scons\src\engine\SCons\Scanner\ProgTests.py
C:\p\python\scons\src\engine\SCons\Scanner\RCTests.py
C:\p\python\scons\src\engine\SCons\Scanner\ScannerTests.py
C:\p\python\scons\src\engine\SCons\Script\.aeignore
C:\p\python\scons\src\engine\SCons\Script\MainTests.py
C:\p\python\scons\src\engine\SCons\Script\SConscriptTests.py
C:\p\python\scons\src\engine\SCons\SubstTests.py
C:\p\python\scons\src\engine\SCons\TaskmasterTests.py
C:\p\python\scons\src\engine\SCons\Tool\.aeignore
C:\p\python\scons\src\engine\SCons\Tool\JavaCommonTests.py
C:\p\python\scons\src\engine\SCons\Tool\PharLapCommonTests.py
C:\p\python\scons\src\engine\SCons\Tool\ToolTests.py
C:\p\python\scons\src\engine\SCons\Tool\f03.xml
C:\p\python\scons\src\engine\SCons\Tool\msvsTests.py
C:\p\python\scons\src\engine\SCons\UtilTests.py
C:\p\python\scons\src\engine\SCons\Variables\BoolVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\EnumVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\ListVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\PackageVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\PathVariableTests.py
C:\p\python\scons\src\engine\SCons\Variables\VariablesTests.py
C:\p\python\scons\src\engine\SCons\WarningsTests.py
C:\p\python\scons\src\engine\SCons\cppTests.py
C:\p\python\scons\src\engine\setup.cfg
C:\p\python\scons\src\engine\setup.py
C:\p\python\scons\src\script\.aeignore
C:\p\python\scons\src\script\scons-post-install.py
C:\p\python\scons\src\script\setup.cfg
C:\p\python\scons\src\script\setup.py
C:\p\python\scons\src\test_aegistests.py
C:\p\python\scons\src\test_files.py
C:\p\python\scons\src\test_interrupts.py
C:\p\python\scons\src\test_pychecker.py
C:\p\python\scons\src\test_setup.py
C:\p\python\scons\src\test_strings.py


Hope this helps clean up your projects too.

P.S. I wish there was a Python script replacement for Process Monitor, or at least that it could be controlled from command line.