Saturday, September 17, 2016

Python Usability Bugs: subprocess.Popen executable

subprocess.Popen seems to be designed as a "swiss army knife" of managing external processes, and while the task is pretty hard to solve in cross-platform way, it seems the people who have contributed to it did manage to achieve that. But it still came with some drawbacks and complications. Let's study one of these that I think is a top one from usability point of view, because it confuses people a lot.

I've got a simple program that prints its name and own arguments (forgive me for Windows code, as I was debugging the issue on Windows, but this works the same on Linux too). The program is written in Go to get single executable, because subprocess has special handling for child Python processes (another usability bug for another time).
>argi.exe 1 2 3 4
prog: E:\argi.exe
args: [1 2 3 4]
Let's execute it with subprocess.Popen, and for that I almost always look up the official documentation for Popen prototype:
subprocess.Popen(argsbufsize=0executable=Nonestdin=None, stdout=Nonestderr=Nonepreexec_fn=Noneclose_fds=False, shell=Falsecwd=Noneenv=Noneuniversal_newlines=False, startupinfo=Nonecreationflags=0)
Quite scary, right? But let's skip confusing part and quickly figure out something out of it (because time is scarce). Looks like this should do the trick:
import subprocess

args = "1 2 3 4".split()
p = subprocess.Popen(args, executable="argi.exe")
After saving this code to "" and running it, you'd probably expect something like this :
> python
prog: E:\argi.exe
args: [1 2 3 4]
And... you won't get this. What you get is this:
> python
prog: 1
args: [2 3 4]
And that's kind of crazy - not only the executable was renamed, but the first argument was lost, and it appears that this is actually a documented behavior. So let's define Python Usability Bug as something that is documented but not expected (by most folks who is going to read the code). The trick to get code do what is expected is never use executable argument to subprocess.Popen:
import subprocess

args = "1 2 3 4".split()
args.insert(0, "argi.exe")
p = subprocess.Popen(args)
prog: argi.exe
args: [1 2 3 4]
The explanation for former "misbehavior" is that executable is a hack that allows to rename program when running subprocess. It should be named substitute, or - even better - altname to work as an alternative name to pass to child process (instead of providing alternative executable for the former name). To make subprocess.Popen even more intuitive, the args argument should have been named command.

From the high level design point of view, the drawbacks of this function is that it *does way too much*, its arguments are not always intuitive - it takes *a lot of time to grok official docs*, and I need to read it *every time*, because there are too many little important details of Popen behavior (have anybody tried to create its state machine?), so over the last 5 years I still discover various problems with it. Today I just wanted to save you some hours that I've wasted myself while debugging pymake on Windows.

That's it for now. Bonus points to update this post with link when I get more time / mana for it:

  • [ ] people who have contributed to it
  • [ ] it came with drawbacks
  • [ ] have anybody tried to create its state machine?
  • [ ] subprocess has special handling for child Python processes

Thursday, June 16, 2016

Why Python community should take UX seriously

UX is a discipline that is dedicated to measuring user emotions and satisfaction while using a product in order to improve them. When you grow experience in any programming language you start to experience negative emotions towards small warts and glitches that constantly get in your way. You compare it to different languages and soon you're no more satisfied with it. Backward compatibility etc. doesn't allow to change things, but they should be changed, and UX is the criteria and method to answer how. But why?

People are dying too quickly. The process of learning new technology is fun and exciting, but when you look at the ideas in the past you'll see that we are not inventing anything new. People thought about many ideas, and couldn't implement them. I still don't have automation line that pours me coffee in the morning and bakes omelet and this idea is like 20 years old. I have no information about where my products are coming from - this idea is about 10 years old, and those fruits that are coming from hydroponics and pretty tasteless, so I'd prefer to buy something bathed in organic sun. Where are those inventions now? I am pretty sure somebody is trying to design these things in own basement, but those inventions will never see the light, because they are fragile, needs a lot of experience and education. Why learning is hard? It needs a lot of experience and education to learn concepts, operating systems internals and get used to various glitches and workarounds from the past. If Python was easier, if operating systems were more easy to learn and intuitively understand, not relying on elitist part of hacker culture, we could hack on more fun and reliable things today.

Tuesday, June 07, 2016

Why Roundup needs a router? is powered by ancient software. Most people nowadays won't install it. They would rather find something newer, with more features (consumers) or invent something from scratch that will be more modern from the start (makers).

How about trying a different approach - how about an approach that means constant evolution of the software? The reason why people reinvent things from scratch is because they often can not understand the complexity of what was already written. The burden of complexity is like tar that glues us to the surface, needs more force to take a step and slows us down. One way to get rid of it is to start again clean. But there is no guarantee that the same story won't happen if you couldn't see past complexity before.

A way of evolution is to face the complexity and deal with it. The way to deal with complexity tar is to learn to fly over it. Visualization, art, story-telling are helping to people to overcome limitations of our attention, interest, motivation and focus. There are tools to make complex things simple, there are things to look at the surface from above and choose a path that is still unexplored.

Router is a component that directs URL request to processing. Router says that http://../file142 should be processed by function that handles files, and http://../issue42 should be handled by function that processes issues. This logic is hardcoded into Roundup, so its URL structure doesn't change. If Roundup had this router, then it could be used as a replacement for existing trackers without breaking links. Router also allows Roundup extensions add new endpoints, such as ones required for OAuth2 processing or REST and ajax interface.

So, how to evolve software? It takes media and coordination skills first, and code review and design second. Additional code that makes code more complex should be backed up by documentation that explains it, every lengthy documentation that gives details should be accompanied by concise overview and reminder picture, every yak shaving process should be automated, every upstream tools need to be improved. Evolution is a global process, and it is hard (if not impossible) to evolve one agent if the whole ecosystem is not changing. Changing ecosystem or tools that are already stable like Python require process shift that drastically changes how people coordinate, approve, mutate and select the branches with features that comfort them.

Wednesday, May 11, 2016

Open Source, Death, and Open Economy

Open Source is a great idea, great movement and a great boost to the progress of civilization. Open Data made transparent governments possible, which allowed people to regulate tough problems, not hide them. Open systems like Linux and Open Stack enabled building more useful software on top of more solid and stable basis (first for single operating system nodes and now for clouds and networks).

So why Open Source is dead? Because there is no such thing as software product - any software is alive as long as its code is live in the heads of its maintainers. As soon as maintainers leave, the product enters death spiral - new people enter the field to patch the product to the death. Open Source is dead, because maintaining Open Source software requires time, and every minute of that time you need to pay for food, for shelter, for the privilege to meet with your friends and be inspired. Open Source maintenance does not release maintainers from paying for all this stuff and doesn't bring them those payment tokens called `money`. So code in maintainers heads will have to die.

How come that current economy is so flawed? Does Open Source produce value? Definitely, yes. Does it solve problems for people? Yes. Do people feel grateful for Open Source? Well, sometimes (but that's also the question of consumer culture forced by current economy). And still open source developers leave their products to join (often meaningless) activities of reinventing things from scratch for other projects. Often doing things that they've paid for and not things they know and like.

How come that people with developed skills doing useful things have nothing to eat, and for food and shelter have to reformat their brain and became compliant with office slavery? Sometimes I feel like we should extend software freedom conservancy model from software to people and remove "people with money" from power part of this equation. Blockchain allows us to calculate personal values to remove this money-for-money gameplay. Gameplay that creates parasites out of people, parasites that are motivated to produce garbage, to consume and to battle each other for "market domination". Parasites don't like to be exposed, and they don't like to be parasites, so maybe Open Economy is an answer that can bring our focus back to the planet with depleting resources and increasing pollution that pushes our health and general feeling of the world down to the level of committing a suicide.

I'd like to believe, but I am loosing my insight sometimes. There are many good people who care about me in this world, and this letter won't be there if not them. But their lifetime is limited, and so is mine. I really wanted to boost progress to help with my health and global problems, but it seems I failed. This post is filled under a Python tag, which always was a part of that unconscious plan. I thought that I could enhance it to be an easier, and more intuitive tool to augment limited human abilities for automation, to help them concentrate on more important things. But it seems I failed. I still see things how they should be done, I still see that DX (developer's experience) need to raise to the top of people priorities. It is just that I am exhausted to compete for food and shelter and feel depressed over the need to reformat my natural intelligence network to do those (often meaningless) office things just to keep my body alive.

Sunday, January 10, 2016

Reaction to Python development moving to GitHub

If you don't know, the Python core developers are considering move to GitHub, which I believe will abandon Roundup, Mercurial and a couple of other initiatives that some of us picked, because they were used by community and were useful to it. For example I added support for Jinja2 to Roundup, tried to add a new router component for it to enable adding new web APIs more easily, and now I am doing a release of new version, because that package is valuable.

Below is my post sent to as an immediate reaction to the PEP that addresses the migration. I think that people need to consider Python as something with much better potential than just "a product to execute computer scripts". I want people to think about it as about open process with all the involved complexity and dynamics issues, a place to study and learn about different culture than following the orders of your owner founder. Some want to see it to be isolated coding playground without all that disturbances and annoying people with external world problems. But it is unavoidable. There are already diversity statement and codes of conducts. It is time to stop being ignorant to economics for the balance.

Blender is a good example. 3D editor that every two years chooses a problem that is actual for community and makes a movie together with building necessary features in the editor. They discuss economics, try to address funding issues by working with funds outside of coding domain, bringing new initiatives, communicating and still staying open - not ignoring the issues and not trying to bring the same hammer on the new "nail". Why Python fails to follow the same route?

I don't know why it bother me that much. Probably because I am jobless and living in northern country where you can't live on street and need to pay rent, so "getting paid for the useful stuff that you do" is extremely important issue for me. But also I feel that my time is limited, and that it is very sad to spend my life behind the screen of computer, reinventing some code that was written numerous time for a company that can not support something that is already written.

I realize that with reduced exposure to a projects like Roundup and Mercurial it will be much harder (if possible at all) to gather critical mass of resources around people to experiment with sustainability models for open source, and doing these things is important, because we coming into the age of extremely powerful automation where people may not be that needed anymore. For me open source is the only way to gather data on what will happen, and maybe the only way to find the balance. Open source economics right now is when resources are limited, but there are many people how could use them, but if they use them, others (who probably can do better) may feel bad about this. This is a just a single problem that needs to be addressed in the process, and I wish that there could be that sustainable working process in the list.

Here is original mail sent:

Thursday, September 03, 2015

SCons build targets

SCons is awesome. Just saying. If you want to know (or troubleshoot) how SCons selects targets to be built, add this snippet at the end of your SConstruct:
def dump_targets(targets):
  for t in targets:
    if type(t) == str:
      name = t
      name =
    print("  <" + str(t.__class__.__name__) + "> " + name)

print("[*] Default targets:")

print("[*] Command line targets:")
print("[*] All build targets:") dump_targets(BUILD_TARGETS)
For my copy of Wesnoth, 'scons .' produces this output:
[*] Default targets:
  <Alias> wesnoth
  <Alias> wesnothd
[*] Command line targets:
  <str> .
[*] All build targets:
  <str> .
And if you want to know how to specify targets or what do they mean, read the second page of SCons man documentation. Just for convenience I quote it here.

scons is normally executed in a top-level directory containing a SConstruct file, optionally specifying as command-line arguments the target file or files to be built.

By default, the command
will build all target files in or below the current directory. Explicit default targets (to be built when no targets are specified on the command line) may be defined the SConscript file(s) using the Default() function, described below.

Even when Default() targets are specified in the SConscript file(s), all target files in or below the current directory may be built by explicitly specifying the current directory (.) as a command-line target:
scons .
Building all target files, including any files outside of the current directory, may be specified by supplying a command-line target of the root directory (on POSIX systems):
scons /
or the path name(s) of the volume(s) in which all the targets should be built (on Windows systems):
scons C:\ D:\
To build only specific targets, supply them as command-line arguments:
scons foo bar
in which case only the specified targets will be built (along with any derived files on which they depend).

Specifying "cleanup" targets in SConscript files is not usually necessary. The -c flag removes all files necessary to build the specified target:
scons -c .
to remove all target files, or:
scons -c build export
to remove target files under build and export. Additional files or directories to remove can be specified using the Clean() function. Conversely, targets that would normally be removed by the -c invocation can be prevented from being removed by using the NoClean() function.

A subset of a hierarchical tree may be built by remaining at the top-level directory (where the SConstruct file lives) and specifying the subdirectory as the target to be built:
scons src/subdir
or by changing directory and invoking scons with the -u option, which traverses up the directory hierarchy until it finds the SConstruct file, and then builds targets relatively to the current subdirectory:
cd src/subdir
scons -u .

Sunday, August 16, 2015

Technical Debts for Python Community

Debts and credits is a new age slavery. Technical debts are better, because they are just a disease. It can fatal if not treated, but for a strong community it is not a problem. Still a good working plan and a vague roadmap is needed to coordinate many eyes to go in the same direction and push things forward where it is hard to reach a consensus.

Dangers of Technical Debts

Technical debt is hard to understand and identify, but once your know about it, it will be easy to spot those things early. The most common symptom of technical debt is stolen time, which if not treated becomes a paralysis. You may know how your system works, and all the components, and where the files are, but you also have the previous experience about how much time it takes to modify and properly test the system, and ensure that everything is correct, and you just know that you just don't have the time, because your family and friends are missing you. We are all volunteers, etc.

Recipe: Simplify Your Systems, Reduce the Complexity, Automate Things and Think in 15 Minutes Slots.

Thinking in 15 Minutes Slots

This may not be so important for enterprise projects where people exchange their time for money, but my opinion is that it is extremely important for open source projects where time is distributed over many people.

Have you ever wondered what will a highly talented person be able to do for Python if we are lucky enough to get 15 minutes of his or her attention?

I am afraid that the sole "contribution" would be making a checkout. Even correcting a mistake in documentation requires that. Maybe the most he or she could accomplish would end in setup of Git or Mercurial. And if the checkout is a long and boring, the person may lose interest and switch to something different even before 15 minutes are expired.

I will continue with how technical debts are evolving into Competence Debt,  but first.. let me take a selfie tell a story. Just to complete the 15min section, I once accepted a challenge to make a design fix for Launchpad. I spent 15 minutes 4 times - an hour - and couldn't even get the checkout. Here are the sessions:

FAIL: edit wiki - move bzr instructions to the top
send a letter to that wiki pages are not editable
Yes, I got distracted during the first session, but I want to make the world better by fixing things on the way. Of course, I'd like those problem not to appear in the first place. Let's see how a next session ended.

# create LXD container for experiments
$ lxc init ubuntu lp
$ lxc start lp
$ lxc exec lp -- bash
# apt-get install bzr
# bzr branch lp:launchpad

The second slot was all about reading instructions and setting up "virtualenv for Linux" to install all the prerequisites without polluting my main system (and drop them without consequences). I already knew about LXD, so my competence here was already high to save some time on learning that. BTW, LXD rocks. Just try it.

# cd launchpad
# apt-get install postgresql make
# ./utilities/launchpad-database-setup $USER
# make schema
FAIL: many errors
# utilities/update-sourcecode

These 15 minutes left me in confusing state without any working instance to get some positive feedback on what I am doing. At this moment I already have a strong desire to just drop everything. And yet after some time I get back to spend another 15 minutes slot trying to tackle the problem.

# drop old LXD container
lxc delete lp
# create new LXD container
lxc init ubuntu lp
lxc start lp
lxc exec lp -- bash
# install basic dependencies
apt-get install bzr make postgresql wget
bash rocketfuel-setup
# ^ need to enter name, and click Y on Apache install prompt
  Making local branch of Launchpad trunk, this may take a while...
  You have not informed bzr of your Launchpad ID, and you must do this to
  write to Launchpad or access private data.  See "bzr help launchpad-login".
  bzr: ERROR: Connection error: Couldn't resolve host '' [Errno -2] Name or service not known
  ERROR: Unable to create local copy of Rocketfuel trunk
# attempt to repeat the script
bash rocketfuel-setup
  bzr: ERROR: No WorkingTree exists for "file:///root/launchpad/lp-branches/devel/.bzr/checkout/".
  ERROR: Your trunk branch in /root/launchpad/lp-branches/devel is corrupted.
         Please delete /root/launchpad/lp-branches/devel and run rocketfuel-setup again.

Now I really drop it. So after that user experience I doubt I will ever get hacking on LaunchPad again. So, if your technical debt provides poor experience for onboarding users, you're likely to lose them for a lifetime.

The Debt of Competence

Competence Debt is a chronic phase of the Technical Debt illness. When time for operation become more than your daily limit, there comes paralysis, and after that the worst thing that can happen is when you no longer know how your systems work and lack skills to restore the picture.

From that moment your project is entering the death spiral and it is only a matter of time when it will be dead. I've seen several examples where programmers was treated like a replaceable material, but the truth is that program lives as long as its code is alive in the heads of its maintainers. There is no such thing as "a software product" anymore - software is more about support and development, than about selling products on a local market.

For open source projects competence debt usually results in various rewrites and long term stale issues. Many attempts to fix them, many hours wasted just to hit the wall with new heads over and over. The power of open source is a little time and little effort that is distributed over many people to create momentum. That was the original idea behind the domain when it all started many years ago. And it is also the result of OpenStreetMap success - small and clear activities that don't require much time and competence to accomplish. This scales well and provides a good gameplay.

Recipe: Invest in Visualization and Learn Visual Tools (SVG, D3.js, Inkscape) to Explore Ways to Transfer Your Competence to Other People

Text is not a natural way for people to consume and produce information. We learn how to read and write, and it takes more than a month to get used to it. But learning to play games like World of Tanks just takes a few minutes. The new generation that I am a part of is used to watch YouTube lectures on 2x speed, read only first 150 letters of the messages and scan long texts without actually thoroughly reading them. That's why I highlight the key points in this post. We were developing tools for audio/visual communication naturally over all these many years - 3D graphics, demoscene, virtual reality, and now deep learning networks, but we still find it hard to produce visual material for communicate other ideas. Because we've been taught to write text, not to produce beautiful art that just works. Learn to draw. It makes people happy learning something new.

OpenStreetMap - The Earth as The Outline

Here is a success story. No, it won't teach how to remove technical debts, but rather give an idea how to restructure it, so that a thousand eyes could make an impact.

The OpenStreetMap has a reference model - it is our Earth. We just copy what we see into vector form to draw a map and everybody could validate with their own eyes. With open source project it is all the same, except that you need to create that reference model and that should be actionable -  split into many pieces that people can validate in parallel separately. Think about specification where every is independent enough to serve both as an entrypoint and a clause to put a checkmark next to it if a condition is true. Think about a canvas, where everybody can draw the common vision and then see who has drawn the components that they can reuse in their own sketches. The reference model is what you need to know where to push so that your small effort could contribute to a greater goal. The Roadmap also tell you where your skills will be more useful, and ensures that your efforts will not be wasted

The Role of Foundations

People think of foundation as of fund. That's not effective. I need about $600/m to cover shelter and food expenses. Travelling, buying clothes and stuff, covering medical expenses raises the plank to $1000/m, girlfriend may add another $500/m, building a house another $500/m, and I don't even want to think about children. There is no chance I will be able to afford this. So at a bare minimum a foundation should provide $600/m per person to deal with EPIC issues that nobody could deal with in their spare time. Forgot the taxes. Add another $600/m on top of that, and that's just one person, and you need at least two of them. So, $2400/m just to make router for so that we can add more URLs endpoints via extensions for interactive frontends and  REST API to it. Nobody will ever pay for that. We tried to hack the problem with Gratipay, but got a flashback from a protection mechanism of U.S. financial system. It is clearly a dead end to fight the World owned by corporations (the World as it was already 100 years ago).

Instead, the role of the foundation is to explain to corporation the above mathematics of time and effort, to enable people in these companies give more time to contribute their professional expertise to deal with complexity and reduce competence debt for the community. Applying the recipes to reduce the technical debt to inspire people. Employing art for documenting the systems and structures, so that people could digest the information easily. Organising in-house sprints to deal with important matters we alone, with less time, but many hands can not tackle on our own.

The role of foundations is not to empower individuals, but collect the data about obstacles, foresee and communicate about them on the path ahead, and organize clean up efforts where they are needed, so that anybody who got those precious 15 minutes knew what to do, and could spend those 15min most effectively to bring their contribution to the common stream that benefits everyone.