Why Git suxx

$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.

$ git checkout https://github.com/gratipay/grtp.co/pull/178
 Checkout aborted
 There's no file https:\github.com\gratipay\grtp.co\pull\178 at HEAD

$ git switch https://github.com/gratipay/grtp.co/pull/178
git: 'switch' is not a git command. See 'git --help'.

Solution

Download https://github.com/tj/git-extras/blob/master/bin/git-pr to your git --exec-path.

Python Usability Bugs: Formatting argparse subcommands

Suppose you want to build a tool with a simple interface:

usage: sub <command>

commands:

  status -  show status
  list   -  print list

Python proposes to use argparse module. And if you follow documentation, the best you can get will be this output:

usage: sub {status,list} ...

positional arguments:
  {status,list}
    status       show status
    list         print list

And it you implement proper formatting, your code will look like this:

import argparse
import sys

class CustomHelpFormatter(argparse.HelpFormatter):
  def _format_action(self, action):
    if type(action) == argparse._SubParsersAction:
      # inject new class variable for subcommand formatting
      subactions = action._get_subactions()
      invocations = [self._format_action_invocation(a) for a in subactions]
      self._subcommand_max_length = max(len(i) for i in invocations)

    if type(action) == argparse._SubParsersAction._ChoicesPseudoAction:
      # format subcommand help line
      subcommand = self._format_action_invocation(action) # type: str
      width = self._subcommand_max_length
      help_text = ""
      if action.help:
          help_text = self._expand_help(action)
      return "  {:{width}} -  {}\n".format(subcommand, help_text, width=width)

    elif type(action) == argparse._SubParsersAction:
      # process subcommand help section
      msg = '\n'
      for subaction in action._get_subactions():
          msg += self._format_action(subaction)
      return msg
    else:
      return super(CustomHelpFormatter, self)._format_action(action)


def check():
  print("status")
  return 0

parser = argparse.ArgumentParser(usage="sub <command>", add_help=False,
             formatter_class=CustomHelpFormatter)

subparser = parser.add_subparsers(dest="cmd")
subparser.add_parser('status', help='show status')
subparser.add_parser('list', help='print list')

# custom help messge
parser._positionals.title = "commands"

# hack to show help when no arguments supplied
if len(sys.argv) == 1:
  parser.print_help()
  sys.exit(0)

args = parser.parse_args()

if args.cmd == 'list':
  print('list')
elif args.cmd == 'status':
  sys.exit(check())

Here you may see the failure of OOP (object oriented programming). The proper answer to this formatting problem is just to define a data structure for command line help in JSON or similar format and let people dump and process it with templates. Once option definition is parsed, the information there is static, so there is no need in those intertwined recursive method calls. So just do it in 2 pass - get dataset and render template.

Technical Debts for Python Community

Debts and credits is a new age slavery. Technical debts are better, because they are just a disease. It can fatal if not treated, but for a strong community it is not a problem. Still a good working plan and a vague roadmap is needed to coordinate many eyes to go in the same direction and push things forward where it is hard to reach a consensus.

Dangers of Technical Debts

Technical debt is hard to understand and identify, but once your know about it, it will be easy to spot those things early. The most common symptom of technical debt is stolen time, which if not treated becomes a paralysis. You may know how your system works, and all the components, and where the files are, but you also have the previous experience about how much time it takes to modify and properly test the system, and ensure that everything is correct, and you just know that you just don't have the time, because your family and friends are missing you. We are all volunteers, etc.

Recipe: Simplify Your Systems, Reduce the Complexity, Automate Things and Think in 15 Minutes Slots.

Thinking in 15 Minutes Slots

This may not be so important for enterprise projects where people exchange their time for money, but my opinion is that it is extremely important for open source projects where time is distributed over many people.

Have you ever wondered what will a highly talented person be able to do for Python if we are lucky enough to get 15 minutes of his or her attention?

I am afraid that the sole "contribution" would be making a checkout. Even correcting a mistake in documentation requires that. Maybe the most he or she could accomplish would end in setup of Git or Mercurial. And if the checkout is a long and boring, the person may lose interest and switch to something different even before 15 minutes are expired.

I will continue with how technical debts are evolving into Competence Debt,  but first.. let me take a selfie tell a story. Just to complete the 15min section, I once accepted a challenge to make a design fix for Launchpad. I spent 15 minutes 4 times - an hour - and couldn't even get the checkout. Here are the sessions:

https://launchpad.net/~inkscape.dev/+archive/ubuntu/stable - fix dialog here

open https://dev.launchpad.net/

open https://dev.launchpad.net/Getting

FAIL: edit wiki - move bzr instructions to the top

send a letter to https://lists.launchpad.net/launchpad-dev/ that wiki pages are not editable

Yes, I got distracted during the first session, but I want to make the world better by fixing things on the way. Of course, I'd like those problem not to appear in the first place. Let's see how a next session ended.

open https://dev.launchpad.net/Getting

# create LXD container for experiments

$ lxc init ubuntu lp

$ lxc start lp

$ lxc exec lp -- bash

# apt-get install bzr

# bzr branch lp:launchpad

open https://dev.launchpad.net/Running

The second slot was all about reading instructions and setting up "virtualenv for Linux" to install all the prerequisites without polluting my main system (and drop them without consequences). I already knew about LXD, so my competence here was already high to save some time on learning that. BTW, LXD rocks. Just try it.

# cd launchpad

# apt-get install postgresql make

# ./utilities/launchpad-database-setup $USER

# make schema

FAIL: many errors

reading https://dev.launchpad.net/Running

# utilities/update-sourcecode

These 15 minutes left me in confusing state without any working instance to get some positive feedback on what I am doing. At this moment I already have a strong desire to just drop everything. And yet after some time I get back to spend another 15 minutes slot trying to tackle the problem.

# drop old LXD container

lxc delete lp

# create new LXD container

lxc init ubuntu lp

lxc start lp

lxc exec lp -- bash

# install basic dependencies

apt-get install bzr make postgresql wget

# talking in https://lists.launchpad.net/launchpad-dev/

wget https://git.launchpad.net/launchpad/plain/utilities/rocketfuel-setup

bash rocketfuel-setup

# ^ need to enter name, and click Y on Apache install prompt

FAIL:

  Making local branch of Launchpad trunk, this may take a while...

  You have not informed bzr of your Launchpad ID, and you must do this to

  write to Launchpad or access private data.  See "bzr help launchpad-login".

  bzr: ERROR: Connection error: Couldn't resolve host 'bazaar.launchpad.net' [Errno -2] Name or service not known

  ERROR: Unable to create local copy of Rocketfuel trunk

# attempt to repeat the script

bash rocketfuel-setup

FAIL:

  bzr: ERROR: No WorkingTree exists for "file:///root/launchpad/lp-branches/devel/.bzr/checkout/".

  ERROR: Your trunk branch in /root/launchpad/lp-branches/devel is corrupted.

         Please delete /root/launchpad/lp-branches/devel and run rocketfuel-setup again.

Now I really drop it. So after that user experience I doubt I will ever get hacking on LaunchPad again. So, if your technical debt provides poor experience for onboarding users, you're likely to lose them for a lifetime.

The Debt of Competence

Competence Debt is a chronic phase of the Technical Debt illness. When time for operation become more than your daily limit, there comes paralysis, and after that the worst thing that can happen is when you no longer know how your systems work and lack skills to restore the picture.

From that moment your project is entering the death spiral and it is only a matter of time when it will be dead. I've seen several examples where programmers was treated like a replaceable material, but the truth is that program lives as long as its code is alive in the heads of its maintainers. There is no such thing as "a software product" anymore - software is more about support and development, than about selling products on a local market.

For open source projects competence debt usually results in various rewrites and long term stale issues. Many attempts to fix them, many hours wasted just to hit the wall with new heads over and over. The power of open source is a little time and little effort that is distributed over many people to create momentum. That was the original idea behind the rainforce.org domain when it all started many years ago. And it is also the result of OpenStreetMap success - small and clear activities that don't require much time and competence to accomplish. This scales well and provides a good gameplay.

Recipe: Invest in Visualization and Learn Visual Tools (SVG, D3.js, Inkscape) to Explore Ways to Transfer Your Competence to Other People

Text is not a natural way for people to consume and produce information. We learn how to read and write, and it takes more than a month to get used to it. But learning to play games like World of Tanks just takes a few minutes. The new generation that I am a part of is used to watch YouTube lectures on 2x speed, read only first 150 letters of the messages and scan long texts without actually thoroughly reading them. That's why I highlight the key points in this post. We were developing tools for audio/visual communication naturally over all these many years - 3D graphics, demoscene, virtual reality, and now deep learning networks, but we still find it hard to produce visual material for communicate other ideas. Because we've been taught to write text, not to produce beautiful art that just works. Learn to draw. It makes people happy learning something new.

OpenStreetMap - The Earth as The Outline

Here is a success story. No, it won't teach how to remove technical debts, but rather give an idea how to restructure it, so that a thousand eyes could make an impact.

The OpenStreetMap has a reference model - it is our Earth. We just copy what we see into vector form to draw a map and everybody could validate with their own eyes. With open source project it is all the same, except that you need to create that reference model and that should be actionable -  split into many pieces that people can validate in parallel separately. Think about specification where every is independent enough to serve both as an entrypoint and a clause to put a checkmark next to it if a condition is true. Think about a canvas, where everybody can draw the common vision and then see who has drawn the components that they can reuse in their own sketches. The reference model is what you need to know where to push so that your small effort could contribute to a greater goal. The Roadmap also tell you where your skills will be more useful, and ensures that your efforts will not be wasted

The Role of Foundations

People think of foundation as of fund. That's not effective. I need about $600/m to cover shelter and food expenses. Travelling, buying clothes and stuff, covering medical expenses raises the plank to $1000/m, girlfriend may add another $500/m, building a house another $500/m, and I don't even want to think about children. There is no chance I will be able to afford this. So at a bare minimum a foundation should provide $600/m per person to deal with EPIC issues that nobody could deal with in their spare time. Forgot the taxes. Add another $600/m on top of that, and that's just one person, and you need at least two of them. So, $2400/m just to make router for https://bugs.python.org/ so that we can add more URLs endpoints via extensions for interactive frontends and  REST API to it. Nobody will ever pay for that. We tried to hack the problem with Gratipay, but got a flashback from a protection mechanism of U.S. financial system. It is clearly a dead end to fight the World owned by corporations (the World as it was already 100 years ago).

Instead, the role of the foundation is to explain to corporation the above mathematics of time and effort, to enable people in these companies give more time to contribute their professional expertise to deal with complexity and reduce competence debt for the community. Applying the recipes to reduce the technical debt to inspire people. Employing art for documenting the systems and structures, so that people could digest the information easily. Organising in-house sprints to deal with important matters we alone, with less time, but many hands can not tackle on our own.

The role of foundations is not to empower individuals, but collect the data about obstacles, foresee and communicate about them on the path ahead, and organize clean up efforts where they are needed, so that anybody who got those precious 15 minutes knew what to do, and could spend those 15min most effectively to bring their contribution to the common stream that benefits everyone.

Lifehack - import directly from command line

If you're obsessed with UX as I am, and long repetitive commands seem to break your flow, you may appreciate this little trick for invoking Python from command line.

I invoke python from console/terminal to check some behavior that I tend to forget after a day in JavaScript or other programming languages. It is often way faster than reading the docs. It also helps to quickly try new PyPI libraries after installing with pip (saves on mouse clicking in IDE etc.).

Today I found a time saver that makes me happy. This example is typed for Windows, but there should be no problem for any Linux hacker to port it to their favorites. So instead of doing this:

> python [ENTER]
>>> import os [ENTER]
>>> os.name [ENTER]
'nt'

I can just type:

> import os [ENTER]
>>> os.name [ENTER]
'nt'

It should be now easy to understand how it works, so I will complicate it a bit. The trick is this little import.bat script, placed in system PATH:

@echo off
py -2 -c "import sys; print(sys.version)"
py -2 -i -c "import %*"

py here is a py.exe launcher that can call either Python 2 or Python 3 depending on argument. It comes installed with Windows Python versions. I like to know which Python I work with, so I've added version into to the command, and to avoid polluting the global space of new interpreter, the version is printed by separate command.

ANN: hexdump 3.2 - packaged and pretty executable

https://pypi.python.org/pypi/hexdump

The new version doesn't add much to the API, but completes a significant research about executable .zip packages with proof of concept distribution that is both an installable Python package that you can upload to PyPI and an executable .zip archive.

The benefit is that you may drop .zip package downloaded from PyPI into your source code repository and invoke it from your helper scripts as python hexdump-3.2.zip Keeping dependencies together with application source code in repository may seem like a bad idea, but we find it useful at Gratipay to setup project from checkout in offline mode with no internet connection.

The solution required three modifications to distutils setup.py (see sources):

• force sdist command to produce .zip file
• strip extra dir created at PyPI package root
• add __main__.py to make .zip executable

If you think that it is cool and should be promoted further, feel free to test Gratipay to send me 0.01+, and I'll get the signal to spend more time on reversing native distutils/packaging gotchas.

Shipping Python tools in executable .zip packages

UP201501: Found out how to force setup.py create .zip instead of .tar.gz

This is about how to make Python source packages executable. As a side effect, this also explains, how to run invoke's tasks.py with a copy of invoke .zip shipped with your source code, but without installing it.

You know, Python can execute .zip files. As simply as:

$ wget https://pypi.python.org/packages/source/h/hexdump/hexdump-3.1.zip
$ python hexdump-3.1.zip
/usr/bin/python: can't find '__main__.py' in 'hexdump-3.1.zip'

Well, you need to place '__main__.py' into the root. It will then import what it needs and do something. The only problem with .zip source packages is that their structure (if created with standard Python tools) includes one additional directory at the top:

`-- hexdump-3.1
    |-- PKG-INFO
    |-- README.txt
    |-- hexdump.py
    |-- hexfile.bin
    `-- setup.py

So, even if you place `__main__.py` into the root, it won't be able to import file from `hexdump-3.1` subdirectory.. unless you prepare for it. Here is the solution:

import os
import sys

# add package .zip to python lookup path
__dir__ = os.path.dirname(__file__)
path = os.path.join(__dir__, 'hexdump-3.1')
sys.path.insert(0, path)

import hexdump
msg = hexdump.dehex("48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21")
print(msg)

Now if you run it, it will print the obvious.

$ python hexdump-3.1.zip
Hello, World!

In Python 3, the result will be wrapped into b'' string. But that's it!

The next hexdump version will likely to ship as executable .zip package

Where you may need it?

It may be useful for scripts automation if you're "vendorizing" dependencies (ship them with your code). It started with need to fix a broken link on Gratipay site. The Gratipay codebase is in public domain clear of any NDA, CLA and all other BS, and that makes it great for people to learn, reuse and enhance. I wish Python internal projects possessed these properties, but PSF politics is another topic.

Gratipay inside site uses make. It doesn't need powerful build tool, such as scons, so I tried to replace it with some simple task automation utility to be more cross-platform. I chose invoke, because of my previous experience with fabric for remote control, and because I saw previous attempts in another Gratipay repository. The necessary condition for new tool is that user experience should not degrade for make users, so main invoke's tasks.py script needed to be self-executable.

I completed the proof of concept by making invoke .zip package executable with the method above, placed it into vendor/ directory, and tuned tasks.py to reexecute itself with invoke .zip This also solved a potential problem with currently unstable invoke API.

tasks.py was modified as following:

import sys
sys.path.insert(0, 'vendor/invoke-0.9.0.zip')

from invoke import task, run

# ... tasks go here

if __name__ == '__main__':
  import sys
  from invoke import cli
  cli.main()

Bonus points

There is one more thing, which is better explained with hexdump example. hexdump package ships `hexfile.bin` data file used for testing. hexdump.py looks for it in its directory (__file__ dirname), and this lookup fails when hexdump.py is in archive. I'd say Python could provide some kind of an API to deal with "virtual import filesystem" (with watchers to track who and when modifies sys.path), but I digress. If you modify the `__main__.py` from the first chapter above to run tests - just call hexdump.runtest() - the execution will finally fail with the following error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "hexdump-3.1.zip/__main__.py", line 13, in
  File "hexdump-3.1.zip/hexdump-3.1/hexdump.py", line 311, in runtest
IOError: [Errno 20] Not a directory: '/root/hexdump-3.1.zip/hexdump-3.1/hexfile.bin'

(yes, I am hexdumping under root). The problem above was explained by Radomir, and I feel like the post is already tool long to write about it myself.

More bonus points

In theory it is possible to hack setup.py to produce executable source .zip package. By default, setup.py sdist command on Linux creates tar.gz files. So, first it needs to be forced to create .zip file. This can be done by overriding default --formats option:

# Override sdist to always produce .zip archive
from distutils.command.sdist import sdist as _sdist
class sdistzip(_sdist):
    def initialize_options(self):
        _sdist.initialize_options(self)
        self.formats = 'zip'

setup(
    ...
    cmdclass={'sdist': sdistzip},
)

I posted relevant documentation links to this StackOverflow question. The rest - how to inject __main__.py into the root of packed .zip - is yet to be covered, and I am not ready to research it just yet. Some hints may be provided by these answers.

Usability conclusion

I am actually quite happy with the ability to execute python .zip packages (which gave a lot of motivation to write this post), because previously I had to care about how to tell people that a tool is a single script, which can be downloaded from repository or from some dedicated download site (I mean Google Code, which does not support this anymore). I also don't trust my own server for downloads, because one day I found some "benzonasos" running there and I don't even have a slightest idea how it got there.

But now it becomes possible to just download and run the stuff from PyPI to test how it works without messing with creating and activating virtualenvs, and installation of package inside. Of course, if your .zip package has dependencies, they still need to be present on your system, but that's still a time saver. Oh, and that means that my tools can now consist of several modules inside.

ANN: sha1 1.0 - command line tool to calculate file hash

Windows doesn't have a native tool to calculate SHA-1, so I've made one that can be easily installed and used from command line on any operating system thanks to Python:

python -m pip install sha1
python -m sha1 <filename>

ANN: xtrace 0.5 - indented function trace in Xdebug format

Xdebug is a PHP tool that allows to trace how PHP code is executed. Today I release a tool called xtrace into public (and more specifically into public domain), which allows to get the same (or at least very identical) output, but for Python.

Few years ago inability to trace function calls in Python came as a showstopper. I decided to write a familiar tool for Python. More than that - I wanted to integrate it into Spyder. But a year ago xtrace itself faced with showstopper. The showstopper was the behavior of execfile function, that I could not get right at that time, because of the docs, of my expectations and poor knowledge of English. Maybe there is a flaw in my cognitive abilities, but I tried to get at this problem several times and failed. Until recently some hackers from ZenSecurity team brought a concept known an pyjail to my attention. The challenge to prove that pyjail concept is impossible allowed me to concentrate on gory details of execfile works and knowing that documentation is totally confusing for my, I found the time to set my own experiments. You can read them at the link I've given above as well as some analysis why documentation that actually includes all the details can be bad and confusing.

The xtrace was basically broken for three years, starting from the version 0.2 - the day I put execfile() call from root to the xtrace module to the main() function. This placement changed the execfile() behavior, and while trying to debug that I also run into confusing dynamic behavior of dictionary returned by locals(). Opened can of worms made those parasites to completely consume my brain, causing much anger and frustration to be spilled around execfile() and locals() concepts over into Python lists. It is kind of relief now that I can name all the problems, analyse them and look back as enlightened. Being jobless I had a plenty of time to investigate, but I really don't want anyone to enter that state of confusing and helplessness that I had a year ago.

Hopefully, my experience with xtrace will clear the confusion for those who will try to use exec type abilities of Python for developing their own tools. Maybe it will result in a better Python API in the future, with better documentation and position-independent behavior.

I am interested to know the feedback that you can leave in xtrace tracker, such as if the output really matches PHP behavior, if it is accepted by PHP tools and how it behaves in different scopes of Python. It is interesting to convert it to Spyder plugin and see the usage in other tools, but I realize that I may not have time for that. The next focus for me is to add an easy API to xtrace to enable people to write they own tracers more easily. Focus on UX and everything else will come.

Roundup Tracker: Create Issues by Email

There is one thing about bugs.python.org and other Roundup issue tracker instances that is not widely known. It is the fact that you can create new issues and update old ones directly from your email client, without visiting web interface at all.

As much as I hate Debian's email-only tracker, I must admit that having email control feature in addition to web interface can save some time, especially if you constantly forget passwords for different trackers like me.

So, to create new issue, just send email to the address that tracker uses to send mail to you. Well-known addresses of Python trackers:

• report@bugs.python.org - for filling bugs in Python with b.p.o tracker
• metatracker@psf.upfronthosting.co.za - for reporting problems with b.p.o tracker itself

Note that your email needs to be present in tracker database for it to accept your request, so you might need to create your account first.

You can also update existing issues by adding suffixes like [status=closed;resolution=invalid] to the subject field of your replies. I just closed issue19825 to test this method. You can try it too next time you feel uncomfortable about escaping from your mailbox.

This stuff is actually documented in official Roundup docs, but who reads the docs, anyway.

Mercurial UX: Undo/Redo Wanted

This is an adapted mail for Mercurial mailing list, which is good to have as a blog post for reference.

For those of you who was born in Github era, Mercurial is an alternative version control system with transparent, pythonic internals. Because all my projects are escaping to Github I took a chance to reiterate over my knowledge of HG and see what I missed over the years of using it. This is just one idea.

This year, Mercurial introduced new ChangesetEvolution concept, which allows to safely mess with repository history. I decided to take a look and started with 'hg fold' command. Quite soon I got into a usual state of missed RTFM evening (you know the evening when you have a plenty of time to read a book with a cup of coffee). I couldn't understand what happened, but I clearly knew it is not something I want, so I wanted to get my repo back into the initial state. There are a lot of commands like 'rollback', 'revert', 'update -C', 'backout', 'strip' to revert the state after some command, but the real problem is to choose the right one. So I thought that it is something that is missing.


In Mercurial (and in other version control systems as well) - there is no concept of "operational transaction". In databases no matter what you do, if transaction is not committed, the state is reverted. These are called atomic transactions. Before Subversion there was CVS with non-atomic commits - if there was an error with some file (merge or something) - you got half of files committed and half not. Awful, right? After SVN all commits are atomic - if something is wrong, nothing is committed. Atomicity is important for user operations too. If something goes wrong - I want to get back from where I started. In Mercurial it works by making a backup copy of your repo. I guess for Git it's the same.

So, no obvious command to revert the last operation, no atomicity on operation level. This makes me feel unsafe and unsure about what can I do in my clone if I am too lazy to make a copy. And I thought that the next step in Mercurial evolution would be going from "user command" to "user operation" concept.


"user command" is a command like `hg inc` that users type in command line. It can affect the state of repository or not.

"user operation" - is a command or commands that change the state of repository. The "user operation" has a property of being "revertible" or "not". Granularity of changes to repository (how many commands is one operation) is decided using the high level user level goal to undo and redo these operations. For example 'hg fold' is an command that can be undone. It is a separate "user operation" and an entry in "undo history".

"user command" that modifies state may have "reverse command" that brings the changes back to the initial state. But maintaining this on command level is too fragile and hard to remember "commit/rollback". "user operation" may not have a "reverse command" - it may just be reverted without dedicated reverse command (like when you replace clone with your backup copy). And for that you need "undo history".

"undo history" is a stack of "user operations". These can be revertible or not - it depends on the logic. And it is not a commit log - it is operations log. The direct analogy is GIMP undo history dialog.

Now that the concept of the feature wanted is clear, some blueprints for the starter.

From the usability POV, a mercurial operations history dialog is a list, where each entry contains:

- operation name

- if it can be undone

  - if not, state the reason

    the reason is necessary to understand either:

       1. current condition of repository

           - what should be adjusted to enable undo

           - why adjustment can not be automated

       2. what should be written in hg itself to make it possible

           - pointer to dev docs and status page

Summary:
 * user command ('hg inc', 'hg ci', ...)
 * user operation (hg command that changes state)
 * undo history (stack of latest user operations)
   * undo history items are frozen if reverting is impossible
   * undo history is local
 * state explanation between operations

Links:
https://www.google.by/search?q=undo+pattern  - command and memento patterns can help
https://bitbucket.org/hstuart/hg-multiundo  - some work on the topic was done by Henrik Stuart

The final test:

 hg undo

 hg redo

 hg undo --list

If you have what to say, but are not subscribed to continue thread in official mercurial@selenic.com mailing list, then I guess it's safe to leave comments here.

Command to generate SSH key on Linux

Because ssh-keygen has human unfriendly command line interface without --help option, here is a quick reminder how to generate very secure SSH key valid for one year (52 weeks) with "13.04" comment for its public part:

ssh-keygen -t rsa -b 4096 -V +52w -C 13.04

This key can be used to avoid typing password when pushing your Python code to sites like Bitbucket or GitHub (Google Code is not there yet) or to securely upload your packages to PyPI.

Program config as a DNA strand

This is a technical followup to the post about mind-altering programming languages, which concentrates, iterates and extends on the abstract DNA part.

Do you know how DNA looks like from the point of view of Python programmer?
DNA is a list. Of genes, and fillers in between.

    [ GENE, GENE, GENE, None, GENE, STUFF, GENE, .... ]

By the analogy with software, one GENE is one option. Here None or STUFF is something that is not recognized and not used in the process right now, but may be used later even in a different place for a different purpose. This leads to very interesting properties of configuration described as a DNA strand.

Property One: Everything is Optional and Configurable

Usually it is a tough choice what to include into your program config, because an option a day keeps users away. But with DNA config it doesn't matter - you can encode everything - every bit of flexibility you wanted, every feature-creeper, forward-thinking matter. Don't bother yourself with restrictions - include everything.

Property Two: Indefinite Logical Groups and Namespaces

To make sense out of tons of information in a DNA strand, you need to concentrate only on the parts that are relevant to you. This is done by "masking" (or marking) parts of the DNA code according to your current task. When you apply a mask, all GENEs of DNA that are not used are cleared. You can have multiple masks for different programs. An awesome feature-creeper would be to figure out mask automatically during the program run (program explicitly marks sequences what it reads and what it skips).

Property Three: Non-linear Option Identification

Because DNA is a list, you may think that you need to know option position to access it. This is not necessary. Index in a DNA is the most straightforward way to lookup an option value, and mask is a tool to help explore and identify its position. But the true power comes from the fact that a option can be identified by just analyzing content. Option can be a single GENE in specific format, or it can be identified as a sequence of GENEs that pass some validation logic. This makes is possible to write generic configuration analyzers, which can read these sequences to detect configuration problems, patterns and meaningful values to produce nice visualization out of them.

Property Four: Application Types

In real world™ DNA is packed in chromosomes. Humans have 46. Potatoes possess 48. Zombies are somewhere in between. In a virtual world of coding it makes sense to use one chromosome per application type. For example, every Python web framework has a lot of common parts, common features and minor details in these features, so a variety of such frameworks can be encoded with a single DNA strand.

Exercise

It may sound too abstract, but try to think how to build a shared DNA strand for two or more mini web application frameworks in Python. Just remember that DNA code includes both hardcoded application features and modifiable configuration options. And as usual look at BoxCar 2D for an inspiration.