Sunday, August 16, 2015

Technical Debts for Python Community

Debts and credits is a new age slavery. Technical debts are better, because they are just a disease. It can fatal if not treated, but for a strong community it is not a problem. Still a good working plan and a vague roadmap is needed to coordinate many eyes to go in the same direction and push things forward where it is hard to reach a consensus.

Dangers of Technical Debts

Technical debt is hard to understand and identify, but once your know about it, it will be easy to spot those things early. The most common symptom of technical debt is stolen time, which if not treated becomes a paralysis. You may know how your system works, and all the components, and where the files are, but you also have the previous experience about how much time it takes to modify and properly test the system, and ensure that everything is correct, and you just know that you just don't have the time, because your family and friends are missing you. We are all volunteers, etc.

Recipe: Simplify Your Systems, Reduce the Complexity, Automate Things and Think in 15 Minutes Slots.

Thinking in 15 Minutes Slots

This may not be so important for enterprise projects where people exchange their time for money, but my opinion is that it is extremely important for open source projects where time is distributed over many people.

Have you ever wondered what will a highly talented person be able to do for Python if we are lucky enough to get 15 minutes of his or her attention?

I am afraid that the sole "contribution" would be making a checkout. Even correcting a mistake in documentation requires that. Maybe the most he or she could accomplish would end in setup of Git or Mercurial. And if the checkout is a long and boring, the person may lose interest and switch to something different even before 15 minutes are expired.

I will continue with how technical debts are evolving into Competence Debt,  but first.. let me take a selfie tell a story. Just to complete the 15min section, I once accepted a challenge to make a design fix for Launchpad. I spent 15 minutes 4 times - an hour - and couldn't even get the checkout. Here are the sessions:

FAIL: edit wiki - move bzr instructions to the top
send a letter to that wiki pages are not editable
Yes, I got distracted during the first session, but I want to make the world better by fixing things on the way. Of course, I'd like those problem not to appear in the first place. Let's see how a next session ended.

# create LXD container for experiments
$ lxc init ubuntu lp
$ lxc start lp
$ lxc exec lp -- bash
# apt-get install bzr
# bzr branch lp:launchpad

The second slot was all about reading instructions and setting up "virtualenv for Linux" to install all the prerequisites without polluting my main system (and drop them without consequences). I already knew about LXD, so my competence here was already high to save some time on learning that. BTW, LXD rocks. Just try it.

# cd launchpad
# apt-get install postgresql make
# ./utilities/launchpad-database-setup $USER
# make schema
FAIL: many errors
# utilities/update-sourcecode

These 15 minutes left me in confusing state without any working instance to get some positive feedback on what I am doing. At this moment I already have a strong desire to just drop everything. And yet after some time I get back to spend another 15 minutes slot trying to tackle the problem.

# drop old LXD container
lxc delete lp
# create new LXD container
lxc init ubuntu lp
lxc start lp
lxc exec lp -- bash
# install basic dependencies
apt-get install bzr make postgresql wget
bash rocketfuel-setup
# ^ need to enter name, and click Y on Apache install prompt
  Making local branch of Launchpad trunk, this may take a while...
  You have not informed bzr of your Launchpad ID, and you must do this to
  write to Launchpad or access private data.  See "bzr help launchpad-login".
  bzr: ERROR: Connection error: Couldn't resolve host '' [Errno -2] Name or service not known
  ERROR: Unable to create local copy of Rocketfuel trunk
# attempt to repeat the script
bash rocketfuel-setup
  bzr: ERROR: No WorkingTree exists for "file:///root/launchpad/lp-branches/devel/.bzr/checkout/".
  ERROR: Your trunk branch in /root/launchpad/lp-branches/devel is corrupted.
         Please delete /root/launchpad/lp-branches/devel and run rocketfuel-setup again.

Now I really drop it. So after that user experience I doubt I will ever get hacking on LaunchPad again. So, if your technical debt provides poor experience for onboarding users, you're likely to lose them for a lifetime.

The Debt of Competence

Competence Debt is a chronic phase of the Technical Debt illness. When time for operation become more than your daily limit, there comes paralysis, and after that the worst thing that can happen is when you no longer know how your systems work and lack skills to restore the picture.

From that moment your project is entering the death spiral and it is only a matter of time when it will be dead. I've seen several examples where programmers was treated like a replaceable material, but the truth is that program lives as long as its code is alive in the heads of its maintainers. There is no such thing as "a software product" anymore - software is more about support and development, than about selling products on a local market.

For open source projects competence debt usually results in various rewrites and long term stale issues. Many attempts to fix them, many hours wasted just to hit the wall with new heads over and over. The power of open source is a little time and little effort that is distributed over many people to create momentum. That was the original idea behind the domain when it all started many years ago. And it is also the result of OpenStreetMap success - small and clear activities that don't require much time and competence to accomplish. This scales well and provides a good gameplay.

Recipe: Invest in Visualization and Learn Visual Tools (SVG, D3.js, Inkscape) to Explore Ways to Transfer Your Competence to Other People

Text is not a natural way for people to consume and produce information. We learn how to read and write, and it takes more than a month to get used to it. But learning to play games like World of Tanks just takes a few minutes. The new generation that I am a part of is used to watch YouTube lectures on 2x speed, read only first 150 letters of the messages and scan long texts without actually thoroughly reading them. That's why I highlight the key points in this post. We were developing tools for audio/visual communication naturally over all these many years - 3D graphics, demoscene, virtual reality, and now deep learning networks, but we still find it hard to produce visual material for communicate other ideas. Because we've been taught to write text, not to produce beautiful art that just works. Learn to draw. It makes people happy learning something new.

OpenStreetMap - The Earth as The Outline

Here is a success story. No, it won't teach how to remove technical debts, but rather give an idea how to restructure it, so that a thousand eyes could make an impact.

The OpenStreetMap has a reference model - it is our Earth. We just copy what we see into vector form to draw a map and everybody could validate with their own eyes. With open source project it is all the same, except that you need to create that reference model and that should be actionable -  split into many pieces that people can validate in parallel separately. Think about specification where every is independent enough to serve both as an entrypoint and a clause to put a checkmark next to it if a condition is true. Think about a canvas, where everybody can draw the common vision and then see who has drawn the components that they can reuse in their own sketches. The reference model is what you need to know where to push so that your small effort could contribute to a greater goal. The Roadmap also tell you where your skills will be more useful, and ensures that your efforts will not be wasted

The Role of Foundations

People think of foundation as of fund. That's not effective. I need about $600/m to cover shelter and food expenses. Travelling, buying clothes and stuff, covering medical expenses raises the plank to $1000/m, girlfriend may add another $500/m, building a house another $500/m, and I don't even want to think about children. There is no chance I will be able to afford this. So at a bare minimum a foundation should provide $600/m per person to deal with EPIC issues that nobody could deal with in their spare time. Forgot the taxes. Add another $600/m on top of that, and that's just one person, and you need at least two of them. So, $2400/m just to make router for so that we can add more URLs endpoints via extensions for interactive frontends and  REST API to it. Nobody will ever pay for that. We tried to hack the problem with Gratipay, but got a flashback from a protection mechanism of U.S. financial system. It is clearly a dead end to fight the World owned by corporations (the World as it was already 100 years ago).

Instead, the role of the foundation is to explain to corporation the above mathematics of time and effort, to enable people in these companies give more time to contribute their professional expertise to deal with complexity and reduce competence debt for the community. Applying the recipes to reduce the technical debt to inspire people. Employing art for documenting the systems and structures, so that people could digest the information easily. Organising in-house sprints to deal with important matters we alone, with less time, but many hands can not tackle on our own.

The role of foundations is not to empower individuals, but collect the data about obstacles, foresee and communicate about them on the path ahead, and organize clean up efforts where they are needed, so that anybody who got those precious 15 minutes knew what to do, and could spend those 15min most effectively to bring their contribution to the common stream that benefits everyone.