tag:blogger.com,1999:blog-16230597075009760082024-03-12T17:20:31.864-07:00another day another vice another roll of the dicehacks, diffs and problemsanatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.comBlogger106125tag:blogger.com,1999:blog-1623059707500976008.post-60688837300571733852019-03-08T10:20:00.000-08:002019-03-08T10:20:37.259-08:00Please protect the right to create public domain works<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://snapcraft.io/">snap</a> is a package format for faster release cycle on Ubuntu. Recently <a href="https://docs.snapcraft.io/the-snap-format/698">its snapcraft.yml description file</a> has got <b>license</b> field:<br />
<br />
<pre><code class="lang-yaml"># License for the snap content, based on SPDX license expressions.
license: <expression></code></pre>
<br />
Which led me to discover corporate looking <a href="https://spdx.org/">SPDX</a> website that aims to set standard on package metadata. And this metadata <b>doesn't allow public domain</b> code:<br />
<br />
<a href="https://wiki.spdx.org/view/Legal_Team/Decisions/Dealing_with_Public_Domain_within_SPDX_Files">https://wiki.spdx.org/view/Legal_Team/Decisions/Dealing_with_Public_Domain_within_SPDX_Files</a><br />
<br />
I expected that such project would be equally inclusive to protect the rights of people to release their code without copyright regulations. I would expect it to work with corporate and public services to enable this option, clarify its meaning and champion changes in legislation. But instead <b>this Linux Foundation project decides that public domain code is too bothersome</b> to deal with.<br />
<br />
Most of my Python code is released <a href="https://pypi.org/search/?q=&o=&c=License+%3A%3A+Public+Domain">as public domain</a>. I like the total freedom of not being forced to deal with strings attached, and I want to give this freedom to people who use my code. I don't even want this code to be "my". I do not need copyright at all. Giving credits to people and supporting those who need support - this activity doesn't need copyright. Technology can make it automated. Being present in copyright header doesn't mean that people care. This problem copyright can not fix.<br />
<br />
Copyright was invented before the internet, <a href="https://en.wikipedia.org/wiki/Copyright#History"> 300 hundred years ago</a>, before git and Mercurial, and lawyers are still (ab)using it in their legal warfare, turning it <a href="https://en.wikipedia.org/wiki/The_Internet%27s_Own_Boy">against those who could show</a> how we could use technology to build a better world, with less stress and more transparency. With the <a href="https://www.youtube.com/watch?v=Yl9TVbAal5s">next generation mentality</a> of the internet children, it is possible to create a next generation of ecosystem with cooperative gameplay and better ecological goals. We really need to ask each other personally - "What do you want? What do you need?" - and help each other grow internally - not externally. Collaborate on building better systems, better code, instead of squabbling on what's already been written.<br />
<br />
Starting with myself - What did I want? I wanted recognition. I didn't want money.<b> Copyright didn't help me to get recognition.</b>
Recognition came from my social activity, from interacting with
people, building open source, collaborating together, trying to solve some problems in a better way. Many years later recognition came from inside. From mindfulness, and that was a huge relief - not trying to compete with people and show off to seem better on outside. <b>Copyright didn't help me to get money</b> - they came from solving problems for other people without adding new ones. Money came as a result of interacting with others over the things that I found challenging and interesting enough.<br />
<br />
I've been paid money, because I can not survive without them, and my
network was responsive to my need. I didn't write CV letters or applied to job fairs or went by advertisement in factories. Once I've become transparent to the network with open hands to be helpful, the money found their flow to compensate me for my time. In seven years I signed only one legal paper, one contract with NDA, and that was the only case when I was not paid after finishing the job. I don't want to live in your legal network - I want to be left in mine. In the network where we transparent and don't attack each other with long legal "electronic documents" that are just a scanned paper. I want to live in a netowrk where my will is a signed data structure on a public blockchain, where people are connected and understand what being a human is not easy, and appreciate our nature and support each other towards greater goal.<br />
<br />
I don't mind people playing the copyright game. People need games to play. But I do not approve it. Suing programmers for their pet projects that got more popular than mundane office project, taking money for access to scientific papers and not paying scientists who produced them. I am tired, I am scared, because I am paranoid, and I want to opt-out from this system.<br />
<br />
<b>I needed support </b>and I joined the Gratipay project. In Gratipay I didn't do much, but in the process <b>I learned to support other people</b>. People whose code I like, people I admire, or people who experienced difficulties, or just people who <a href="http://paulgraham.com/pypar.html">know how to have fun</a>. I started contributing to Gratipay, because it was public domain and it was completely transparent down to the last penny - I could leave at any point and use the code that I learned in my other projects. I tried to make it better and contribute back. I learned a lot and the more questions I asked, the more interesting things I learned. About people, about economics, about financial gameplay. That was an awesome time and a great experience. I wish that everybody in their live could join a project like this. Helping completely open and transparent company. Gratipay could not survive in U.S., and now its public domain code is serving Liberapay in France.<br />
<br />
Public domain was the only way to opt-out from copyright, and with the decision to abandon the fight for it, Linux Foundation closes the door for me to share my code with no strings attached. Treating public domain works "the same way as any license" is the worst decision the SPDX project could take. <b>Public domain is in the "conflict of interests" zone of copyright lawyers</b>, because it is about ignoring their domain, about the right not to care about their business and not being touched by their activities. The task for protecting public domain is for true <b>legal hackers</b>, the ones who go beyond compliance to see the people on the other end. Because there is always a person behind Twitter handle or GitHub account and the person means a little bit more than a "copyright holder".<br />
<br />
SPDX raised an important point - <b>Public Domain is not copyright</b>. The right programmatic solution is it to add a separate <b>copyright-optout: true</b> field that is mutually exclusive with license, because this semantic is completely different from the paper based kludges and cargo cults of the past. It cleans up the space for the next generation of internet connected people to ask questions - how do we give credits and how do we support other people in the age of technology.<br />
<br />
Please fight for Public Domain - it may appear more important for positive development of technology than we currently see, and it may provide a foundation for better bringing us together.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com2tag:blogger.com,1999:blog-1623059707500976008.post-20788163137700682232017-05-17T00:28:00.000-07:002017-05-17T00:57:12.958-07:00Why Git suxx<div dir="ltr" style="text-align: left;" trbidi="on">
<pre class="prettyprint">$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
</pre>
<pre class="prettyprint">$ git checkout https://github.com/gratipay/grtp.co/pull/178
Checkout aborted
There's no file https:\github.com\gratipay\grtp.co\pull\178 at HEAD
</pre>
<pre class="prettyprint">$ git switch https://github.com/gratipay/grtp.co/pull/178
git: 'switch' is not a git command. See 'git --help'.</pre>
<h4 style="text-align: left;">
Solution</h4>
Download <a href="https://github.com/tj/git-extras/blob/master/bin/git-pr">https://github.com/tj/git-extras/blob/master/bin/git-pr</a> to your <span style="background-color: #cccccc; font-family: Courier New, Courier, monospace;">git --exec-path</span>.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-12275518394420019022016-11-02T05:23:00.000-07:002016-11-02T05:24:15.157-07:00Python Usability Bugs: Formatting argparse subcommands<div dir="ltr" style="text-align: left;" trbidi="on">Suppose you want to build a tool with a simple interface:<br />
<pre class="prettyprint">usage: sub <command>
commands:
status - show status
list - print list
</pre>Python proposes to use <a href="https://docs.python.org/3/library/argparse.html">argparse</a> module. And if you follow documentation, the best you can get will be this output:<br />
<pre class="prettyprint">usage: sub {status,list} ...
positional arguments:
{status,list}
status show status
list print list
</pre><div>And it you implement proper formatting, your code will look like this:</div><pre class="prettyprint">import argparse
import sys
class CustomHelpFormatter(argparse.HelpFormatter):
def _format_action(self, action):
if type(action) == argparse._SubParsersAction:
# inject new class variable for subcommand formatting
subactions = action._get_subactions()
invocations = [self._format_action_invocation(a) for a in subactions]
self._subcommand_max_length = max(len(i) for i in invocations)
if type(action) == argparse._SubParsersAction._ChoicesPseudoAction:
# format subcommand help line
subcommand = self._format_action_invocation(action) # type: str
width = self._subcommand_max_length
help_text = ""
if action.help:
help_text = self._expand_help(action)
return " {:{width}} - {}\n".format(subcommand, help_text, width=width)
elif type(action) == argparse._SubParsersAction:
# process subcommand help section
msg = '\n'
for subaction in action._get_subactions():
msg += self._format_action(subaction)
return msg
else:
return super(CustomHelpFormatter, self)._format_action(action)
def check():
print("status")
return 0
parser = argparse.ArgumentParser(usage="sub <command>", add_help=False,
formatter_class=CustomHelpFormatter)
subparser = parser.add_subparsers(dest="cmd")
subparser.add_parser('status', help='show status')
subparser.add_parser('list', help='print list')
# custom help messge
parser._positionals.title = "commands"
# hack to show help when no arguments supplied
if len(sys.argv) == 1:
parser.print_help()
sys.exit(0)
args = parser.parse_args()
if args.cmd == 'list':
print('list')
elif args.cmd == 'status':
sys.exit(check())
</pre><div>Here you may see the failure of OOP (object oriented programming). The proper answer to this formatting problem is just to define a data structure for command line help in JSON or similar format and let people dump and process it with templates. Once option definition is parsed, the information there is static, so there is no need in those intertwined recursive method calls. So just do it in 2 pass - get dataset and render template.</div></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-13275723682174531062016-09-17T11:46:00.000-07:002016-09-17T11:46:48.601-07:00Python Usability Bugs: subprocess.Popen executable<div dir="ltr" style="text-align: left;" trbidi="on"><a href="https://docs.python.org/2/library/subprocess.html#subprocess.Popen">subprocess.Popen</a> seems to be designed as a "swiss army knife" of managing external processes, and while the task is pretty hard to solve in cross-platform way, it seems the people who have contributed to it did manage to achieve that. But it still came with some drawbacks and complications. Let's study one of these that I think is a top one from usability point of view, because it confuses people a lot.<br />
<br />
I've got a simple <a href="https://gist.github.com/techtonik/98ff54ef262f88bf98f29b0cd41890d8">program</a> that prints its name and own arguments (forgive me for Windows code, as I was debugging the issue on Windows, but this works the same on Linux too). The program is written in Go to get single executable, because subprocess has special handling for child Python processes (another usability bug for another time).<br />
<pre class="prettyprint">>argi.exe 1 2 3 4
prog: E:\argi.exe
args: [1 2 3 4]
</pre><div>Let's execute it with <b>subprocess.Popen</b>, and for that I almost always look up the official documentation for <a href="https://docs.python.org/2/library/subprocess.html#subprocess.Popen">Popen prototype</a>:<br />
<blockquote class="tr_bq"> <code class="descclassname" style="font-size: 0.95em; padding: 0px">subprocess.</code><code class="descname" style="font-size: 1.2em; font-weight: bold; padding: 0px 1px;">Popen</code><span class="sig-paren" style="font-family: sans-serif; font-size: 16px;">(</span><em style="font-family: sans-serif; font-size: 16px;">args</em><span style="background-color: white;"><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">bufsize=0</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">executable=None</em>, <em style="font-family: sans-serif; font-size: 16px;">stdin=None</em><span style="font-family: sans-serif; font-size: 16px;">,</span></span><span style="font-family: sans-serif; font-size: 16px;"> </span><em style="font-family: sans-serif; font-size: 16px;">stdout=None</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">stderr=None</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">preexec_fn=None</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">close_fds=False</em>, <em style="font-family: sans-serif; font-size: 16px;">shell=False</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">cwd=None</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">env=None</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">universal_newlines=False</em>, <em style="font-family: sans-serif; font-size: 16px;">startupinfo=None</em><span style="font-family: sans-serif; font-size: 16px;">, </span><em style="font-family: sans-serif; font-size: 16px;">creationflags=0</em><span class="sig-paren" style="font-family: sans-serif; font-size: 16px;">)</span></blockquote>Quite scary, right? But let's skip confusing part and quickly figure out something out of it (because time is scarce). Looks like this should do the trick:</div><pre class="prettyprint">import subprocess
args = "1 2 3 4".split()
p = subprocess.Popen(args, executable="argi.exe")
p.communicate()
</pre>After saving this code to "subs.py" and running it, you'd probably expect something like this :<br />
<pre class="prettyprint">> python subs.py
prog: E:\argi.exe
args: [1 2 3 4]
</pre><div>And... you won't get this. What you get is this:</div><pre class="prettyprint">> python subs.py
prog: 1
args: [2 3 4]
</pre><div><div>And that's kind of crazy - not only the executable was renamed, but the first argument was lost, and it appears that this is actually a documented behavior. So let's define <b>Python Usability Bug</b> as <b>something that is documented but not expected</b> (by most folks who is going to read the code). The trick to get code do what is expected is <b>never use executable</b> argument to <b>subprocess.Popen</b>:</div><pre class="prettyprint">import subprocess
args = "1 2 3 4".split()
args.insert(0, "argi.exe")
p = subprocess.Popen(args)
p.communicate()
</pre><pre class="prettyprint">>python suby.py
prog: argi.exe
args: [1 2 3 4]
</pre><div>The explanation for former "misbehavior" is that <b>executable is a hack</b> that allows to rename program when running subprocess. It should be named <b>substitute</b>, or - even better - <b>altname</b> to work as an alternative name to pass to child process (instead of providing alternative executable for the former name). To make <b>subprocess.Popen</b> even more intuitive, the <b>args</b> argument should have been named <b>command</b>.</div></div><div><br />
</div>From the high level design point of view, the drawbacks of this function is that it *does way too much*, its arguments are not always intuitive - it takes *a lot of time to grok official docs*, and I need to read it *every time*, because there are too many little important details of <b>Popen</b> behavior (have anybody tried to create its state machine?), so over the last 5 years I still discover various problems with it. Today I just wanted to save you some hours that I've wasted myself while debugging <a href="https://github.com/mozilla/pymake">pymake</a> on Windows.<br />
<div><br />
</div><div>That's it for now. Bonus points to update this post with link when I get more time / mana for it:</div><br />
<ul style="text-align: left;"><li>[ ] people who have contributed to it</li>
<li>[ ] it came with drawbacks</li>
<li>[ ] have anybody tried to create its state machine?</li>
<li>[ ] subprocess has special handling for child Python processes</li>
</ul></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-82871713277325570872016-06-16T07:29:00.001-07:002016-06-16T07:29:45.092-07:00Why Python community should take UX seriously<div dir="ltr" style="text-align: left;" trbidi="on">
UX is a discipline that is dedicated to measuring user emotions and satisfaction while using a product in order to improve them. When you grow experience in any programming language you start to experience negative emotions towards small warts and glitches that constantly get in your way. You compare it to different languages and soon you're no more satisfied with it. Backward compatibility etc. doesn't allow to change things, but they should be changed, and UX is the criteria and method to answer <b>how</b>. But <b>why</b>?<br />
<br />
People are dying too quickly. The process of learning new technology is fun and exciting, but when you look at the ideas in the past you'll see that we are not inventing anything new. People thought about many ideas, and couldn't implement them. I still don't have automation line that pours me coffee in the morning and bakes omelet and this idea is like 20 years old. I have no information about where my products are coming from - this idea is about 10 years old, and those fruits that are coming from hydroponics and pretty tasteless, so I'd prefer to buy something bathed in organic sun. Where are those inventions now? I am pretty sure somebody is trying to design these things in own basement, but those inventions will never see the light, because they are fragile, needs a lot of experience and education. Why learning is hard? It needs a lot of experience and education to learn concepts, operating systems internals and get used to various glitches and workarounds from the past. If Python was easier, if operating systems were more easy to learn and intuitively understand, not relying on elitist part of hacker culture, we could hack on more fun and reliable things today.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-16702280622586129862016-06-07T00:00:00.000-07:002016-06-07T00:00:09.189-07:00Why Roundup needs a router?<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://bugs.python.org/">http://bugs.python.org/</a> is
powered by ancient software. Most people nowadays won't install it. They
would rather find something newer, with more features (consumers) or
invent something from scratch that will be more modern from the start
(makers).<br />
<br />
How about trying a different approach - how about an approach that means <b>constant evolution</b>
of the software? The reason why people reinvent things from scratch is
because they often can not understand the complexity of what was already
written. The burden of complexity is like tar that glues us to the
surface, needs more force to take a step and slows us down. One way to
get rid of it is to start again clean. But there is no guarantee that
the same story won't happen if you couldn't see past complexity before.<br />
<br />
A way of evolution is to face the complexity and deal with it. The way to deal with complexity tar is to learn to fly over it. <b>Visualization</b>, <b>art</b>, <b>story-telling</b>
are helping to people to overcome limitations of our attention,
interest, motivation and focus. There are tools to make complex things
simple, there are things to look at the surface from above and choose a
path that is still unexplored.<br />
<br />
<b>Router</b> is a component that directs URL request to processing. Router says that <span style="font-family: "Courier New",Courier,monospace;">http://../file142</span> should be processed by function that handles files, and <span style="font-family: "Courier New",Courier,monospace;">http://../issue42</span>
should be handled by function that processes issues. This logic is
hardcoded into Roundup, so its URL structure doesn't change. If Roundup
had this router, then it could be used as a replacement for existing
trackers <a href="https://www.w3.org/Provider/Style/URI.html">without breaking links</a>.
Router also allows Roundup extensions add new endpoints, such as ones
required for OAuth2 processing or REST and ajax interface.<br />
<br />
So, <b>how to evolve software</b>?
It takes media and coordination skills first, and code review and
design second. Additional code that makes code more complex should be
backed up by documentation that explains it, every lengthy documentation
that gives details should be accompanied by concise overview and
reminder picture, every yak shaving process should be automated, every
upstream tools need to be improved. Evolution is a global process, and it is
hard (if not impossible) to evolve one agent if the whole ecosystem is not changing.
Changing ecosystem or tools that are already stable like Python require
process shift that drastically changes how people coordinate,
approve, mutate and select the branches with features that comfort them.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-77426962202253980532016-05-11T04:57:00.001-07:002016-05-11T04:57:50.665-07:00Open Source, Death, and Open Economy<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Open Source</b> is a great idea, great movement and a great boost to the progress of civilization. <b>Open Data</b> made transparent governments possible, which allowed people to regulate tough problems, not hide them. <b>Open systems</b> like Linux and Open Stack enabled building more useful software on top of more solid and stable basis (first for single operating system nodes and now for clouds and networks).<br />
<br />
So why Open Source is dead? Because there is no such thing as software product - any software is alive as long as its code is live in the heads of its maintainers. As soon as maintainers leave, the product enters death spiral - new people enter the field to patch the product to the death. Open Source is dead, because maintaining Open Source software requires time, and every minute of that time you need to pay for food, for shelter, for the privilege to meet with your friends and be inspired. Open Source maintenance does not release maintainers from paying for all this stuff and doesn't bring them those payment tokens called `money`. So code in maintainers heads will have to die. <br />
<br />
How come that current economy is so flawed? Does Open Source produce <b>value</b>? Definitely, yes. Does it solve problems for people? Yes. Do people feel grateful for Open Source? Well, sometimes (but that's also the question of consumer culture forced by current economy). And still open source developers leave their products to join (often meaningless) activities of reinventing things from scratch for other projects. Often doing things that they've paid for and not things they know and like.<br />
<br />
How come that people with developed skills doing useful things have nothing to eat, and for food and shelter have to reformat their brain and became compliant with office slavery? Sometimes I feel like we should extend <a href="https://sfconservancy.org/">software freedom conservancy</a> model from software to people and remove "people with money" from power part of this equation. Blockchain allows us to calculate personal values to remove this money-for-money gameplay. Gameplay that creates parasites out of people, parasites that are motivated to produce garbage, to consume and to battle each other for "market domination". Parasites don't like to be exposed, and they don't like to be parasites, so maybe <b>Open Economy</b> is an answer that can bring our focus back to the planet with depleting resources and increasing pollution that pushes our health and general feeling of the world down to the level of committing a suicide.<br />
<br />
I'd like to believe, but I am loosing my insight sometimes. There are many good people who care about me in this world, and this letter won't be there if not them. But their lifetime is limited, and so is mine. I really wanted to boost progress to help with my health and global problems, but it seems I failed. This post is filled under a Python tag, which always was a part of that unconscious plan. I thought that I could enhance it to be an easier, and more intuitive tool to augment limited human abilities for automation, to help them concentrate on more important things. But it seems I failed. I still see things how they should be done, I still see that DX (developer's experience) need to raise to the top of people priorities. It is just that I am exhausted to compete for food and shelter and feel depressed over the need to reformat my natural intelligence network to do those (often meaningless) office things just to keep my body alive.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com1tag:blogger.com,1999:blog-1623059707500976008.post-68652484947326248162016-01-10T01:27:00.000-08:002016-01-10T11:23:45.788-08:00Reaction to Python development moving to GitHub<div dir="ltr" style="text-align: left;" trbidi="on">
If you don't know, the Python core developers <a href="https://mail.python.org/pipermail/core-workflow/2016-January/000345.html">are considering move to GitHub</a>, which I believe will abandon Roundup, Mercurial and a couple of other initiatives that some of us picked, because they were used by community and were useful to it. For example I added support for Jinja2 to Roundup, tried to add a new router component for it to enable adding new web APIs more easily, and now I am doing a release of new version, because that package is valuable.<br />
<br />
Below is my post sent to <a href="https://mail.python.org/mailman/listinfo/core-workflow">https://mail.python.org/mailman/listinfo/core-workflow</a> as an immediate reaction to the PEP that addresses the migration. I think that people need to consider Python as something with much better potential than just "a product to execute computer scripts". I want people to think about it as about <b>open process with all the involved complexity</b> and dynamics issues, a place to study and learn about different culture than following the orders of your <strike>owner</strike> founder. Some want to see it to be isolated coding playground without all that disturbances and annoying people with external world problems. But it is unavoidable. There are already diversity statement and codes of conducts. It is time to stop being ignorant to economics for the balance.<br />
<br />
Blender is a good example. 3D editor that every two years chooses a problem that is actual for community and makes a movie <b>together with building necessary features</b> in the editor. They discuss economics, try to address funding issues by <b>working with funds</b> outside of coding domain, bringing new initiatives, communicating and still staying open - not ignoring the issues and not trying to bring the same hammer on the new "nail". Why Python fails to follow the same route?<br />
<br />
I don't know why it bother me that much. Probably because I am jobless and living in northern country where you can't live on street and need to pay rent, so "getting paid for the useful stuff that you do" is extremely important issue for me. But also I feel that my time is limited, and that it is very sad to spend my life behind the screen of computer, reinventing some code that was written numerous time for a company that can not support something that is already written.<br />
<br />
I realize that with reduced exposure to a projects like Roundup and Mercurial it will be much harder (if possible at all) to <b>gather critical mass of resources around people to experiment with sustainability models for open source</b>, and doing these things is important, because we coming into the age of extremely powerful automation where people may not be that needed anymore. For me open source is the only way to gather data on what will happen, and maybe the only way to find the balance. <b>Open source economics</b> right now is when resources are limited, but there are many people how could use them, but if they use them, others (who probably can do better) may feel bad about this. This is a just a single problem that needs to be addressed in the process, and I wish that there could be that sustainable working process in the list.<br />
<br />
Here is original mail sent:<br />
<br />
<br />
<a name='more'></a><br />
<br />
<br />
I think that the missing part is the analysis of why community (or PSF, or ...)<br />
failed to create a streamlined development process themselves. In particular,<br />
what steps were made to implement:<br />
<br />
1. online code editing with patch creating through hg.python.org interface<br />
2. HG plugin that can fetch a bugXXX patch and apply it to local copy<br />
3. mercurial queue server to allow people to maintain their own queues of<br />
patches in parallel and compare between them<br />
4. what steps were made to make "our fork of the Rietveld code review tool"<br />
to be the "our installation" or the "global Roundup service"<br />
5. create testing and building infrastructure (or integrating with CI services<br />
such as Drone.IO) for downloading patches, making sure they applied<br />
cleanly and running the test suite<br />
<br />
For simple things like "spelling in documentation" the whole thing with<br />
patch production and attachment could be done in 6 months provided that<br />
development for 4 people is funded (2 code, 1 frontend, 1 art).<br />
<br />
<br />
The most important stuff - what are the current activities for PSF (or whatever)<br />
to being raise funds to make it a paid work instead of relying on a really few<br />
core developers and volunteers to do professional work on infra that requires<br />
rather tight timely coordination and support?<br />
<br />
For me, there is a more core problem inside. For example, why work under<br />
GSoC 2015 for Roundup was not submitted upstream? Because it was not<br />
part of the job. People have problems allocating their free time, because<br />
time management practices at their HR departments are getting better and<br />
better not leaving much for contributions.<br />
<br />
I think that the core issue here is the money. The new generation talks<br />
about startups and monetization, so if we don't address this, there is little<br />
hope that new generation will be attracted, at least that's my observation.<br />
<br />
I think that if PSF can help us with legal issues concerning funding open<br />
source activities, we can construct a few teams through Gratipay and do<br />
the development in open way. It will be more effective for Python in the<br />
long run, because it will showcase that Python is capable of, and provide<br />
people a playground to learn about good engineering practices. I am not<br />
saying that this will 100% work - maybe there are things ahead that I do not<br />
see as well, but what's drives me mad that it seems that nobody is even<br />
trying to preserve the open source spirit in all of that.<br />
<br />
If we will be sacrificing the open source so easily, then the next Python with<br />
static typing will be 'Microsoft Python' or 'Facebook Python' will be rewritten<br />
from scratch, and all credits gathered through all these years will be lost<br />
behind the shiny name of another corporation. Sad.<br />
<br /></div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-3773854869877328862015-09-03T10:00:00.000-07:002015-09-03T10:03:28.179-07:00SCons build targets<div dir="ltr" style="text-align: left;" trbidi="on">SCons is awesome. Just saying. If you want to know (or troubleshoot) how SCons selects targets to be built, add this snippet at the end of your SConstruct:<br />
<pre class="prettyprint">def dump_targets(targets):
for t in targets:
if type(t) == str:
name = t
else:
name = t.name
print(" <" + str(t.__class__.__name__) + "> " + name)
print("[*] Default targets:")
dump_targets(DEFAULT_TARGETS)
print("[*] Command line targets:")
dump_targets(COMMAND_LINE_TARGETS)
<div>print("[*] All build targets:")
dump_targets(BUILD_TARGETS)
</pre></div><div>For my copy of <a href="https://github.com/wesnoth/wesnoth">Wesnoth</a>, 'scons .' produces this output:</div><pre class="prettyprint">[*] Default targets:
<Alias> wesnoth
<Alias> wesnothd
[*] Command line targets:
<str> .
[*] All build targets:
<str> .</pre><div>And if you want to know how to specify targets or what do they mean, read the second page of <a href="http://www.scons.org/doc/HTML/scons-man.html">SCons man documentation</a>. Just for convenience I quote it here.</div><div><br />
<hr/><br />
</div><div><div><b>scons</b> is normally executed in a top-level directory containing a <b>SConstruct</b> file, optionally specifying as command-line arguments the target file or files to be built.</div><div><br />
</div><div>By default, the command</div><pre class="prettyprint">scons</pre><div>will build all target files in or below the current directory. Explicit default targets (to be built when no targets are specified on the command line) may be defined the <b>SConscript</b> file(s) using the <b>Default()</b> function, described below.</div><div><br />
</div><div>Even when <b>Default()</b> targets are specified in the <b>SConscript</b> file(s), all target files in or below the current directory may be built by explicitly specifying the current directory (<b>.</b>) as a command-line target:</div><pre class="prettyprint">scons .</pre><div>Building all target files, including any files outside of the current directory, may be specified by supplying a command-line target of the root directory (on POSIX systems):</div><pre class="prettyprint">scons /</pre><div>or the path name(s) of the volume(s) in which all the targets should be built (on Windows systems):</div><pre class="prettyprint">scons C:\ D:\</pre><div>To build only specific targets, supply them as command-line arguments:</div><pre class="prettyprint">scons foo bar</pre><div>in which case only the specified targets will be built (along with any derived files on which they depend).</div><div><br />
</div><div>Specifying "cleanup" targets in <b>SConscript</b> files is not usually necessary. The <b>-c flag</b> removes all files necessary to build the specified target:</div><pre class="prettyprint">scons -c .</pre><div>to remove all target files, or:</div><pre class="prettyprint">scons -c build export</pre><div>to remove target files under build and export. Additional files or directories to remove can be specified using the <b>Clean()</b> function. Conversely, targets that would normally be removed by the <b>-c</b> invocation can be prevented from being removed by using the <b>NoClean()</b> function.</div><div><br />
</div><div>A subset of a hierarchical tree may be built by remaining at the top-level directory (where the <b>SConstruct</b> file lives) and specifying the subdirectory as the target to be built:</div><pre class="prettyprint">scons src/subdir</pre><div>or by changing directory and invoking <b>scons</b> with the <b>-u option</b>, which traverses up the directory hierarchy until it finds the <b>SConstruct</b> file, and then builds targets relatively to the current subdirectory:</div><pre class="prettyprint">cd src/subdir
scons -u .
</pre></div></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-19502481304010667102015-08-16T05:05:00.000-07:002015-08-16T05:05:15.088-07:00Technical Debts for Python Community<div dir="ltr" style="text-align: left;" trbidi="on">
Debts and credits is a new age slavery. Technical debts are better, because they are just a disease. It can fatal if not treated, but for a strong community it is not a problem. Still a good working plan and a vague roadmap is needed to coordinate many eyes to go in the same direction and push things forward where it is hard to reach a consensus.<br />
<br />
<h4 style="text-align: left;">
Dangers of Technical Debts</h4>
<br />
Technical debt is hard to understand and identify, but once your know about it, it will be easy to spot those things early. The most common symptom of technical debt is <b>stolen time</b>, which if not treated becomes a <b>paralysis</b>. You may know how your system works, and all the components, and where the files are, but you also have the previous experience about how much time it takes to modify and properly test the system, and ensure that everything is correct, and you just know that you just don't have the time, because your <b>family and friends are missing you</b>. We are all volunteers, etc.<br />
<br />
<b>Recipe:</b> Simplify Your Systems, Reduce the Complexity, Automate Things and Think in 15 Minutes Slots.<br />
<br />
<b>Thinking in 15 Minutes Slots</b><br />
<br />
This may not be so important for enterprise projects where people exchange their time for money, but my opinion is that it is extremely important for open source projects where time is distributed over many people.<br />
<br />
Have you ever wondered what will a highly talented person be able to do for Python if we are lucky enough to get 15 minutes of his or her attention?<br />
<br />
I am afraid that the sole "contribution" would be making a checkout. Even correcting a mistake in documentation requires that. Maybe the most he or she could accomplish would end in setup of Git or Mercurial. And if the checkout is a long and boring, the person may lose interest and switch to something different even before 15 minutes are expired.<br />
<br />
I will continue with how technical debts are evolving into <b>Competence Debt</b>, but first.. let me <strike>take a selfie</strike> tell a story. Just to complete the 15min section, I once accepted a challenge to make a design fix for Launchpad. I spent 15 minutes 4 times - an hour - and couldn't even get the checkout. Here are the sessions:<br />
<br />
<blockquote class="tr_bq">
<div class="" id="magicdomid2">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://launchpad.net/%7Einkscape.dev/+archive/ubuntu/stable">https://launchpad.net/~inkscape.dev/+archive/ubuntu/stable</a></span><span class="author-g-daeaqwpt6kjfxjf0"> - fix dialog here</span></span></div>
<div class="" id="magicdomid3">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">open </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://dev.launchpad.net/">https://dev.launchpad.net/</a></span></span></div>
<div class="" id="magicdomid4">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">open </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://dev.launchpad.net/Getting">https://dev.launchpad.net/Getting</a></span></span></div>
<div class="" id="magicdomid5">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">FAIL: edit wiki - move bzr instructions to the top</span></span></div>
<div class="" id="magicdomid6">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">send a letter to </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://lists.launchpad.net/launchpad-dev/">https://lists.launchpad.net/launchpad-dev/</a></span><span class="author-g-daeaqwpt6kjfxjf0"> that wiki pages are not editable</span></span></div>
</blockquote>
Yes, I got distracted during the first session, but I want to make the world better by fixing things on the way. Of course, I'd like those problem not to appear in the first place. Let's see how a next session ended.<br />
<br />
<blockquote class="tr_bq">
<div class="" id="magicdomid2">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">open </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://dev.launchpad.net/Getting">https://dev.launchpad.net/Getting</a></span></span></div>
<div class="" id="magicdomid3">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># create LXD container for experiments</span></span></div>
<div class="" id="magicdomid4">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">$ lxc init ubuntu lp</span></span></div>
<div class="" id="magicdomid5">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">$ lxc start lp</span></span></div>
<div class="" id="magicdomid6">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">$ lxc exec lp -- bash</span></span></div>
<div class="" id="magicdomid7">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># apt-get install bzr</span></span></div>
<div class="" id="magicdomid8">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># bzr branch lp:launchpad</span></span></div>
<div class="" id="magicdomid9">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">open </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://dev.launchpad.net/Running">https://dev.launchpad.net/Running</a></span></span></div>
</blockquote>
<br />
The second slot was all about reading instructions and setting up "virtualenv for Linux" to install all the prerequisites without polluting my main system (and drop them without consequences). I already knew about LXD, so my competence here was already high to save some time on learning that. BTW, <a href="https://linuxcontainers.org/lxd/getting-started-cli/">LXD rocks</a>. Just try it.<br />
<br />
<blockquote class="tr_bq">
<div class="" id="magicdomid2">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># cd launchpad</span></span></div>
<div class="" id="magicdomid3">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># apt-get install postgresql make</span></span></div>
<div class="" id="magicdomid4">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># ./utilities/launchpad-database-setup $USER</span></span></div>
<div class="" id="magicdomid5">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># make schema</span></span></div>
<div class="" id="magicdomid6">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">FAIL: many errors</span></span></div>
<div class="" id="magicdomid7">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">reading </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://dev.launchpad.net/Running">https://dev.launchpad.net/Running</a></span></span></div>
<div class="" id="magicdomid8">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># utilities/update-sourcecode</span></span></div>
<div class="" id="magicdomid9">
<br /></div>
</blockquote>
These 15 minutes left me in confusing state without any working instance to get some positive feedback on what I am doing. At this moment I already have a strong desire to just drop everything. And yet after some time I get back to spend another 15 minutes slot trying to tackle the problem.<br />
<br />
<blockquote class="tr_bq">
<div class="" id="magicdomid2">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># drop old LXD container</span></span></div>
<div class="" id="magicdomid3">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">lxc delete lp</span></span></div>
<div class="" id="magicdomid4">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># create new LXD container</span></span></div>
<div class="" id="magicdomid5">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">lxc init ubuntu lp</span></span></div>
<div class="" id="magicdomid6">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">lxc start lp</span></span></div>
<div class="" id="magicdomid7">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">lxc exec lp -- bash</span></span></div>
<div class="" id="magicdomid8">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># install basic dependencies</span></span></div>
<div class="" id="magicdomid9">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">apt-get install bzr make postgresql wget</span></span></div>
<div class="" id="magicdomid10">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># talking in </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://lists.launchpad.net/launchpad-dev/">https://lists.launchpad.net/launchpad-dev/</a></span></span></div>
<div class="" id="magicdomid11">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">wget </span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://git.launchpad.net/launchpad/plain/utilities/rocketfuel-setup">https://git.launchpad.net/launchpad/plain/utilities/rocketfuel-setup</a></span></span></div>
<div class="" id="magicdomid12">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">bash rocketfuel-setup</span></span></div>
<div class="" id="magicdomid13">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># ^ need to enter name, and click Y on Apache install prompt</span></span></div>
<div class="" id="magicdomid14">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">FAIL:<br /> </span></span></div>
<div class="" id="magicdomid15">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> Making local branch of Launchpad trunk, this may take a while...</span></span></div>
<div class="" id="magicdomid16">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> You have not informed bzr of your Launchpad ID, and you must do this to</span></span></div>
<div class="" id="magicdomid17">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> write to Launchpad or access private data. See "bzr help launchpad-login".</span></span></div>
<div class="" id="magicdomid18">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> bzr: ERROR: Connection error: Couldn't resolve host 'bazaar.launchpad.net' [Errno -2] Name or service not known</span></span></div>
<div class="" id="magicdomid19">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> ERROR: Unable to create local copy of Rocketfuel trunk</span></span></div>
<div class="" id="magicdomid20">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"># attempt to repeat the script</span></span></div>
<div class="" id="magicdomid21">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">bash rocketfuel-setup</span></span></div>
<div class="" id="magicdomid22">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0">FAIL:</span></span></div>
<div class="" id="magicdomid23">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> bzr: ERROR: No WorkingTree exists for "</span><span class="author-g-daeaqwpt6kjfxjf0 url"><a href="https://draft.blogger.com/null">file:///root/launchpad/lp-branches/devel/.bzr/checkout/</a></span><span class="author-g-daeaqwpt6kjfxjf0">".</span></span></div>
<div class="" id="magicdomid24">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> ERROR: Your trunk branch in /root/launchpad/lp-branches/devel is corrupted.</span></span></div>
<div class="" id="magicdomid25">
<span style="font-size: x-small;"><span class="author-g-daeaqwpt6kjfxjf0"> Please delete /root/launchpad/lp-branches/devel and run rocketfuel-setup again.</span></span></div>
</blockquote>
<br />
Now I really drop it. So after that user experience I doubt I will ever get hacking on LaunchPad again. So, <b>if your technical debt provides poor experience for onboarding users, you're likely to lose them for a lifetime</b>.<br />
<br />
<h3 style="text-align: left;">
The Debt of Competence</h3>
<br />
Competence Debt is a chronic phase of the Technical Debt illness. When time for operation become more than your daily limit, there comes paralysis, and after that the worst thing that can happen <b>is when you no longer know how your systems work</b> and lack skills to restore the picture.<br />
<br />
From that moment your project is entering the <a href="https://en.wikipedia.org/wiki/Graveyard_spiral">death spiral</a> and it is only a matter of time when it will be dead. I've seen several examples where programmers was treated like a replaceable material, but the truth is that program lives as long as its code is alive in the heads of its maintainers. There is no such thing as "a software product" anymore - software is more about support and development, than about selling products on a local market.<br />
<br />
<b>For open source projects</b> competence debt usually results in various rewrites and long term stale issues. Many attempts to fix them, many hours wasted just to hit the wall with new heads over and over. The power of open source is a little time and little effort that is distributed over many people to create momentum. That was the original idea behind the rainforce.org domain when it all started many years ago. And it is also the result of OpenStreetMap success - small and clear activities that don't require much time and competence to accomplish. This scales well and provides a <b>good gameplay</b>.<br />
<br />
<b>Recipe:</b> Invest in Visualization and Learn Visual Tools (SVG, D3.js, Inkscape) to Explore Ways to Transfer Your Competence to Other People<br />
<br />
<b>Text is not a natural way for people to consume and produce information</b>. We learn how to read and write, and it takes more than a month to get used to it. But learning to play games like World of Tanks just takes a few minutes. The new generation that I am a part of is used to watch YouTube lectures on 2x speed, read only first 150 letters of the messages and scan long texts without actually thoroughly reading them. That's why I highlight the key points in this post. We were developing tools for audio/visual communication naturally over all these many years - 3D graphics, demoscene, virtual reality, and now deep learning networks, but we still find it hard to produce visual material for communicate other ideas. Because we've been taught to write text, not to <b>produce beautiful art that just works</b>. Learn to draw. It <a href="https://xkcd.com/353/">makes people happy</a> learning something new.<br />
<br />
<h3 style="text-align: left;">
OpenStreetMap - The Earth as The Outline</h3>
<br />
Here is a success story. No, it won't teach how to remove technical debts, but rather give an idea how to <b>restructure it, so that a thousand eyes could make an impact</b>.<br />
<br />
The OpenStreetMap has a reference model - it is our Earth. We just copy what we see into vector form to draw a map and everybody could <b>validate</b> with their own eyes. With open source project it is all the same, except that you need to create that reference model and that should be <b>actionable</b> - split into many pieces that people can validate in parallel separately. Think about specification where every is independent enough to serve both as an entrypoint and a clause to put a checkmark next to it if a condition is true. Think about a canvas, where everybody can draw the common vision and then see who has drawn the components that they can reuse in their own sketches. The reference model is what you need to know where to push so that your small effort could contribute to a greater goal. <b>The Roadmap</b> also tell you where your skills will be more useful, and <b>ensures</b> that your efforts will not be wasted<br />
<br />
<h3 style="text-align: left;">
The Role of Foundations</h3>
<br />
People think of foundation as of fund. That's not effective. I need about $600/m to cover shelter and food expenses. Travelling, buying clothes and stuff, covering medical expenses raises the plank to $1000/m, girlfriend may add another $500/m, building a house another $500/m, and I don't even want to think about children. There is no chance I will be able to afford this. So at a bare minimum a foundation should provide $600/m per person to deal with <b>EPIC</b> issues that nobody could deal with in their spare time. Forgot the taxes. Add another $600/m on top of that, and that's just one person, and you need at least two of them. So, <b>$2400/m</b> just to make router for <a href="https://bugs.python.org/">https://bugs.python.org/</a> so that we can add more URLs endpoints via extensions for interactive frontends and REST API to it. Nobody will ever pay for that. We tried to hack the problem with Gratipay, but got a flashback from a protection mechanism of U.S. financial system. It is clearly a dead end to fight the World owned by corporations (the World as it was already 100 years ago).<br />
<br />
Instead, the role of the foundation is to explain to corporation the above mathematics of time and effort, to enable people in these companies give more time to contribute their professional expertise to deal with complexity and <b>reduce competence debt for the community</b>. Applying the recipes to reduce the technical debt to inspire people. Employing art for documenting the systems and structures, so that people could digest the information easily. Organising in-house sprints to deal with important matters we alone, with less time, but many hands can not tackle on our own.<br />
<br />
The role of foundations is not to empower individuals, but collect the data about obstacles, foresee and communicate about them on the path ahead, and organize clean up efforts where they are needed, so that anybody who got those precious 15 minutes knew what to do, and could spend those 15min most effectively to bring their contribution to the common stream that benefits everyone.<br />
<br /></div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-27826901585757962802015-06-21T09:38:00.000-07:002015-06-21T09:39:04.125-07:00Lifehack - import directly from command line<div dir="ltr" style="text-align: left;" trbidi="on">If you're obsessed with <a href="https://ux.stackexchange.com/questions?sort=votes">UX</a> as I am, and long repetitive commands seem to break your <a href="http://psygrammer.com/2011/02/10/the-flow-programming-in-ecstasy/">flow</a>, you may appreciate this little trick for invoking Python from command line.<br />
<div><br />
</div>I invoke python from console/terminal to check some behavior that I tend to forget after a day in JavaScript or other programming languages. It is often way faster than <a href="http://pyfound.blogspot.com/2015/05/read-docs-growing-with-little-help-from.html">reading the docs</a>. It also helps to quickly try new PyPI libraries after installing with pip (saves on mouse clicking in IDE etc.).<br />
<br />
Today I found a time saver that makes me happy. This example is typed for Windows, but there should be no problem for any Linux hacker to port it to their favorites. So instead of doing this:<br />
<pre class="prettyprint">> python [ENTER]
>>> import os [ENTER]
>>> os.name [ENTER]
'nt'
</pre>I can just type:<br />
<pre class="prettyprint">> import os [ENTER]
>>> os.name [ENTER]
'nt'
</pre>It should be now easy to understand how it works, so I will complicate it a bit. The trick is this little <b>import.bat</b> script, placed in system PATH:<br />
<pre class="prettyprint">@echo off
py -2 -c "import sys; print(sys.version)"
py -2 -i -c "import %*"
</pre><b>py</b> here is a <a href="https://www.python.org/dev/peps/pep-0397/"><b>py.exe</b> launcher</a> that can call either Python 2 or Python 3 depending on argument. It comes installed with Windows Python versions. I like to know which Python I work with, so I've added version into to the command, and to avoid polluting the global space of new interpreter, the version is printed by separate command.</div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-22061672708177119152015-05-03T09:37:00.000-07:002015-05-03T09:37:01.362-07:00ANN: patch 1.14.2 - utility to apply unified diffs<div dir="ltr" style="text-align: left;" trbidi="on">Google Code shutdown kicked me to finally release the Python patch utility on PyPI.<br />
<ul style="text-align: left;"><li>Python 2 only for now (<a href="https://github.com/techtonik/python-patch/issues/10">issue #10</a>)</li>
<li>single <a href="https://github.com/techtonik/python-patch/blob/1.14.2/patch.py">patch.py file</a> that can be dropped directly into your repository</li>
<li>command line tool</li>
<li>importable diff parser library </li>
<li>doesn't create and remove files (<a href="https://github.com/techtonik/python-patch/issues/3">issue #3</a>)</li>
</ul><div>It can be run in from command line standard way<br />
<pre class="prettyprint"><code> python patch.py <patch.diff>
</code></pre><b>or</b> <a href="http://techtonik.rainforce.org/2015/01/shipping-python-tools-in-executable-zip.html">directly from .zip file</a>:</div><pre class="prettyprint"><code> python patch-1.14.zip <patch.diff>
</code></pre>Get it from PyPI: <a href="https://pypi.python.org/pypi/patch">https://pypi.python.org/pypi/patch</a></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com1tag:blogger.com,1999:blog-1623059707500976008.post-61402066907387270812015-03-09T13:12:00.003-07:002015-03-09T13:12:35.568-07:00Python CLA, Community and Node.js<div dir="ltr" style="text-align: left;" trbidi="on">
I was trying to write on CLA multiple times, since the day that I sent "My CLA" letter in hope for some explanation to python lists. Now search for "My CLA" leads to new model of Mercedes Benz released in 2014 and I wonder, why do I waste my time arguing on the things that should not matter to me? Why do they matter at all? If I could afford a Mercedes Benz and had time to enjoy it, everything else probably didn't matter.<br />
<br />
<a href="https://www.joyent.com/blog/broadening-node-js-contributions">https://www.joyent.com/blog/broadening-node-js-contributions</a><br />
<br />
The post by Bryan Cantrill is all good, but this quote in particular got my attention:<br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #666666; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; line-height: 26px;">While node.js is a Joyent-led project, I also believe that communities must make their own decisions—and a CLA is a sufficiently nuanced issue that reasonable people can disagree on its ultimate merits.</span></blockquote>
I don't remember that steps were taken to explain the necessity of CLA or ask people what they think about it. There was no community process that I've seen and my requests to get some clarification did not went good. It took months of non-constructive critics and diplomatic skills of Ezio to at least add electronic signature to the paper form. There was no incident, nothing in public to cover that. Just somebody decided to do this and then there was a lot of peer pressure to force you to comply, because "lawyer know better". I can hardly name it a community process. OpenStack fights for it, to keep process open and inclusive, tries to analyze problems, to seek solution in open way, at least it is visible.<br />
<br />
<a href="http://lists.openstack.org/pipermail/openstack-dev/2015-February/056551.html">http://lists.openstack.org/pipermail/openstack-dev/2015-February/056551.html</a><br />
<br />
Python was an open source language, and a good and unique one that deserves its own license. Guido is always open and sincere behind the community of language supporters, but are languages supporters evolve the same, do they possess the same understanding into the complicated nature of human processes to take the baton? Did the core of community become a closed elitist circle of people with good relationships? Are they able to handle the insane amount of ideas and communication that coordination around core development and surrounded infrastructure is needed? Is new generation involved in solving these challenges or all they do now is dreaming about startups? Is it a community problem or economy problem already with all these CLA and other issues that are impossible to hide?<br />
<br />
<a href="https://mail.python.org/pipermail/distutils-sig/2015-March/025807.html">https://mail.python.org/pipermail/distutils-sig/2015-March/025807.html</a><br />
<br /></div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-26024025200260045192015-02-16T12:10:00.000-08:002015-02-16T12:10:35.333-08:00ANN: hexdump 3.2 - packaged and pretty executable <div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://pypi.python.org/pypi/hexdump">https://pypi.python.org/pypi/hexdump</a><br />
<br />
The new version doesn't add much to the API, but completes a significant research about <a href="http://techtonik.rainforce.org/">executable .zip packages</a> with proof of concept distribution that is <b>both an installable</b> Python package that you can upload to PyPI <b>and an executable</b> .zip archive.<br />
<br />
The benefit is that you may drop .zip package downloaded from PyPI into your source code repository and invoke it from your helper scripts as <b>python hexdump-3.2.zip</b> Keeping dependencies together with application source code in repository may seem like a bad idea, but we find it useful <a href="https://github.com/gratipay/gratipay.com/tree/master/vendor">at Gratipay</a> to setup project from checkout in offline mode with no internet connection.<br />
<br />
The solution required three modifications to distutils <b>setup.py</b> (<a href="https://bitbucket.org/techtonik/hexdump/src/d227091343776aef1105c19813ae03beeff9d5b4/setup.py?at=default#cl-40">see sources</a>):<br />
<ul style="text-align: left;">
<li>force sdist command to produce .zip file</li>
<li>strip extra dir created at PyPI package root</li>
<li>add __main__.py to make .zip executable</li>
</ul>
<br />
If you think that it is cool and should be promoted further, feel free to test Gratipay to send me 0.01+, and I'll get the signal to spend more time on reversing native distutils/packaging gotchas.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-58895358375169316402015-01-08T01:46:00.001-08:002015-01-18T10:00:55.914-08:00Shipping Python tools in executable .zip packages<div dir="ltr" style="text-align: left;" trbidi="on"><b>UP201501</b>: Found out how to <a href="http://stackoverflow.com/questions/27998581/force-sdist-to-create-zip-archive-even-on-linux">force setup.py create .zip</a> instead of .tar.gz<br />
<br />
This is about how to make <a href="https://docs.python.org/2/distutils/sourcedist.html">Python source packages</a> executable. As a side effect, this also explains, how to run <a href="pyinvoke.org">invoke</a>'s tasks.py with a copy of invoke .zip shipped with your source code, but without installing it.<br />
<br />
You know, Python can execute .zip files. As simply as:</div><pre><code class="prettyprint">$ wget https://pypi.python.org/packages/source/h/hexdump/hexdump-3.1.zip
$ python hexdump-3.1.zip
/usr/bin/python: can't find '__main__.py' in 'hexdump-3.1.zip'</code></pre><div>Well, you need to place '__main__.py' into the root. It will then import what it needs and do something. The only problem with .zip source packages is that their structure (if created with standard Python tools) includes one additional directory at the top:<br />
<pre class="prettyprint">`-- hexdump-3.1
|-- PKG-INFO
|-- README.txt
|-- hexdump.py
|-- hexfile.bin
`-- setup.py</pre><div>So, even if you place `__main__.py` into the root, it won't be able to import file from `hexdump-3.1` subdirectory.. unless you prepare for it. Here is the solution:</div><pre class="prettyprint">import os
import sys
# add package .zip to python lookup path
__dir__ = os.path.dirname(__file__)
path = os.path.join(__dir__, 'hexdump-3.1')
sys.path.insert(0, path)
import hexdump
msg = hexdump.dehex("48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21")
print(msg)</pre><div>Now if you run it, it will print the obvious.<br />
</div><pre class="prettyprint">$ python hexdump-3.1.zip
Hello, World!</pre><div>In Python 3, the result will be wrapped into b'' string. But that's it!<br />
</div><br />
<div>The next hexdump version will likely to ship as executable .zip package<br />
</div><div><br />
</div><h3 style="text-align: left;"><b>Where you may need it?</b></h3><div><br />
</div><div>It may be useful for scripts automation if you're "<a href="bitprophet.org/blog/2012/06/07/on-vendorizing/">vendorizing</a>" dependencies (ship them with your code). It started with need to fix a broken link on <b>Gratipay</b> site. The Gratipay <a href="https://github.com/gratipay/gratipay.com/blob/master/COPYING">codebase is in public domain</a> clear of any NDA, CLA and all other BS, and that makes it great for people to learn, reuse and enhance. I wish Python internal projects possessed these properties, but PSF politics is another topic.<br />
<br />
<a href="http://inside.gratipay.com/">Gratipay inside</a> site uses make. It doesn't need powerful build tool, such as <a href="scons.org">scons</a>, so I tried to replace it with some simple task automation utility to be more cross-platform. I chose <a href="http://www.pyinvoke.org/">invoke</a>, because of my previous experience with <a href="http://www.fabfile.org/">fabric</a> for remote control, and because I saw previous attempts in another Gratipay repository. The necessary condition for new tool is that user experience should not degrade for make users, so main invoke's tasks.py script needed to be self-executable.<br />
<br />
I completed the proof of concept by making invoke .zip package executable with the method above, placed it into vendor/ directory, and tuned tasks.py to reexecute itself with invoke .zip This also solved a potential problem with currently unstable invoke API.<br />
<br />
tasks.py was modified as following:<br />
</div><pre class="prettyprint">import sys
sys.path.insert(0, 'vendor/invoke-0.9.0.zip')
from invoke import task, run
# ... tasks go here
if __name__ == '__main__':
import sys
from invoke import cli
cli.main()
</pre><br />
<h3 style="text-align: left;">Bonus points</h3><div><br />
</div><div>There is one more thing, which is better explained with <b>hexdump</b> example. <a href="https://pypi.python.org/pypi/hexdump">hexdump package</a> ships `hexfile.bin` data file used for testing. hexdump.py looks for it in its directory (__file__ dirname), and this lookup fails when hexdump.py is in archive. I'd say Python could provide some kind of an API to deal with "virtual import filesystem" (with watchers to track who and when modifies <b>sys.path</b>), but I digress. If you modify the `__main__.py` from the first chapter above to run tests - just call hexdump.runtest() - the execution will finally fail with the following error:</div><div><pre class="prettyprint">Traceback (most recent call last):
File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "hexdump-3.1.zip/__main__.py", line 13, in
File "hexdump-3.1.zip/hexdump-3.1/hexdump.py", line 311, in runtest
IOError: [Errno 20] Not a directory: '/root/hexdump-3.1.zip/hexdump-3.1/hexfile.bin'</pre></div><div>(yes, I am hexdumping under root). The problem above was explained by <a href="http://wiki.sheep.art.pl/Your%20Python%20Application%20as%20a%20Single%20File">Radomir</a>, and I feel like the post is already tool long to write about it myself.</div><div><br />
</div><h3 style="text-align: left;">More bonus points</h3><div><br />
</div><div>In theory it is possible to hack setup.py to produce executable source .zip package. By default, <b>setup.py sdist</b> command on Linux creates tar.gz files. So, first it needs to be forced to create .zip file. This can be done by overriding default --formats option:<br />
<pre class="prettyprint"># Override sdist to always produce .zip archive
from distutils.command.sdist import sdist as _sdist
class sdistzip(_sdist):
def initialize_options(self):
_sdist.initialize_options(self)
self.formats = 'zip'
setup(
...
cmdclass={'sdist': sdistzip},
)
</pre>I posted relevant documentation links to <a href="http://stackoverflow.com/questions/27998581/force-sdist-to-create-zip-archive-even-on-linux">this StackOverflow question</a>. The rest - how to inject __main__.py into the root of packed .zip - is yet to be covered, and I am not ready to research it just yet. Some hints may be provided by <a href="https://stackoverflow.com/questions/6193999/how-to-use-distutils-to-create-executable-zip-file">these answers</a>.</div><div><br />
</div><h3 style="text-align: left;">Usability conclusion</h3><div><br />
</div><div>I am actually quite happy with the ability to execute python .zip packages (which gave a lot of motivation to write this post), because previously I had to care about how to tell people that a tool is a single script, which can be downloaded from repository or from some dedicated download site (I mean Google Code, which does not support this anymore). I also don't trust my own server for downloads, because one day I found some "benzonasos" running there and I don't even have a slightest idea how it got there.</div><div><br />
</div><div>But now it becomes possible to just download and run the stuff from PyPI to test how it works without messing with creating and activating virtualenvs, and installation of package inside. Of course, if your .zip package has dependencies, they still need to be present on your system, but that's still a time saver. Oh, and that means that my tools can now consist of several modules inside.</div><div><br />
</div></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com7tag:blogger.com,1999:blog-1623059707500976008.post-50552100808917529332014-11-17T03:37:00.001-08:002014-11-17T03:37:30.719-08:00C/C++ Must Die<div dir="ltr" style="text-align: left;" trbidi="on">
It is a four hour forward only text. I am not in the mood to clean. reread, refactor, split or fix any mistakes right now, and I am not saying sorry for that, because it is how it is written with this limited keyboard interface that I practiced over my lifetime. I am sorry only if you won't be able to read or understand that, because it would mean I made it all worse. I am already at the point that I want to filter it straight to the subject, reread and fix all typos, obscure moments, digressions and words, but I've already out of time and it might happen that there won't be any time for that later.<br />
<br />
<br />
My system background is [assembly] --> [pascal] --> [c] --> [php] --> [python], so C is not my favorite language, I am in any way attached to it and don't exhibit a feeling of nostalgia and deep sadness when thinking about what changed in the past 40 years that passed since its invention. What I really regret is that there is no a place like Bell Labs anymore that could prevent bloody enterprise from formatting people to the bloody business mindset. Formatting so badly that you even need a ton of startups to just produce ideas (not mentioning getting them right).<br />
<br />
So, in the past 40 years, a new language appeared called Python. I don't think that Guido knew the words "usability", "user experience" or CJM when inventing it - it was just natural and human like, just because it was not engineered, but made through feeling of a human, who likes Monty Python and doesn't follow accepted rituals of human organisms in suits. I understand his desire to keep CWI copyright in place - it is a living reminder about another environment that made this thing possible. Unlike these bloody corporations that keep pressure on what you need to do and when to deliver. There is not much creativity and freedom right now. Those tiny groups of people who gather through the internet try to change the things. But the poor expressive power of non-native English language, time gap and invisible mood and attitude changes make it almost impossible to provide a stable communication channel that will not limit itself only to a narrow band of appreciation, joy and positive emoji. All negative emotions, drama, critics and depressions exists there for a reason - to express the importance of details and adapt them.<br />
<br />
Accepted rituals of human organisms in suits. It is hard to be human. So hard that the rational part of us will never accept the idea that I, me, myself are limited. Humans had somehow devised the ethics, culture and rituals to deal with that. Implemented complex system of oppressing behavior and enforcing rules to make everyone the same and keep balance. Capitalist economy made it possible to compete, to make a gameplay to measure and estimate your value compared to someone else instead of just killing him, because his will to live just dampens yours. Money had been chosen as a universal way to solve conflicts and it worked for a few generations. Until a new generation appeared that didn't not care about conflicts and didn't see why would anyone should conflict if everything you need is money. We are still limited humans who fail to see the balance and yet we are trying to prove that we are the best, we need to be respected etc, etc. Whatever our rituals are, they are a shroud on on our eyes. The shroud that doesn't allow us to see how human really work, that our perception is not discrete and the expressive power of art is something we should explore as a discipline of human interfaces.<br />
<br />
C/C++ is one such interface. It is a written language that no civilization from the outer world will be able to decipher. I didn't research this idea to far, so I leave it to you. It is a formal interface with brackets, asterisks and letter combinations that convey some ideas and concepts. It brings the former poor language of assembly to a whole new level. The level of more toys to play with and *exchange*. Every language invented after it was adding or removing these toys for humans to play with, but they still have hard time to combine all programming languages into the garden of toys as a whole. The garden that you can explore with the rest of your senses (further that arithmetic and text processor units in your head) to see how things work, tear them apart, which toys go well together and which just don't belong. There should be right toy for the game. It is in our mindset. We use the toy that we've used to, learned the first and like opiate, it instills us with the knowledge that this is the best tool in the world, in your world, because everything else if perhaps just worse. The nostalgic feeling brings us emotions, knowledge and experience keeps us in a comfort zone. Egoism. We want to world to be the same. But while we were playing with our toys, it changed.<br />
<br />
Many toys appeared and vanished. Patterns made some of us high. UML - the art erased out of drawing - made slave drivers happy. These were adopted ways to transmit the knowledge, format the culture, and narrow the communication down to the limited capabilities of any human put into the bloody cage of enterprise machine. Bloody doesn't mean we can escape it. It is just bloody. I see it as squashing the humans into the cubes of comfort zones and I am afraid of it. There is no progress outside the cells of enterprise organisms and these organisms are still mostly parasites hooked on resources that they are extracting from ecosystem and killing the planet by their exhaust. If you think that Google is good - take a look to the streets - if there is any litter around - how do you think - could people from this Enterprise remove this litter with their bare hands? I think they could. It is easy and even fun. But will they?<br />
<br />
C/C++ is not a litter. It is a tool and a toy. The world had grown a lot, but the tool and the toy stayed the same. In these 40 years we've learned that we can only have juggle 7 things in our head at a time. We have only 60 hertz in out heads compared to 3Ghz of modern CPU. 7 things is the most that can survive through the neural network while signal passes from one domain (limited 3D area) through the whole volume of absolutely parallel processing unit. The amount of thing that we need to keep in mind is bytes, chars and pointers, even for string processing. And as a humans we operate with strings. Even bytes in our heads look like hexadecimal and we are still constantly ignoring this. We are continuing to invent rational logical abstractions - new toys - instead of switching off the conscious part of our brain and concentrate on the part that scans things. In the past 42 years we've upgraded our interface from reading to scanning and nobody noticed that. The older people still insist on reading books, going through lengthy letters in mailing lists, and lots of articles. But we've invented movies, animation, visual arts, demoscene. The last one appeared and vanished - it was a sudden ray of warm and bright light - an excitement that was impossible to rationalize and explain, and that gave a birth to a whole generation of game designers out there.<br />
<br />
C/C++ is not a game. It is a legacy, a hack, a combination of mysterious symbols that make impossible things possible. This combination has expressive power and concepts that will survive a whole new generations from now on. You will still need an instrument to manipulate memory on low level. An instrument to control hardware with high level concepts. But it is not C/C++ that needs to be used for that, because it doesn't make it optional. C/C++ doesn't help you scan programs faster, read operations behind combinations of asterisks, symbols and brackets to fill the 7 slots in your head with something that makes sense. It doesn't instill me with a sense of confidence, because the code once already written may leak in different places.<br />
<br />
Back in a days we were playing with reversing the software. You choose the entrypoint and then try to figure out what software does. Most of the time you had to work with assembly, follow the jumps and procedures, you were writing tools to help you deal with the complexity and automate the journey. The process didn't change since. What changed is that you don't need to think about memory allocations anymore, algorithms can tackle strings more efficiently than humans, so this slot is freed too. You can concentrate on your DNA handling logic, on algorithm, on final result that you need to achieve and this result will make sense for ordinary humans, not for people with computer science camp behind. Python made this possible by providing this interface for humans when hardware allowed that. Python adapted itself to humans who now scan things instead of reading them. You don't need a whole usability discipline to note a small nifty things to make improvements like these, but when you have this whole new usability and ux buzzworld emerged, it is somehow wrong not to take an advantage of that. It is quite natural and right from the point of standard human specie, but wrong from the point of augmented human that needs tools to be evolved to allow more freedom. Evolution in Python led to the Zen of it. It is not a speccy, not a book, not a long text in a foreign language. It is an artpiece - a thing that could only be born among people who value the artpiece of Monty Python, and appreciate irrational culture of humans and play that. Without the art squeaky PyPI will be remove and the new warehouse will be built, welcome sign will be replaced with patent paranoia, you will have to sign a paper and undergo passport scan to pass through. People attempt to save things that are important to them if they can see them. You can't see the irrational, you can't explain it, especially things that came naturally. Humans are limited in understanding things, in expressing things, complex associations and irrational thoughts that came out of their emotions, knowledge, fears and beliefs. They are limited in their ability to share this information with others, they don't have time to pass info anymore, can't communicate so fast and this is where the progress stops. Humans are limited, they can not deal with the complexity, and they naturally developed arts to deal with that. Arts communicate on different levels of perception and education, they are waiting for you to get there to feel them or came down on you rapidly right away. It is a way to look for answers, to explore, express and communicate them. It is about at least some kind of freedom in dealing with paralysis of complexity. But this is about C/C++ and I need to finish this.<br />
<br />
C/C++ is insecurity, but at the same time it brings trust and freedom. At least it was many years ago. It is the last piece of freedom that makes hacking of the systems possible. It was fun and you don't want to kill the fun once you're there. It is not hackers, it is humans, sometimes among them, who kill the fun. Paranoia, fear, social engineering techniques that make talented hackers kill themselves. It is not fun anymore to hack things in this world of fight of limited species over limited resource. Perhaps the backdoor needs to close. I feel deep sadness and sorrow by thinking about that, but who cares anyway. There is a huge demand for a better readable human interfaces for controlling new technologies and devices that will be born to augment humans and while we are messing with bytes and pointers leaning of some memory scattered over the table, we skip the cosmos of possibilities and things that shine in depths behind our necks. There is a place for better coupling between human intention to care or not care about low level details, better interface between the though to operate on some level and the tools that make the thoughts real. It is important to keep it human oriented, because the picture that one human drawn should be intuitive and clean for other human to comprehend. If that second human is familiar with concepts, it should not undergo the long training in decyphering glyphs just to get into details. We've already seen that this is possible. Just need to experiment with this a little bit more to pave our way in the way art does it. Art is the ultimate hack of all times after the science.<br />
<br />
And for us, for new generation of coders, C/C++ must die to let evolution continue.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-38292653607789779762014-10-09T01:07:00.001-07:002014-10-09T01:17:51.607-07:00Domain disaster, identity, history, account balance and horizons<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpZXtKgMHcStb04bHLxSZI_yTi4Ms9qmuXngnC-bSHBTvK6rL6o5yfO2bQQ0t4_0OT7_1u2kVE8gbQclKeevuVNbxwnTY3S_eP1MWuOJ1941sgbs0T1yeDZePSXbwBRPDM4awB3-7o4Kc/s1600/domain-disaster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpZXtKgMHcStb04bHLxSZI_yTi4Ms9qmuXngnC-bSHBTvK6rL6o5yfO2bQQ0t4_0OT7_1u2kVE8gbQclKeevuVNbxwnTY3S_eP1MWuOJ1941sgbs0T1yeDZePSXbwBRPDM4awB3-7o4Kc/s1600/domain-disaster.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
I've added Python tag to this post, because the traffic above is mostly Python related, so it might be interesting to those concerned about preservation on their posts (or about essential properties of well-known trolls in certain communities). Blogging is not an easy process, especially if you're not an English native and it takes about 4 hours on average to rewrite and reword the content until it satisfies you, so preserving this knowledge is somewhat important. Some of the most emotional and big posts are never published at all (like black hate of Python 3 default system encoding or drafts of Spyder plugin architecture) .<br />
<br />
So, what happened with the graph above?<br />
<br />
With zero income over the past 2+ years I couldn't keep up with my bills and the domain name registration expired after I couldn't make it. It doesn't cost much ~$10 a year, but it is on the bill with DreamHost web hosting plan that costs $120 yearly + minimal VPS instance for $5/month to run Bitbucket <a href="https://bitbucket.org/rirror">mirror</a>s and other services. I got the money from a $500 gift that I wanted to make to my girlfriend after we broke apart. To be honest I never had a girlfrend, but it was a nice try. It was painful, but in the end these $500 helped to save on debts without sacrificing my desire to meet with friends. I must say it is really hard to sustain any kind of relationship (timely replies, calls, switching to the context of another person, sharing interests, job activities) when you have an ADHD mindset, especially in those moments when you "catch the flow" and just escape everything outside until the feeling passes. The escape is possible. Otherwise it would be a direct way nowhere, but it is always stressful and takes its toll on psychosomatic resilience (how much stress a person can handle until it starts to grow on person's in physical form). Other factors may be contributing too, such as wiring scheme in ADHD trained brain - which can be more flexible so that it is able to diverge resources from maintenance of body systems to cognitive abilities. If there is some virus or generic modification in your body (chronic disease is also an option) - it might (probably will) take this chance to capture more territory when under a stress condition. At a time when rewiring process switches off lights in corridors near the dark corners of immune system. But that's is just a theory, an indication how easy it is for me to become distracted, which in turn is the cause why the situation emerged at all.<br />
<br />
Looking at the chart of traffic I think that if that traffic was in any way useful at all, I wish it could be directly converted into paying for domain name and hosting. I don't need extra and don't want to "monetize" content by introducing pestering ads and other nasty stuff from the "business end" - just need the power to keep the systems up and operational. It is quite sad knowing that only a year needs to pass after you die to erase traces of your work from the internet, and even <a href="https://archive.org/about/faqs.php#14">web.archive.org don't help</a> when new owner of your domain uploads robots.txt to block resource-sucking spiders. It will be much more comfortable knowing that what you do will persist to help somebody else save some time in future.<br />
<br />
The easy answer would be to just blog on *.blogger.com *.wordpress.com or facebook if you like, and that's already an option for a newer generation, but you own domain is one of the achievements in the ladder of your technical skills. Or sometimes it is just matters for some reason.<br />
<br />
One high tech idea to build a sustainable human-less domain preservation mechanism was to build a closed loop around automated recurring payments with bitcoins. That one needs a lot of time to develop and clean up roadblocks for an automatic chain that transforms useful traffic to <a href="https://www.ethereum.org/">Etherium</a> or any other similar Bitcoin 2.0 concept. There are two concerns here. First is that <b>useful</b> needs to be defined more clearly. I am sure for some reason that Google already knows and calculates the metric even for posts without views, likes or +1 buttons. Second concern is that there is so much buzz about Bitcoin 2.0, that it looks like people are already fighting to capture this playfield, and I am not sure that these new technologies are really worthy. Investigation and review needs time and it is hard to tell how much exactly without trying.<br />
<br />
Another option to sustain the costs was to use DreamHost <a href="http://www.dreamhost.com/r.cgi?105926/affiliates/">referral system</a> that through a referral link allows to get bonus for every newly registered user. rainforce.org is a very old account - it uses a deprecated recurring bonus scheme, that shares 10% of new user's payments instead of one-time bonuses, but so far I had only $5 from 10 people registered over last 5 years or so, and I don't think it is a good route to take unless you want to force people to hate you, because you're obsessed with selling them to DreamHost as referrals. rainforce.org also possesses a nice looking donation URL that looks like <a href="http://www.dreamhost.com/donate.cgi?id=3333">http://www.dreamhost.com/donate.cgi?id=3333</a> but it does not answer the questions "how much does this this blog needs to be alive and why?", "what part of a share should I pay?" and more import "why should I pay for this?". In other words there is no transparency and trust in "modern" economy, because openness and transparency hurts it. And if resources become scarce, the economy starts a killing cycle. It is not from economy - it is from ecosystems.<br />
<br />
There are always ideas for new experiments and horizons in this area, such as marrying <a href="https://gratipay.com/about/">Gratipay</a> concept with <a href="https://whispersystems.org/blog/bithub/">BitHub</a>. These project are less obsessed with "doing money software for the money", but they can not escape the "<a href="http://en.wikipedia.org/wiki/Preferential_attachment">preferential attachment</a>" of people around it, meaning that people will support projects that support money movement more than projects that need actual support, such as <a href="http://pypy.org/">PyPy</a>, <a href="https://code.google.com/p/spyderlib/">Spyder</a> and <a href="http://qt-project.org/wiki/PySide">PySide</a>. It might be not even about project support, but more about people contributing to these projects. I don't think that I have a solution to the problem, because it complex, but I think I'd be happy to try if I had a chance.<br />
<br />
I tried to submit a proposal for rainforce.org activities to <a href="http://ec.europa.eu/programmes/horizon2020/">H2020</a> program as a last chance to escape the corporate clutches and dedicate some time to experiment, but I could not find a single idea over which to subscribe myself to work over for the next five years. In the end I realized that I am alone and hardly the right the right person to do coordination and communication work even though it might be extremely interesting at some point (and hence extremely difficult to resist).<br />
<br />
So, in the conclusion of this letter to make it more useful and worthy of 4 hour time, I'd really like to see more people from hard core Python projects that are somehow supported in Europe (such as PyPy and PySide), and especially people from outside who interested in helping them, to take a look at the Horizon 2020 program. It is a funding scheme for EU companies, but organizations from outside (including US and Russia) are welcome to participate with their own funds if they contribute something unique. I believe it is the only way for independent R&D to continue and it will be a pity to miss this chance to experiment with transparent economy schemes and trust systems to support people in open source and open science projects. And it requires more than just coding skills to handle. In fact, it may require a <a href="https://bitbucket.org/duangle">completely different</a> set of skills, even art or design or cognitive sciences - everything that helps better understanding and communication. Because all problems in the world are the problems of communication.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-13486468924911671122014-08-15T23:12:00.001-07:002014-08-15T23:13:12.559-07:00ANN: sha1 1.0 - command line tool to calculate file hash<div dir="ltr" style="text-align: left;" trbidi="on">Windows doesn't have a native tool to calculate SHA-1, so I've made one that can be easily installed and used from command line on any operating system thanks to Python:<br />
<pre><code class="prettyprint">python -m pip install sha1
python -m sha1 <filename></code></pre></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-20255504718279225152014-05-31T07:00:00.003-07:002014-05-31T07:00:41.056-07:00Python 3.2 has some deadly infection<div dir="ltr" style="text-align: left;" trbidi="on">
Another not really positive low quality post, so if you're not in a mood to hear some rants, just skip it.<br />
<br />
<br />
I don't consider talks about different programming languages to be blasphemy. Quite otherwise - they are good to let you rethink what you do daily in your own language. In a recent discussion about languages (and after many fruitless attempts to compile Wesnoth with GCC on Windows) I thought that C/C++ must die, because really all major security problems are because of it. Memory problems, reinventing the wheels, wasting time in Makefile's and Autoconf / Visual Studio project files. Everybody is just forced to use it, people feel like they've become an elite by learning it, but in fact they are just wasting time. Well, you can't blame people for that. Everybody entertains in a way he can, and me while writing this post too.<br />
<br />
Yes, early hacker's culture is still here, but is it here to hack on real world problems anymore? NumPy, SciPy, PANDAS, BioPython - this is the example of tools that modern hackers use. C is only useful to speed up parts of application and that's it. It is not designed to make app more secure, to save developer's time, to increase productivity, even to be hardware independent, and I believe the main reason for its popularity is that it is just the default for Unix. C for Unix is a Visual Basic for Windows. This doesn't make Visual Basic better - this makes C worse.<br />
<br />
What about Python? Well, here are some stats.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj09kLsshjYjZvQ-Plf-xmK1oF4pHARaHO6LphgO34683Tgm_O7O530uYAK3Fbbs9VT9nU_KGNRO9gq9nv0KfvDMd2IGGjpTSvgGbcQH1w85bRoYb9q7J45zvp3R5WvtVPXQKyW5Fev4ik/s1600/python-top.png" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
These are stats from TIOBE (clickable). They don't measure anything - just show some lines that correlate to each other.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
It looks like the peak of Python was on February 2011, and since then there was a significant drop. The next peak you see is February 2013 - month before PyCon in California, followed by a deeper decline. Let's see <a href="https://www.python.org/doc/versions/">what happened on February 2011</a>:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<ul class="simple" style="background-color: #f9f9f9; border: 0px; box-sizing: border-box; color: #444444; font-family: SourceSansProRegular, Arial, sans-serif; font-size: 15px; line-height: 1.875; list-style: square; margin: 0px 0px 1.3125em 1.5em; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; box-sizing: border-box; font-family: inherit; font-size: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: 1.65em; margin: 0.2625em 0px; padding: 0px; vertical-align: baseline;"><a class="reference external" href="http://docs.python.org/release/3.2/" style="-webkit-tap-highlight-color: rgb(30, 41, 51); border-bottom-color: rgb(255, 223, 118); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #3776ab; display: inline; text-decoration: none;">Python 3.2</a>, documentation released on 20 February 2011.</li>
</ul>
<div class="separator" style="clear: both; text-align: left;">
Right. Everybody knew that Python 3.1 is going to be bad, and everybody expected Python 3.2 to be good, but that didn't happen. You may tell people anything - "don't expect", "it is planned", "it is better" - regardless if people hear you or not, they will still have some expectations, and if these expectations don't match the experience , you see the "disappointment line" decline. You see the second peak didn't bring any good news either.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
I am disappointed too, and if you don't want to read yet another boring complain, just skip this paragraph. I expected Python 3 to be ready for the internet age, with cross-platform behavior preferred over system-dependent one, with clear semantics to work with binary data, with non-confusing error handling and API that gives hints and helps to pave your way on uncharted grounds, such as abstract unicode. Maybe Python 3 is beautiful, but I fail to see how - it is not more modular than Python 2 - I can not switch features of the language on and off to taste experimental stuff of propose my own, I don't see any pictures of improved internal structures or . I didn't expect more things to work by default, but I hoped that explicit interfaces continue to be intuitive. There are many little things that sum up and spoil the fun like trailing whitespaces in JSON output that play significant role in providing backward compatibility for Python. I don't know if that is fixed in Python 3, and I don't have any means to track that. When these little things sum up, you realize that you're just wasting time trying to improve things that people don't want to improve. They don't want to improve the process. They don't realize that the problem is not in the language, but in the way they don't want to hear each other. Technology showed that people want to be heard, that they opinion should be accounted , not closed as won't fix , or works for me . It is not a community process, when you rely on abilities of certain individuals to monitor and respond to all traffic and wishes, especially when they fail to do so.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Community process is when there are tools that people can collaborate about, there is research, a place for sharing opinions and sum up the result, a time to track the progress and do the work, and an open environment to experiment and evolve. Where every opinion counts and adds up to become a signal. Where signals are reacted to not because there are people who feel responsible, but because it is fun and has a positive feedback.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There is a certain paralysis. Many people are stuck in corporate against their will, bound by economy. But I am sure in every company it is possible to dedicate a "SyncFriday" - day dedicated to sync with other developers outside to do things in open source with all necessary stuff prepared. If a company can not afford it - why not just leave it. What's the point in helping corporation that doesn't make the world better from your own point of view too?</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
And much as with corporations, I don't see why anyone should be interested to help Python 3.x to become better. It takes too much energy and is so hopeless, at least for me. Singing papers, writing long explanatory PEP notes, discussing things with your core boss. There are no stats, no summaries, no analysis and comparisons, root cause research - only a lots of text and opinionated and busy individuals like myself.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Can this be changed? Not with text. More people who can draw, animate, make beautiful graphics work to make hard problems visible. This will help. Work with people expectations - if these are unreal - do not try to change them, explain them. Make the problems visible, give people the freedom to try, do not demand, do not press, just create environment, and do not use text - it is useless. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Art is the future. And the future is both visually appealing, nice to touch and easy to follow.</div>
</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-31833678095107673932014-04-29T05:57:00.001-07:002017-03-07T05:32:08.338-08:00Writing SCons Plugin: Discovery<div dir="ltr" style="text-align: left;" trbidi="on"><b>UP201410</b>: Add version info and link to Parts concept.<br />
<br />
<div dir="ltr" style="text-align: left;" trbidi="on"><a href="http://scons.org/wiki/">SCons</a> is a build tool written completely in Python, so that you can just put it into your repository, extend for your own purposes and run directly from checkout without installing anything. SCons documentation doesn't use word <b>plugin</b> to refer to extensions. It calls them <b>Tools</b>. If you're not familiar with SCons concepts, check out <a href="http://scons.org/wiki/BasicConcepts">this wiki page</a>.<br />
<br />
<h4 style="text-align: left;">Plugin Discovery</h4><br />
<div>SCons tries its best to avoid magical behavior. That's why it ignores paths and options set in system environment and requires that you specify everything explicitly. The same is true for plugins. Example from SCons man page:</div><div></div><pre class="prettyprint">env = Environment(tools = ['default', 'foo'], toolpath = ['tooldir'])</pre><div>This creates build environment, initializes default tools, sets lookup path for plugins to <i>tooldir/</i> and enables tool named <i>foo</i>, which is located in <i>tooldir/foo.py. </i><i>foo.py</i> should have two functions - generate(env) and exists(env).<br />
<br />
To test that your tool is found correctly, check env['TOOLS']:<br />
<pre class="prettyprint">print(env['TOOLS'])</pre>If filename is not there - SCons was unable to find it. If it was a problem with contents of <i>foo.py</i> - SCons would fail with exception.<br />
<br />
SCons has an automatic tool discovery mechanism - if you don't like to specify <i>toolpath</i> directly - you may place your tool in one of several locations that SCons scans before executing SConstruct. See description of the <a href="http://www.scons.org/doc/HTML/scons-man.html">--site-dir=dir option</a> for details.</div></div><br />
<h4 style="text-align: left;">Version compatibility and future</h4><br />
<div>Described behavior is true for <a href="https://bitbucket.org/scons/scons/src/tip/src/CHANGES.txt?at=default">SCons 2.3.4</a> and earlier versions. It may be not very intuitive, so things may change in future, especially if we find resources to integrate <a href="https://pairlist2.pair.net/pipermail/scons-dev/2014-September/001939.html">Parts</a>.</div></div>anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-5885281667633408092014-03-28T04:25:00.004-07:002014-04-02T08:07:15.709-07:00Python C API/ABI compatibility report<div dir="ltr" style="text-align: left;" trbidi="on">
UPDATE: There is now an official thread on <a href="http://www.gossamer-threads.com/lists/python/dev/1130757">python-dev</a>.<br />
<br />
<a href="http://ispras.linuxbase.org/index.php/Upstream_Tracker">Upstream Tracker</a> is an open source (GPL) tool that allows to track <a href="http://stackoverflow.com/a/3784697/239247">API/ABI</a> changes between releases of C/C++ libraries. I asked <a href="http://wiki.rosalab.ru/en/index.php?title=User:Andrey.ponomarenko">Andrey Ponomarenko</a>, who is the main maintainer of the project, to add Python to the list and here is the result: <br />
<br />
<a href="http://upstream-tracker.org/versions/python.html">http://upstream-tracker.org/versions/python.html</a><br />
<br />
Hope you like it.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-74773024241109398472014-02-17T08:24:00.000-08:002014-02-17T08:24:42.087-08:00ANN: xtrace 0.5 - indented function trace in Xdebug format<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="http://xdebug.org/docs/execution_trace">Xdebug</a> is a PHP tool that allows to trace how PHP code is executed. Today I release a tool called <a href="https://pypi.python.org/pypi/xtrace">xtrace</a> into public (and more specifically into public domain), which allows to get the same (or at least very identical) output, but for Python.<br />
<br />
Few years ago inability to trace function calls in Python came as a showstopper. I decided to write a familiar tool for Python. More than that - I wanted to integrate it into <a href="https://code.google.com/p/spyderlib/">Spyder</a>. But a year ago xtrace itself faced with showstopper. The showstopper was the <a href="http://execfile%20behaviour/">behavior of execfile</a> function, that I could not get right at that time, because of the docs, of my expectations and poor knowledge of English. Maybe there is a flaw in my cognitive abilities, but I tried to get at this problem several times and failed. Until recently some hackers from <a href="http://zensecurity.su/">ZenSecurity</a> team brought a concept known an <b>pyjail</b> to my attention. The challenge to prove that pyjail concept is impossible allowed me to concentrate on gory details of execfile works and knowing that documentation is totally confusing for my, I found the time to set my own experiments. You can read them at the link I've given above as well as some analysis why documentation that actually includes all the details can be bad and confusing.<br />
<br />
The <a href="https://pypi.python.org/pypi/xtrace">xtrace</a> was basically broken for three years, starting from the version 0.2 - the day I put <b>execfile()</b> call from root to the xtrace module to the main() function. This placement changed the execfile() behavior, and while trying to debug that I also run into confusing dynamic behavior of dictionary returned by locals(). Opened can of worms made those parasites to completely consume my brain, causing much anger and frustration to be spilled around execfile() and locals() concepts over into Python lists. It is kind of relief now that I can name all the problems, analyse them and look back as enlightened. Being jobless I had a plenty of time to investigate, but I really don't want anyone to enter that state of confusing and helplessness that I had a year ago.<br />
<br />
Hopefully, my experience with xtrace will clear the confusion for those who will try to use exec type abilities of Python for developing their own tools. Maybe it will result in a better Python API in the future, with better documentation and position-independent behavior.<br />
<br />
I am interested to know the feedback that you can leave in xtrace tracker, such as if the output really matches PHP behavior, if it is accepted by PHP tools and how it behaves in different scopes of Python. It is interesting to convert it to Spyder plugin and see the usage in other tools, but I realize that I may not have time for that. The next focus for me is to add an easy API to xtrace to enable people to write they own tracers more easily. Focus on UX and everything else will come.</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-26364932798794029262014-02-02T04:11:00.000-08:002014-02-02T04:12:32.617-08:00ANN: hexdump 2.0 - view/edit your binary with hex tool<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://pypi.python.org/pypi/hexdump">https://pypi.python.org/pypi/hexdump</a><br />
<br />
Finally some prod that can be named feature-complete for release. It is cross-platform, meaning it should run the same on Windows (tested), Linux, and OS X. It is Python 2 and Python 3 compatible. And it released into public domain, so that you won't have any problems in reusing it for your commercial and non-commercial hacking.<br />
<br />
For those who are unaware of what hexdump is, <b>hexdump</b> is a representation of any binary data in human readable form. This form is good for hacking, inspecting and debugging binary data and protocols, but it is also good for editing such data. I am not pasting the output if the tool to encourage you to play with it yourself.<br />
<br />
It can be used as command line tool and as a library. The most simple way is to use it as a tool:<br />
<pre><code class="prettyprint"># install
$ python -m pip install hexdump
# dump
$ python -m hexdump binary.bin > dump.txt
...
# restore
$ python -m hexdump --restore dump.txt
...
</code>
</pre>
<b>P.S.</b> I don't mind including `hexdump` as <a href="http://www.python.org/dev/peps/pep-0411/">provisional package</a> in Python standard library if anyone will be able to convince PSF to accept public domain, CC0 or MIT licensed code.<br />
<br /></div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0tag:blogger.com,1999:blog-1623059707500976008.post-3823262202761379032014-01-10T20:29:00.003-08:002014-03-26T05:42:16.951-07:00 Draw a pixel with PySDL2<div dir="ltr" style="text-align: left;" trbidi="on">
This is a minimal code to output pixel on the screen using <a href="https://pypi.python.org/pypi/PySDL2/">PySDL2</a>.<br />
<br />
UPD: (March 2014) Up for PySDL2 0.9.0 (RenderContext renamed to Renderer)<br />
<pre class="prettyprint"><code>#!/usr/bin/env python
"""
The code is placed into public domain
by anatoly techtonik <techtonik@gmail.com>
"""
import sdl2
import sdl2.ext as lib
lib.init()
window = lib.Window('', size=(300, 100))
window.show()
renderer = lib.Renderer(window)
renderer.draw_point([10,10], lib.Color(255,255,255))
renderer.present()
running = True
while running:
for e in lib.get_events():
if e.type == sdl2.SDL_QUIT:
running = False
break
if e.type == sdl2.SDL_KEYDOWN:
if e.key.keysym.sym == sdl2.SDLK_ESCAPE:
running = False
break
</code></pre>
</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com1tag:blogger.com,1999:blog-1623059707500976008.post-10418268691309169972014-01-07T07:35:00.002-08:002014-01-07T07:39:35.239-08:00Open Source / Free Standards vs ISO/IEC<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
Intro</h3>
While trying to use Galaxy Note 10.1 as a tablet and remote control device for my Windows and Linux stations, I discovered awesome MIT licensed <a href="https://github.com/rfc2822/GfxTablet">GfxTablet</a> project (draw on your PC via your Android device):<br />
<blockquote class="tr_bq">
<div style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px; margin-bottom: 15px; margin-top: 15px;">
GfxTablet shall make it possible to use your Android device (especially tablets) like a graphics tablet.</div>
<div style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px; margin-bottom: 15px; margin-top: 15px;">
It consists of two components:</div>
<ul style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px; margin: 15px 0px; padding: 0px 0px 0px 30px;">
<li>the GfxTablet Android app</li>
<li>the input driver for your PC</li>
</ul>
<div style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px; margin-bottom: 15px; margin-top: 15px;">
The GfxTablet app sends motion and touch events via UDP to a specified host on port 40118.</div>
</blockquote>
It was so awesome in its simplicity and protocol that I couldn't resist to build <span id="goog_70947996"></span>a <a href="https://bitbucket.org/techtonik/discovery/src/tip/graphics/gfxtablet/?at=default">Python client<span id="goog_70947997"></span> for it</a>. It didn't take long (well, a day maybe), before I noticed two errors in <a href="https://github.com/rfc2822/GfxTablet/blob/master/doc/protocol.txt">its protocol</a>. First is that byte order for fields with 2 bytes length was not described and appeared to be <a href="http://simple.wikipedia.org/wiki/Endianness">big endian</a> (while I assumed the opposite). Second is that one of the fields described as 2 bytes <span style="font-family: Courier New, Courier, monospace;">ushort</span> was actually 1 byte size <span style="font-family: Courier New, Courier, monospace;">octet</span>. After reading the source, I found the mistakes and edited the protocol description from the web to fix them, which resulted in this (already merged, yay!) <a href="https://github.com/rfc2822/GfxTablet/pull/46">pull request</a>.<br />
<br />
<h3 style="text-align: left;">
Standard on Byte Size</h3>
I've got an interesting comment on my <a href="https://github.com/rfc2822/GfxTablet/pull/46">pull request</a>:<br />
<blockquote class="tr_bq">
<span style="background-color: #fbfbfb; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 13px; line-height: 22px;">"octet" is more accurate than "byte" because there are systems and programming languages which have a byte size that is not 8 bits</span></blockquote>
My natural reaction was "No way! That's can't be true.", but Wikipedia <a href="http://en.wikipedia.org/wiki/Byte">said I am wrong</a>. Luckily, it also said that:<br />
<blockquote class="tr_bq">
<span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 19.1875px;">The </span><a href="http://en.wikipedia.org/wiki/De_facto_standard" style="background-color: white; background-image: none; color: #0b0080; font-family: sans-serif; font-size: 13px; line-height: 19.1875px; text-decoration: none;" title="De facto standard"><i>de facto</i> standard</a><span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 19.1875px;"> of eight bits is a convenient </span><a href="http://en.wikipedia.org/wiki/Power_of_two" style="background-color: white; background-image: none; color: #0b0080; font-family: sans-serif; font-size: 13px; line-height: 19.1875px; text-decoration: none;" title="Power of two">power of two</a><span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 19.1875px;"> permitting the values 0 through 255 for one byte. The international standard </span><a href="http://en.wikipedia.org/wiki/IEC_80000-13" style="background-color: white; background-image: none; color: #0b0080; font-family: sans-serif; font-size: 13px; line-height: 19.1875px; text-decoration: none;" title="IEC 80000-13">IEC 80000-13</a><span style="background-color: white; font-family: sans-serif; font-size: 13px; line-height: 19.1875px;"> codified this common meaning.</span></blockquote>
This was the first time I thought that ISO did something right, so I decided to take a look myself at this standard. I found two copies - <a href="http://webstore.iec.ch/webstore/webstore.nsf/artnum/039180!opendocument">IEC</a> and <a href="http://www.iso.org/iso/catalogue_detail?csnumber=31898">ISO</a>. ISO site has a better SEO department, so they've got a better Google position for their shop. Yes, the ISO and IEC are shops - the price to get official size of byte is ISO CHF 154,00 or IEC CHF 150. They really like to have accounts in Swiss banks for some reasons. I'd advice to sell those standard in Bitcoins instead - it is more profitable in a long term.<br />
<br />
<h3 style="text-align: left;">
ISO/IEC as Commercial De-Facto Authorities</h3>
Why do we need some organizations like ISO/IEC that place their name and put limitations around access to de-facto standards that more like any other information <a href="http://en.wikipedia.org/wiki/Information_wants_to_be_free">want to be free</a>? I'd say that our awesome decentralized and independent approach to develop what do you feel and support what do you want is just not widely exposed to those conservative oldschool bureaucrats, who still live in their own world of central authorities that should dictate people what to do.<br />
<br />
Don't get me wrong. There are conflicting points when you DO need to set a standard, and an enforcing organization like ISO/IEC is required (enforcing, because market force business to comply no matter how "recommended" the standard is). The costs of dealing with conflicting parties and convincing them is high, and that's why they set price on papers (the calculation if the price is fair is thankfully out of scope for this post). But the thing that bothers me more is that they set price on assessing the facts that are de-facto standards and common knowledge.<br />
<br />
The problem here that we separately don't have a tool to say something as whole. The problem here is that if anybody will try to speak for the whole net, the net would resist and that's natural. Because people tend to say too much in one phrase and they are too smart. The reason is to keep the facts short, clear voices from responsibility and make it all countable and strictly out of politics.<br />
<br />
<h3 style="text-align: left;">
What can be fixed here?</h3>
Usually people sign petitions. I propose to extend this just for fun. Make a technical statement that countries should agree on (no politics, please), give people an opportunity to support these openly, say if they don't want to support openly and give ability to support in closed manner (?respect privacy), the same way for disapprovals. Disapprovals may carry a reason. Once this data is in place, let people upvote and see what will happen.<br />
<br />
The statement - "the byte is 8 bits".<br />
<br />
Then build a list of countries that nationally accepted this statement. Then name this initiative somehow - it is important to keep this strictly technical to shot the zombies.<br />
<br />
Once a statement reaches some degree of exposure and human votes per country, country can decide to accept it by placing official signed statement online. Over the time, the statements can be combined into free will packs and signed too. This will allow to sync.<br />
<br />
Arguments. I am not sure they are needed for de-facto standards, because you reach consensus not by persuasion, but by collecting overall feeling. However, if there are problems with de-facto ways, and people feel there is something wrong, there should have an ability to "opt-in for a change" to upvote/downvote such arguments too. People should not be ashamed to set a value of "my butthurt" meter when voting or proposing counter-arguments, because we are irrational by our design, and technical problems with standards need more feedback than any other area of development on the butthurt effect.<br />
<br />
So, simple statements, public voting, open process, feddback, realtime status and summary on nation adoptions.<br />
<br />
<h3 style="text-align: left;">
Who should do this?</h3>
<div>
I'd be interested in working on this if I had some place to live in of my own. I'd start from contacting guys from Stack Overflow to reuse open source parts of their experience. Quite boring, right? Well, I am not saying that I want and plan to work on this alone. I am just saying that I am not in a position to take a role of coordinator. I just want to says that if you like the idea, maybe even in some crippled variant, found the resources to go, and want to try, feel free to ping me.</div>
<div>
<br /></div>
<div>
One of the tools that was really close and impressed was (now defunct) <a href="http://hammerprinciple.com/">http://hammerprinciple.com/</a> which helped me to discover bad things about my favorite version control and programming language without too much butthurt injury. Hopefully, it will strike back again.</div>
</div>
anatoly techtonikhttp://www.blogger.com/profile/07847778636782899903noreply@blogger.com0