Friday, August 15, 2014

ANN: sha1 1.0 - command line tool to calculate file hash

Windows doesn't have a native tool to calculate SHA-1, so I've made one that can be easily installed and used from command line on any operating system thanks to Python:
python -m pip install sha1
python -m sha1 <filename>

Saturday, May 31, 2014

Python 3.2 has some deadly infection

Another not really positive low quality post, so if you're not in a mood to hear some rants, just skip it.


I don't consider talks about different programming languages to be blasphemy. Quite otherwise - they are good to let you rethink what you do daily in your own language. In a recent discussion about languages (and after many fruitless attempts to compile Wesnoth with GCC on Windows) I thought that C/C++ must die, because really all major security problems are because of it. Memory problems, reinventing the wheels, wasting time in Makefile's and Autoconf / Visual Studio project files. Everybody is just forced to use it, people feel like they've become an elite by learning it, but in fact they are just wasting time. Well, you can't blame people for that. Everybody entertains in a way he can, and me while writing this post too.

Yes, early hacker's culture is still here, but is it here to hack on real world problems anymore? NumPy, SciPy, PANDAS, BioPython - this is the example of tools that modern hackers use. C is only useful to speed up parts of application and that's it. It is not designed to make app more secure, to save developer's time, to increase productivity, even to be hardware independent, and I believe the main reason for its popularity is that it is just the default for Unix. C for Unix is a Visual Basic for Windows. This doesn't make Visual Basic better - this makes C worse.

What about Python? Well, here are some stats.

These are stats from TIOBE (clickable). They don't measure anything - just show some lines that correlate to each other.

It looks like the peak of Python was on February 2011, and since then there was a significant drop. The next peak you see is February 2013 - month before PyCon in California, followed by a deeper decline. Let's see what happened on February 2011:

  • Python 3.2, documentation released on 20 February 2011.
Right. Everybody knew that Python 3.1 is going to be bad, and everybody expected Python 3.2 to be good, but that didn't happen. You may tell people anything - "don't expect", "it is planned", "it is better" - regardless if people hear you or not, they will still have some expectations, and if these expectations don't match the  experience , you see the "disappointment line" decline. You see the second peak didn't bring any good news either.

I am disappointed too, and if you don't want to read yet another boring complain, just skip this paragraph. I expected Python 3 to be ready for the internet age, with cross-platform behavior preferred over system-dependent one, with clear semantics to work with binary data, with non-confusing error handling and API that gives hints and helps to pave your way on uncharted grounds, such as abstract unicode. Maybe Python 3 is beautiful, but I fail to see how - it is not more modular than Python 2 - I can not switch features of the language on and off to taste experimental stuff of propose my own, I don't see any pictures of improved internal structures or . I didn't expect more things to work by default, but I hoped that explicit interfaces continue to be intuitive. There are many little things that sum up and spoil the fun like trailing whitespaces in JSON output that play significant role in providing backward compatibility for Python. I don't know if that is fixed in Python 3, and I don't have any means to track that. When these little things sum up, you realize that you're just wasting time trying to improve things that people don't want to improve. They don't want to improve the process. They don't realize that the problem is not in the language, but in the way they don't want to hear each other. Technology showed that people want to be heard, that they opinion should be  accounted  , not closed as  won't fix  , or  works for me  . It is not a community process, when you rely on abilities of certain individuals to monitor and respond to all traffic and wishes, especially when they fail to do so.

Community process is when there are tools that people can collaborate about, there is research, a place for sharing opinions and sum up the result, a time to track the progress and do the work, and an open environment to experiment and evolve. Where every opinion counts and adds up to become a signal. Where signals are reacted to not because there are people who feel responsible, but because it is fun and has a positive feedback.

There is a certain paralysis. Many people are stuck in corporate against their will, bound by economy. But I am sure in every company it is possible to dedicate a "SyncFriday" - day dedicated to sync with other developers outside to do things in open source with all necessary stuff prepared. If a company can not afford it - why not just leave it. What's the point in helping corporation that doesn't make the world better from your own point of view too?

And much as with corporations, I don't see why anyone should be interested to help Python 3.x to become better. It takes too much energy and is so hopeless, at least for me. Singing papers, writing long explanatory PEP notes, discussing things with your core boss. There are no stats, no summaries, no analysis and comparisons, root cause research - only a lots of text and opinionated and busy individuals like myself.

Can this be changed? Not with text. More people who can draw, animate, make beautiful graphics work to make hard problems visible. This will help. Work with people expectations - if these are unreal - do not try to change them, explain them. Make the problems visible, give people the freedom to try, do not demand, do not press, just create environment, and do not use text - it is useless. 

Art is the future. And the future is both visually appealing, nice to touch and easy to follow.

Tuesday, April 29, 2014

Writing SCons Plugin: Discovery

SCons is a build tool written completely in Python, so that you can just put it into your repository, extend for your own purposes and run directly from checkout without installing anything. SCons documentation doesn't use word plugin to refer to extensions. It calls them Tools. If you're not familiar with SCons concepts, check out this wiki page.

Plugin Discovery


SCons tries its best to avoid magical behavior. That's why it ignores paths and options set in system environment and requires that you specify everything explicitly. The same is true for plugins. Example from SCons man page:
env = Environment(tools = ['default', 'foo'], toolpath = ['tooldir'])
This creates build environment, initializes default tools, sets lookup path for plugins to tooldir/ and enables tool named foo, which is located in tooldir/foo.py. foo.py should have two functions - generate(env) and exists(env).

To test that your tool is found correctly, check env['TOOLS']:
print(env['TOOLS'])
If filename is not there - SCons was unable to find it. If it was a problem with contents of foo.py - SCons would fail with exception.

SCons has an automatic tool discovery mechanism - if you don't like to specify toolpath directly - you may place your tool in one of several locations that SCons scans before executing SConstruct. See description of the --site-dir=dir option for details.

Friday, March 28, 2014

Python C API/ABI compatibility report

UPDATE: There is now an official thread on python-dev.

Upstream Tracker is an open source (GPL) tool that allows to track API/ABI changes between releases of C/C++ libraries. I asked Andrey Ponomarenko, who is the main maintainer of the project, to add Python to the list and here is the result:

http://upstream-tracker.org/versions/python.html

Hope you like it.

Monday, February 17, 2014

ANN: xtrace 0.5 - indented function trace in Xdebug format

Xdebug is a PHP tool that allows to trace how PHP code is executed. Today I release a tool called xtrace into public (and more specifically into public domain), which allows to get the same (or at least very identical) output, but for Python.

Few years ago inability to trace function calls in Python came as a showstopper. I decided to write a familiar tool for Python. More than that - I wanted to integrate it into Spyder. But a year ago xtrace itself faced with showstopper. The showstopper was the behavior of execfile function, that I could not get right at that time, because of the docs, of my expectations and poor knowledge of English. Maybe there is a flaw in my cognitive abilities, but I tried to get at this problem several times and failed. Until recently some hackers from ZenSecurity team brought a concept known an pyjail to my attention. The challenge to prove that pyjail concept is impossible allowed me to concentrate on gory details of execfile works and knowing that documentation is totally confusing for my, I found the time to set my own experiments. You can read them at the link I've given above as well as some analysis why documentation that actually includes all the details can be bad and confusing.

The xtrace was basically broken for three years, starting from the version 0.2 - the day I put execfile() call from root to the xtrace module to the main() function. This placement changed the execfile() behavior, and while trying to debug that I also run into confusing dynamic behavior of dictionary returned by locals(). Opened can of worms made those parasites to completely consume my brain, causing much anger and frustration to be spilled around execfile() and locals() concepts over into Python lists. It is kind of relief now that I can name all the problems, analyse them and look back as enlightened. Being jobless I had a plenty of time to investigate, but I really don't want anyone to enter that state of confusing and helplessness that I had a year ago.

Hopefully, my experience with xtrace will clear the confusion for those who will try to use exec type abilities of Python for developing their own tools. Maybe it will result in a better Python API in the future, with better documentation and position-independent behavior.

I am interested to know the feedback that you can leave in xtrace tracker, such as if the output really matches PHP behavior, if it is accepted by PHP tools and how it behaves in different scopes of Python. It is interesting to convert it to Spyder plugin and see the usage in other tools, but I realize that I may not have time for that. The next focus for me is to add an easy API to xtrace to enable people to write they own tracers more easily. Focus on UX and everything else will come.

Sunday, February 02, 2014

ANN: hexdump 2.0 - view/edit your binary with hex tool

https://pypi.python.org/pypi/hexdump

Finally some prod that can be named feature-complete for release. It is cross-platform, meaning it should run the same on Windows (tested), Linux, and OS X. It is Python 2 and Python 3 compatible. And it released into public domain, so that you won't have any problems in reusing it for your commercial and non-commercial hacking.

For those who are unaware of what hexdump is, hexdump is a representation of any binary data in human readable form. This form is good for hacking, inspecting and debugging binary data and protocols, but it is also good for editing such data. I am not pasting the output if the tool to encourage you to play with it yourself.

It can be used as command line tool and as a library. The most simple way is to use it as a tool:
# install
$ python -m pip install hexdump

# dump
$ python -m hexdump binary.bin > dump.txt
...

# restore
$ python -m hexdump --restore dump.txt
...

P.S. I don't mind including `hexdump` as provisional package in Python standard library if anyone will be able to convince PSF to accept public domain, CC0 or MIT licensed code.

Friday, January 10, 2014

Draw a pixel with PySDL2

This is a minimal code to output pixel on the screen using PySDL2.

UPD: (March 2014) Up for PySDL2 0.9.0 (RenderContext renamed to Renderer)
#!/usr/bin/env python
"""
The code is placed into public domain
by anatoly techtonik <techtonik@gmail.com>
"""
import sdl2
import sdl2.ext as lib

lib.init()

window = lib.Window('', size=(300, 100))
window.show()

renderer = lib.Renderer(window)
renderer.draw_point([10,10], lib.Color(255,255,255))
renderer.present()

running = True
while running:
  for e in lib.get_events():
    if e.type == sdl2.SDL_QUIT:
      running = False
      break
    if e.type == sdl2.SDL_KEYDOWN:
      if e.key.keysym.sym == sdl2.SDLK_ESCAPE:
        running = False
        break