Program config as a DNA strand

This is a technical followup to the post about mind-altering programming languages, which concentrates, iterates and extends on the abstract DNA part.

Do you know how DNA looks like from the point of view of Python programmer?
DNA is a list. Of genes, and fillers in between.

    [ GENE, GENE, GENE, None, GENE, STUFF, GENE, .... ]

By the analogy with software, one GENE is one option. Here None or STUFF is something that is not recognized and not used in the process right now, but may be used later even in a different place for a different purpose. This leads to very interesting properties of configuration described as a DNA strand.

Property One: Everything is Optional and Configurable

Usually it is a tough choice what to include into your program config, because an option a day keeps users away. But with DNA config it doesn't matter - you can encode everything - every bit of flexibility you wanted, every feature-creeper, forward-thinking matter. Don't bother yourself with restrictions - include everything.

Property Two: Indefinite Logical Groups and Namespaces

To make sense out of tons of information in a DNA strand, you need to concentrate only on the parts that are relevant to you. This is done by "masking" (or marking) parts of the DNA code according to your current task. When you apply a mask, all GENEs of DNA that are not used are cleared. You can have multiple masks for different programs. An awesome feature-creeper would be to figure out mask automatically during the program run (program explicitly marks sequences what it reads and what it skips).

Property Three: Non-linear Option Identification

Because DNA is a list, you may think that you need to know option position to access it. This is not necessary. Index in a DNA is the most straightforward way to lookup an option value, and mask is a tool to help explore and identify its position. But the true power comes from the fact that a option can be identified by just analyzing content. Option can be a single GENE in specific format, or it can be identified as a sequence of GENEs that pass some validation logic. This makes is possible to write generic configuration analyzers, which can read these sequences to detect configuration problems, patterns and meaningful values to produce nice visualization out of them.

Property Four: Application Types

In real world™ DNA is packed in chromosomes. Humans have 46. Potatoes possess 48. Zombies are somewhere in between. In a virtual world of coding it makes sense to use one chromosome per application type. For example, every Python web framework has a lot of common parts, common features and minor details in these features, so a variety of such frameworks can be encoded with a single DNA strand.

Exercise

It may sound too abstract, but try to think how to build a shared DNA strand for two or more mini web application frameworks in Python. Just remember that DNA code includes both hardcoded application features and modifiable configuration options. And as usual look at BoxCar 2D for an inspiration.

Multidimensional programming

From time to time I reverse some piece of open source code just to understand how it works. The biggest problem with that is the amount of things I need to juggle in my head until they find a place on the canvas of reversed blueprint. If there are too many - I give up or put the project on the back burner. This was with Stackless and Twisted, but I really glad to finally get to them.

Quite often is starts with some bug that seems easy to fix and I go for it. In ideal world it shouldn't take more than 15 minutes for studying the code to gain understanding what should be changed, but it is also important to have a confidence that the change won't break anything.

I won't tell you how to deal with that complexity, but I'd like to share an idea that I've got from Large Hadron Collider. =) Let me tell you that story...


I always had troubles trying to squeeze more than 3 dimensions into my head. I could imagine a dot moving along X or Y axes easily, could imagine it moving along Z axis, but everything more than that caused a confusion.

I could have my brain exploded when some time ago a friend of mine tried to explain what's going on in LHC. He said that physics and mathematicians are trying to figure out how many dimensions our universe has. They assume that our universe has more than 4 dimensions (3 coordinates and 1 time value). In fact they argue that the truth is somewhere between 9 and 26!

I am not a fan of scientific theories - my neurons are pretty calm to that matter, but when you just need to understand, because the person in front of you is knowledgeable and tries all the best to explain  - here is when you start to feel colliding brain cells under your skull. To the honor of my friend, he didn't try to build-up suspense (as I would probably do before explaining properly) and managed to draw a clear picture in my mind. A standard plot with two axes - X and Y, and a dot.

"Look at this dot on X/Y plane. This dot has coordinates in two dimensions - X and Y", - he explained. "These are only two you can see here. But this dot is from the real world, so it also has Z coordinate", - he added putting a label "Z=0.1" next to the dot. "This dot also have speed", - he drew "V=0.1 m/s". "Now we can see values for all 4 traditional dimensions of our dot, but we can add more. There are many things that we can classify our dot. It probably has color.", - he drew a color box and a hex value. "It has temperature", - a new label "t = 25С" appeared in the column. "There are 6 already, and you can add your own.". I immediately imagined a dot travelling through canvas of "Stars!" game with all those labels nearby that constantly change their values as the dot moved. "Wow! Now I see", - that was a nice feeling - I must admit I can be pretty dumb sometimes. =) It didn't make a science fan out of me, but did make an important short-circuit in the depths of my head, which popped up a few months later..

A few months later.

As it usually happens in software development you need at least some basic design before setting down to code, and with time design phase completely faded from our process into "who made it first - wins" motto. It is hard to argue with that, so I didn't. But as a result after some time, the stability of our releases dropped. People were not communicating, and started to forget about some aspects of our system that could fire at any time in completely unexpected places. Even though all our code undergoes reviews, the reviewers tend to forget about those aspect as well. The process went out of control - we couldn't keep all the aspects in our heads when planning for the next feature to be released, and I found myself in same state I was when trying to understand the complexity of multidimensional string theories. That lead me to the idea that we need to control the amount of aspects that we need to keep in mind when coding and restructure our architecture to keep those aspects at minimum. This will help us to regain sense of confidence into what we are doing, and save some money from the bills from the nearest bar on the name of our release manager.


So, the `multidimensional programming` concept means that at any point of your code there are multiple things that you should be aware of. These things are the 'dimensions', and the more you have - the more complex your application is. Basically, these are the things that can be broken by any change at this place. Good application architecture is orthogonal - you can work at only one dimension at a time without thinking too much about all others. But you need to know all of them anyway to gain a sense of confidence.


For example, a recent change in Spyder IDE requires me to rename some file. This breaks the code, which I grep and fix - that's one dimension. But it will also likely to break a translation for the strings in that file, because the string is now at a different place. I imagine that nobody will be interested to check and translate the same stuff over and over again, so that's one more thing I'd like to avoid, so I should keep that in mind.

Another example are web applications. You need to keep in mind 'user privileges', type of HTTP request ('ajax', ...) and response ('json', ...) required. You need to make sure `critical errors` are handled and reported, and `static files` are correctly served by web server. You need to save incomplete `data between requests`, and cleanup it where possible. Make sure there is sufficient `XSRF protection` and `browser compatibility`. There are a lot more to it, and so far these have nothing to do with the logic of your web application. Frameworks help to deal with that, but inside they are still multidimensional. If framework is not flexible for you - that probably means it tries to keep some dimensions orthogonal, and there could be a good reason for that.


Maybe that's not much, but at least now you can argument that Large Hadron Collider experiments have much in common with software engineering, and when somebody asks about your job - you can proudly state that you're on par with scientists with their string theory, but in your own big enterprise application universe. =)

Bloodless partition backup

Sometimes it may happen that a system blackbox arrives at your desk together with a person begging to do something to make it work again. It may have been hardware failure, power outage, virus or just a prank that made this panicked person pull the plug out of the outlet and shamefully accept that "I am afraid to turn it on after that!!" What should be the first reaction? Backup what is left. There are tons of backup software available, but most of it just copy files from one place to another on a regular basis. What you need in this situation is to copy a whole disk to network drive without booting the system preferably on the same hardware to avoid mounting HDDs. This is a task for disk cloning tools.

And while it seems logical to think about Acronis True Image or Norton Ghost on the second thought they are not really great when you need quick burn-and-use ISO image that they are unlikely to provide in their downloads section. So the best way is to check what open source has to offer. There are several tools:

• Clonezilla


• FOG - Free OpenSource Ghost


• g4u - Ghost for Unix


• G4L - Ghost for Linux


• LRS - Linbox Rescue Server


• Mondo Rescue


• PartImage


• Trinity Rescue Kit

The list of 8 product seems enough to choose one. Let me remind that we need ready-to-run Live CD capable to save partition or disk image to a remote network location. Throw out Mondo - it doesn't provide Live CD. TRK 3.2 also goes to the shelf - this very nice Boot CD doesn't include tools to backup/restore partition to a network drive. A pity. Linbox Rescue Server - free GPLed version can't backup NTFS partitions - won't work. FOG - project named "Free Ghost" doesn't have Live CD! and even with such promising name is designed solely for distributing the same disk image to multiple workstations over the network. Removed. So what is left:

• Clonezilla


• g4u - Ghost for Unix


• G4L - Ghost for Linux


• PartImage

After browsing these sites for a bit the first intention was to get rid of tools with ugly web-pages and short feature list. And it is where g4u beats them all. How can two floppy images in total 3Mb long compete with the raw power of 90Mb image of Clonezilla? Well, let's see. It was very easy to download and run g4u 2.3 from VirtualBox. Boot message gives almost all necessary information to go on except it doesn't say that you need user "install" on your FTP server. FTP is the only supported method for upload/download, but it's ok. Because of simplicity this tool had all the chances to be the chosen one of this review. Even in Windows you can easily setup FTP server given Far Manager, Python and this plugin. g4u is a tool that does the job of backing up partition, whole drive and compressing it via GZIP, but!:

1. There is no command history - and you have to retype whole command every time you've made a mistake
2. There is no anonymous FTP login option - extra hassle to setup "install" FTP user
3. Empty password is not allowed
4. No progress bar with total MB/percentage completed, not speaking about estimated time
5. Rather obscure GZIP option, which basically means compression level (default is 9 - the biggest)

To end up with g4u - an ideal approach for this awesome and minimal tool with limited usability would be to grow its ISO to 10Mb disk image by including Python and provide some wizards in .py

Next is Clonezilla - wizard interface. No FTP. Only SSH, SAMBA or NFS server that needs to be mounted for an image file. For folks without Linux experience the most complicated stuff is probably to understand "mount clonezilla image directory" dialog which basically means to attach "share" where image files can be stored (sorry for the lame comment). Saving partition image worked like a charm - not only it displayed progress window, but also detected space in use! and saved only used part. Magic! I just wonder if I'll be able to restore image manually in case Clonezilla fails. In directory with image file there are text files with disk and partition parameters, but image file itself is in custom format. In case of NTFS system - in ntfsclone 2.0 To conclude - Clonezilla version 1.0.9-10 CD is nice and usable, but a single mistake during wizard requires to start the process anew. Let's skim over alternatives.

Ghost for Linux - doesn't have its own web-page - only standard SourceForge project template, but provides ISO image for downloads. The project itself is similar to g4u - the same FTP upload method with compression, but much better interface. Version 0.24 is GPLed and Linux-based. In comparison g4u is BSD-licensed and NetBSD-based. Ok, about the contents. Multiple boot options varied mostly by linux kernel. Weird. I do not want to try every one to find out which of them will do the job. Good reference is shown right after startup documenting two options, but lazy one would prefer only the first - to enter "g4l" at command line to start wizard interface. This preferably could be just one line reminder before the system proposes shell. The nice thing about g4l wizards is that they allow to go back to correct options. Nevertheless, the real test failed on VirtualBox, because there was no default 'img' directory on FTP, and even though g4l created it, it hung afterwards. After manual restart and mkdir 'img' on FTP the process successfully completed. It can be FTP or VirtualBox bug - no idea, but with this minimal correction the test passed. Resume: setup is minimal and intuitive, though partition selection doesn't give any info about partition type. Ideal for FTP backup, but for other network access types I would still prefer Clonezilla. g4l like Clonezilla includes ntfsclone utility for NTFS backup, but its version is slightly outdated and should be selected explicitly. If you remember - there is no partition information displayed.

PartImage doesn't have its own Live CD, but seems to be the tool of choice used on PING "Backup and Restore Disk Partitions" Live CD and on SystemRescueCd. PING 2.01.10 is even more simple than Clonezilla, includes wizard interface, for network storage supports only SMB shares and requires that a predefined set of directories is already created in the destination share. This predefined set is only described on official site and that's strange, because Clonezilla manages to create all required directories automatically. So PING is not an option for burn-and-boot solution. Another annoyance is that Alt-Tab pressed within VirtualBox ends PING session prematurely. Too bad. Let's see the last one. At last!

SystemRescueCd is a whole bunch of useful tools for plumbing works. However, even with the nice help displayed at startup knowledge of *nix and the tools is still required. If I haven't used Clonezilla or g4l I had no chance to know that I needed ntfsclone. In addition I must admit I still do not remember what cmd I have to type to mount SMB share. SystemRescueCd may be the best Rescue Live CD, but it is not as convenient and intuitive as more specialized tools. As a typical lame windows user I've failed to use it properly. I.e. for backup, but there is still awesome utility named 'gparted' for disk partitioning that I would recommend to everyone. In fact there is a very awesome project that combines SystemRescueCD and Clonezilla Live CD in one disk. Grab it here - you won't regret. =)

The verdict.
To backup your disk or partition via SSH, SAMBA or NFS use Clonezilla. For convenient backup to FTP use g4l.

That's all. Enjoy!

Make Java program work through the proxy

It is not surprising that not many users know how to make a Java program work through a domain proxy if there is no place to enter proxy settings. Just because they are not developers they do not know that it is enough to launch this program with the following command line:

java -Dhttp.proxyHost=192.168.1.1 -Dhttp.proxyPort=3128 -jar SoftWare.jar %*

This works for anonymous or domain proxies only, because there are no password settings. To work through password-authenticated proxy, you will have to setup additional local or personal proxy that uses user/pass settings to authenticate and pass traffic to upstream one. Unfortunately, I can't name any software for user/pass authentication because I've never had to work with this problem, but I know that at least Privoxy is capable to forward requests.