Not so long ago I decided to play with launchpad - site for open source collaboration. It is built around decentralized Bazaar version control system that reached version 1.6 in its development up to this moment. I read that centralized VCS are bad and DVCS are good, but never had enough free time to check this in practice and finally..
The first thing I liked in Bazaar is that it is written in Python - that means that bugs can be found and fixed rather easily. This was also the thing that I disliked, because standalone bzr distribution includes Python and takes 14Mb and package for installed Python distribution doesn't add bzr to PATH
Accessibility. I often work through tunnels, proxies, firewalls and other stuff that kind people tend to put here and there in each Local Area Network. Not only I have to deal with them myself, but sometimes also need to teach others. In SVN proxy settings are stored in documented configuration file called 'servers'. In Bazaar there is no such file and all configuration is done with environment variables and Python settings that are not documented. To use Bazaar through a proxy - set environment variables:
set http_proxy=127.0.0.1:1080
set https_proxy=127.0.0.1:1080
These proxy settings are rather common for linux/unix machines, but cause particular pain with windows machines in a domain. If SVN is able to transparently authenticate against domain proxy without asking sensitive passwords then Bazaar lacks such feature and needs another authorization proxy server (NTLMAPS) middleware.
Usability. In general distributed version control systems should be better than centralized ones. A lot of articles and tutorials describe the theory very well, but in practice there are many things is are not that brilliant. For example, I needed to fix stubborn DevPak plugin for Code:Blocks that didn't want to install one specific version of a curl library. In SVN I would do a partial checkout, make modifications and upload patch to the forum. Not very convenient. In Bazaar I would have to make a branch of the whole Code::Blocks repository, start patching plugin and then merge the whole branch back. But branching the whole repository is an overkill for such little plugin and you can't make a branch out of subtree in Bazaar. More then that - Bazaar doesn't even allow to export subtree of repository. Download either the whole thing or nothing. Not too effective, especially when the traffic is 0.02 cents for Mb.
So, I manually created directory structure for the plugin similar to existing project and started a new Bazaar repository from scratch. After a couple of revisions I released fixed plugin for stable Code::Blocks and would like to switch to latest development code and merge changes made to plugin in trunk into my version. Again, Bazaar disappointed me. I knew that SVN won't allow merging with different repository, but at least I could grab files from the source tree and merge them manually. Bazaar didn't allow me even that. With almost 100 commands it just couldn't export some files.
It may be that Bazaar is a good system for some small projects, but in my case it appeared unwieldy and too expensive even being open and free.
Wednesday, August 27, 2008
Wednesday, July 02, 2008
Applying Unified Diffs with Python
Windows has a lot of annoyances for developers. One of these is that it lacks some precious tools - namely "diff" and "patch". They can be downloaded from the Internet, but when the latest patch binary provided by Win32 ports of version 2.5.9 refused to apply a patch built with "svn diff" and closed with an error, I decided to write my own version in python. If it will be included in standard python distributive as a logical complement to Scripts\diff.py utility then at least for people with python there will be no problem with applying patches in windows. One limitation though - the script parses only the most popular format of patches - unified diff.
To start out I've outlined a structure of unified diff using information from Guido van Rossum blog and wikipedia.
Parsing logic is implemented using brute-force regex parsing approach to avoid dependencies on parsing libraries (like pyparsing etc.). I took this approach to compare the code with the different techniques of Text Processing in Python by David Merz and learn how can I improve it.
Linefeeds are handled in automagic mode. Proper line ending is detected during scanning of source file. If source file has mixed line endings - lines from patch file are not transformed and written "as is". If lines in source files end with the same sequence - lines from patch file are stripped of their own line ends and applied.
The project doesn't have all UNIX patch options, but should be useful even without them. You may find it with sources (MIT license) at http://code.google.com/p/python-patch/
To start out I've outlined a structure of unified diff using information from Guido van Rossum blog and wikipedia.
Parsing logic is implemented using brute-force regex parsing approach to avoid dependencies on parsing libraries (like pyparsing etc.). I took this approach to compare the code with the different techniques of Text Processing in Python by David Merz and learn how can I improve it.
Linefeeds are handled in automagic mode. Proper line ending is detected during scanning of source file. If source file has mixed line endings - lines from patch file are not transformed and written "as is". If lines in source files end with the same sequence - lines from patch file are stripped of their own line ends and applied.
The project doesn't have all UNIX patch options, but should be useful even without them. You may find it with sources (MIT license) at http://code.google.com/p/python-patch/
Tuesday, June 10, 2008
Yet another CVS to SVN transition
Some time ago I've compiled a short comparison of CVS vs SVN. It should not be hard to get the idea from it that SVN is better - easier to understand, easier to setup and to use. While there are many new decentralized systems around like Git, Darcs and Mercurial, SVN still has the best windows working clients that are even able to transparently authenticate against domain proxies. That's why SVN is the most accessible way of getting sources.
Many projects realize that and make switch to SVN to expose their public repositories to a wider audience even if some conservative developers are unwilling to change their habits and tools. Still the convenience that brought by SVN worth the efforts of rewriting tools and moving legacy codebase over to new rails. CVS deserves to be honored a special award in the history of software development, but perspective open source projects realize that evolution never stops and time comes to replace legacy CVS with tools that value developer's time. Here is a list of such projects that made the decision to help with yours:
Many projects realize that and make switch to SVN to expose their public repositories to a wider audience even if some conservative developers are unwilling to change their habits and tools. Still the convenience that brought by SVN worth the efforts of rewriting tools and moving legacy codebase over to new rails. CVS deserves to be honored a special award in the history of software development, but perspective open source projects realize that evolution never stops and time comes to replace legacy CVS with tools that value developer's time. Here is a list of such projects that made the decision to help with yours:
- May 5, 2005 - KDE
- June 7, 2006 - pygame
- September, 2006 - ArgoUML
- December 14, 2006 - SWIG
- January 1, 2007 - GNOME
- January 24, 2008 - phpBB
- June 2008 - FreeBSD
Monday, May 26, 2008
CGI Python scripts on Apache for Windows
Just a quick tip how to make a CGI script running on Windows under Apache. The main problem with CGI scripts is that their first line usually contains a path to executable program - Python interpreter - and if on *nix platform this line is more or less the same:
#!/usr/bin/env python
in windows it usually different from one installation to another. If in *nix environment you do not have to change this line most of the time, in windows most of the time you'll need. Luckily, Apache developers invented an option to lookup the path for script interpreter from the registry by following association of file extension. Look at this .htaccess for example:
Here ScriptInterpreterSource Registry is the magic phrase to turn on the lookup. It is well described in Apache manual. FilesMatch is another directive to limit the scope of lookup to .py files only.
Finally, a reminder of how a minimal CGI script in Python looks like.
#!/usr/bin/env python
in windows it usually different from one installation to another. If in *nix environment you do not have to change this line most of the time, in windows most of the time you'll need. Luckily, Apache developers invented an option to lookup the path for script interpreter from the registry by following association of file extension. Look at this .htaccess for example:
Options +ExecCGI
AddHandler cgi-script .py
<FilesMatch "\.py$">
# Use the interpreter found in registry by file association
ScriptInterpreterSource Registry
</FilesMatch>
Here ScriptInterpreterSource Registry is the magic phrase to turn on the lookup. It is well described in Apache manual. FilesMatch is another directive to limit the scope of lookup to .py files only.
Finally, a reminder of how a minimal CGI script in Python looks like.
#!/usr/bin/env python
print "Content-Type: text/html" # HTML is following
print # blank line, end of headers
print "xxx"
Tuesday, May 06, 2008
Writing Far Manager plugin in C
Far Manager is a file manager for Windows, however the most functionality in Far is contained in plugins. Here is a short introduction how to create one.
You'll need Far Manager with Development Pack, GCC compiler and Developer's Encyclopedia at hand. Encyclopedia is also available in Development Pack in .chm format.
Plugin is a .dll file compiled from .c source. For the minimal example you'll need to create .c file with at least one function GetPluginInfo() that tells Far about plugin capabilities.
This Sequence Diagram illustrates communication of Far with a plugin that does absolutely nothing. The source of the plugin is below:
Encyclopedia states that StructSize should be filled with the size of PluginInfo structure to maintain backwards compatibility in case of future API changes. Because plugin really does nothing, it is impossible to tell if it works or not. To prove it really works let's add logging to file.
Copy "plugin.hpp" near to mini.c and execute GCC to compile the plugin:
gcc -shared -o mini.dll mini.c -Wl,--kill-at
Place mini.dll into plugin path or execute Far with /p parameter pointing to directory with mini.dll Press Ctrl-R and look for "minilog.txt" file in current dir containing burning proof that plugin works. This is enough to get started and follow Developer's Encyclopedia on your own, there are still some technical details that may answer some questions about GCC options and .dll writing not covered in Encyclopedia.
GCC parameters shown above instruct it to compile mini.c into shared library mini.dll To be recognized as a plugin the .dll should make GetPluginInfo() function visible to Far (i.e. exported from the library) the same way as any other function that Far calls in plugins. By default all functions in .dll are exported and this is visible in .dll as export table that can be checked for example with BIEW. The list of exported functions can (and usually should) be narrowed to increase performance and clean API either with .DEF file or with __declspec(dllexport) addition to function prototype. Before going with example there is one more thing left to explain - -Wl,--kill-at parameter.
By default function names are exported with "at" suffix like GetPluginInfo@4 where 4 denotes the number of bytes the argument takes. When looking for plugins Far looks for clean names without @4. The last option to gcc -Wl,--kill-at is required to strip "at" suffix from exported function name. Read this link for more details about this calling convention.
To illustrate how __declspec(dllexport) works we move logging code into separate function and compile it with:
gcc -shared -o mini.dll mini.c
If you now launch BIEW on mini.dll and press Alt-F3 to get to export table, you'll see two exported functions - logfile and GetPluginInfo. Latter with @ suffix. @ is added by WINAPI calling convention.
To remove suffix recompile plugin with:
gcc -shared -o mini.dll mini.c -Wl,--kill-at
To leave only GetPluginInfo() function in export table - add __declspec(dllexport) to its definition. If __declspec(dllexport) is present in at least one functions definition, all other functions that doesn't have this tag will be excluded from export.
Inspecting .dll to see if the "at" suffix is gone (thanks to GCC options) and only one function is present in export table (the one marked with __declspec).
There are four basic export functions in Far Plugin API that come handy at start. Plugins export these function to let Far call them to supply information about itself and gather data about plugin. Functions (if present) are invoked in particular order which is illustrated in the following table.
With this yet another lame Sequence Diagram there should be enough useful information to get started. Among improvement that could be done to mini.c code is to replace stdio.h library calls with native Windows API and Far API calls, adding help, menus and language files. However this falls out of scope of this post, so the best way to move further is to checkout Encyclopedia. Good luck!
You'll need Far Manager with Development Pack, GCC compiler and Developer's Encyclopedia at hand. Encyclopedia is also available in Development Pack in .chm format.
Plugin is a .dll file compiled from .c source. For the minimal example you'll need to create .c file with at least one function GetPluginInfo() that tells Far about plugin capabilities.
This Sequence Diagram illustrates communication of Far with a plugin that does absolutely nothing. The source of the plugin is below:
#include "plugin.hpp"
void WINAPI GetPluginInfo(struct PluginInfo *Info) {
Info->StructSize = sizeof(struct PluginInfo);
}
Encyclopedia states that StructSize should be filled with the size of PluginInfo structure to maintain backwards compatibility in case of future API changes. Because plugin really does nothing, it is impossible to tell if it works or not. To prove it really works let's add logging to file.
#include <stdio.h>
#include "plugin.hpp"
void WINAPI GetPluginInfo(struct PluginInfo *Info) {
FILE *file;
file = fopen("minilog.txt", "a+");
fprintf(file, "%s\n", "Fired.");
fclose(file);
Info->StructSize = sizeof(struct PluginInfo);
}
Copy "plugin.hpp" near to mini.c and execute GCC to compile the plugin:
gcc -shared -o mini.dll mini.c -Wl,--kill-at
Place mini.dll into plugin path or execute Far with /p parameter pointing to directory with mini.dll Press Ctrl-R and look for "minilog.txt" file in current dir containing burning proof that plugin works. This is enough to get started and follow Developer's Encyclopedia on your own, there are still some technical details that may answer some questions about GCC options and .dll writing not covered in Encyclopedia.
DLL, exports and GCC options
GCC parameters shown above instruct it to compile mini.c into shared library mini.dll To be recognized as a plugin the .dll should make GetPluginInfo() function visible to Far (i.e. exported from the library) the same way as any other function that Far calls in plugins. By default all functions in .dll are exported and this is visible in .dll as export table that can be checked for example with BIEW. The list of exported functions can (and usually should) be narrowed to increase performance and clean API either with .DEF file or with __declspec(dllexport) addition to function prototype. Before going with example there is one more thing left to explain - -Wl,--kill-at parameter.
By default function names are exported with "at" suffix like GetPluginInfo@4 where 4 denotes the number of bytes the argument takes. When looking for plugins Far looks for clean names without @4. The last option to gcc -Wl,--kill-at is required to strip "at" suffix from exported function name. Read this link for more details about this calling convention.
To illustrate how __declspec(dllexport) works we move logging code into separate function and compile it with:
gcc -shared -o mini.dll mini.c
#include <stdio.h>
#include "plugin.hpp"
void logfile(const char* msg) {
FILE *file;
file = fopen("minilog.txt", "a+");
fprintf(file, "%s\n", msg);
fclose(file);
}
void WINAPI GetPluginInfo(struct PluginInfo *Info) {
Info->StructSize = sizeof(struct PluginInfo);
logfile("GetPluginInfo called.");
}
If you now launch BIEW on mini.dll and press Alt-F3 to get to export table, you'll see two exported functions - logfile and GetPluginInfo. Latter with @ suffix. @ is added by WINAPI calling convention.
To remove suffix recompile plugin with:
gcc -shared -o mini.dll mini.c -Wl,--kill-at
To leave only GetPluginInfo() function in export table - add __declspec(dllexport) to its definition. If __declspec(dllexport) is present in at least one functions definition, all other functions that doesn't have this tag will be excluded from export.
#include <stdio.h>
#include "plugin.hpp"
void logfile(const char* msg) {
FILE *file;
file = fopen("minilog.txt", "a+");
fprintf(file, "%s\n", msg);
fclose(file);
}
void WINAPI __declspec(dllexport) GetPluginInfo(struct PluginInfo *Info) {
Info->StructSize = sizeof(struct PluginInfo);
logfile("GetPluginInfo called.");
}
Inspecting .dll to see if the "at" suffix is gone (thanks to GCC options) and only one function is present in export table (the one marked with __declspec).
Further Steps
There are four basic export functions in Far Plugin API that come handy at start. Plugins export these function to let Far call them to supply information about itself and gather data about plugin. Functions (if present) are invoked in particular order which is illustrated in the following table.
| Name | Required | Order | Comment |
| GetMinFarVersion() | no | 1 | called first |
| SetStartupInfo() | no | 2 | always called if present - good for extra initialization code |
| GetPluginInfo() | yes | 3 | cached |
| OpenPlugin() | no | 4 | not required, but without it plugin is pretty useless |
With this yet another lame Sequence Diagram there should be enough useful information to get started. Among improvement that could be done to mini.c code is to replace stdio.h library calls with native Windows API and Far API calls, adding help, menus and language files. However this falls out of scope of this post, so the best way to move further is to checkout Encyclopedia. Good luck!
Tuesday, April 29, 2008
CVS vs SVN
Open Source for many developers means getting the hands dirty in many projects simultaneously sending patches here and there as they polish their own way through the code to make sure the bug encountered today will not pester anyone in the future. Everybody seems to be interested in reporting the bugs and sharing patches, but in reality it doesn't happen too often, mostly because of lack of time necessary to contribute. Following modern enterprise tendency to define various measures and estimates let me call this parameter as Time-To-Contribute value. TTC is directly influenced by such factors as activity of developers and availability of the code as well as usability and knowledge of development tools.
It would be interesting to get deep into the details of code contributions, but to keep the long story short and justify the title let's compare SVN and CVS. These tools provide access to source code and poor decision may affect developers experience and desire to contribute in the future. This post based on RFC submitted to PHPDOC community a year ago, which may be useful for other conservative CVS parties out there like MinGW
Facts only.
++++ Accessibility:
CVS usually blocked by proxies (needs dedicated port)
SVN works over HTTP and HTTPS via WebDAV
CVS doesn't work behind a proxy
SVN works with proxies
CVS checkout is complicated, it is hard to remember all the prerequisites
cvs -d :pserver:cvsread@cvs.php.net:/repository login
cvs -z3 -d :pserver:cvsread@cvs.php.net:/repository checkout -P phpdoc
SVN project checkout command is easy to remember
svn co http://svn.php.net/repository/phpdoc
CVS is complicated to learn
SVN has a perfect book
CVS is abandoned by developers
SVN is supported
++++ Security:
CVS password is transmitted over network in cleartext with simple rot13-like translation
SVN works over HTTPS, supports Apache authentication schemes
++++ Usability:
Command set is mostly the same
CVS fetches previous revision online to build diff of changes
SVN builds diff offline
CVS takes twice less disk space
SVN stores full copies of checked out files for comparison
CVS screws linefeeds
SVN doesn't screw linefeeds
CVS is file based - history is separated for each file
SVN is atomic - modification of group of files is a whole
CVS maintains independent revision numbers for each file
SVN revisions are global for repository
CVS leaves deleted directories in repository tree
SVN keeps directory tree tidy
CVS has convenient concept of branches/tags
SVN branches/tags are just directory copies in repository
CVS branches/tags concept is complicated
SVN branches/tags are easy to understand
I think that for the most of us the choice is rather obvious if there are no practice to use $Id$ to track how many changes a file underwent since last time, or if people are not too addicted to CVS branches. Nevertheless there are still many projects that use CVS and the true reasons why people do this are: old habits, absence of time to learn something new and dependencies on CVS in hard-coded legacy scripts. Of course, laziness is also a reason and it's funny that this laziness sometimes pushes to seek more convenient tools.
It would be interesting to get deep into the details of code contributions, but to keep the long story short and justify the title let's compare SVN and CVS. These tools provide access to source code and poor decision may affect developers experience and desire to contribute in the future. This post based on RFC submitted to PHPDOC community a year ago, which may be useful for other conservative CVS parties out there like MinGW
Facts only.
++++ Accessibility:
CVS usually blocked by proxies (needs dedicated port)
SVN works over HTTP and HTTPS via WebDAV
CVS doesn't work behind a proxy
SVN works with proxies
CVS checkout is complicated, it is hard to remember all the prerequisites
cvs -d :pserver:cvsread@cvs.php.net:/repository login
cvs -z3 -d :pserver:cvsread@cvs.php.net:/repository checkout -P phpdoc
SVN project checkout command is easy to remember
svn co http://svn.php.net/repository/phpdoc
CVS is complicated to learn
SVN has a perfect book
CVS is abandoned by developers
SVN is supported
++++ Security:
CVS password is transmitted over network in cleartext with simple rot13-like translation
SVN works over HTTPS, supports Apache authentication schemes
++++ Usability:
Command set is mostly the same
CVS fetches previous revision online to build diff of changes
SVN builds diff offline
CVS takes twice less disk space
SVN stores full copies of checked out files for comparison
CVS screws linefeeds
SVN doesn't screw linefeeds
CVS is file based - history is separated for each file
SVN is atomic - modification of group of files is a whole
CVS maintains independent revision numbers for each file
SVN revisions are global for repository
CVS leaves deleted directories in repository tree
SVN keeps directory tree tidy
CVS has convenient concept of branches/tags
SVN branches/tags are just directory copies in repository
CVS branches/tags concept is complicated
SVN branches/tags are easy to understand
I think that for the most of us the choice is rather obvious if there are no practice to use $Id$ to track how many changes a file underwent since last time, or if people are not too addicted to CVS branches. Nevertheless there are still many projects that use CVS and the true reasons why people do this are: old habits, absence of time to learn something new and dependencies on CVS in hard-coded legacy scripts. Of course, laziness is also a reason and it's funny that this laziness sometimes pushes to seek more convenient tools.
Wednesday, April 09, 2008
503 Service Temporarily Unavailable
Just a typical 503 page source to know what to look for in web site monitoring scripts. This one is usually displayed by Apache when triggered by Tomcat.
--cut-[503.html]-
--cut-[503.html]-
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">-----------------
<HTML><HEAD>
<TITLE>503 Service Temporarily Unavailable</TITLE>
</HEAD><BODY>
<H1>Service Temporarily Unavailable</H1>
The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.
</BODY></HTML>
Subscribe to:
Posts (Atom)