Showing posts with label mingw. Show all posts
Showing posts with label mingw. Show all posts

Wednesday, January 09, 2008

Compiling Python extension with GCC

In my previous post I've described how to make C code accessible from Python. I used Visual C++ compiler cl.exe to build an extension (or module) for Python. This follow-up shows how to compile the same extension for windows using GCC. I bet you already know what GCC is and that it is available from MinGW install as a result of install procedure described long ago.

Grab the source from the previous post - it won't change. Everything what is going to happen are just changes in .exe and its command line options. Saving source as farpython.c and starting GCC to compile it:

gcc farpython.c

As usual, this won't produce anything useful except errors.

farpython.c:14:20: Python.h: No such file or directory
farpython.c:18: error: syntax error before '*' token
farpython.c:19: error: syntax error before '*' token
...

Additional include search path is specified using -I option in GCC.

gcc -IE:\ENV\Python25\include farpython.c

A different picture, but the output is still grim.

D:\Temp/ccyOaaaa.o(.text+0x1c):farpython.c: undefined reference to `_imp__PyArg_ParseTuple'
D:\Temp/ccyOaaaa.o(.text+0x4c):farpython.c: undefined reference to `_imp__Py_BuildValue'
D:\Temp/ccyOaaaa.o(.text+0x88):farpython.c: undefined reference to `_imp__Py_InitModule4'
D:\Temp/ccyOaaaa.o(.text+0xbe):farpython.c: undefined reference to `_imp__Py_InitModule4'
E:/ENV/MSYS/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../libmingw32.a(main.o)(.text+0x106):main.c: undefined reference to `
WinMain@16'
collect2: ld returned 1 exit status

Luckily these errors are not concerned with the code. They are from linker (ld) complaining it could not find library with binaries for functions defined in Python.h. The last one about undefined reference to WinMain is different though, but let's skip it until we deal with missing libraries. Python libraries are located at E:\ENV\Python25\libs and option to GCC is -L.

gcc -IE:\ENV\Python25\include -LE:\ENV\Python25\libs farpython.c

The output is still the same. The problem here is that linker doesn't know which specific library we need to link with to get binary bits for Python.h functions. cl.exe from Visual C++ was able to detect the correct library somehow, but for GCC we have to specify its name explicitly with -l option. Note that this option goes after a name of all compiled .c files. It is because params up to and including .c files are for compiler component and everything that goes after can be treated as linker's.

gcc -IE:\ENV\Python25\include -LE:\ENV\Python25\libs farpython.c -lpython25

Check the output.

E:/ENV/MSYS/mingw/bin/../lib/gcc/mingw32/3.4.2/../../../libmingw32.a(main.o)(.text+0x106):main.c: undefined reference to `
WinMain@16'
collect2: ld returned 1 exit status

WinMain is an entrypoint or starting point of any program on windows platform, but Python extension is not a program that starts execution itself. .pyd is a .dll, or shared library with functions to be called by other programs. To tell that to GCC we add -shared switch to command line.

gcc -IE:\ENV\Python25\include -LE:\ENV\Python25\libs farpython.c -lpython25 -shared

Now everything seems fine, but instead of far.pyd or farpython.pyd we've got a.exe Default output filename is easily corrected with yet another option -o

gcc -IE:\ENV\Python25\include -LE:\ENV\Python25\libs farpython.c -lpython25 -shared -o far.pyd

Test.

E:\>python
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import far
>>> far.example("echo")
ECHO is on.
0

Monday, May 28, 2007

How to compile pysqlite for Windows

If you are reading this page that means you know that pysqlite is and probably need updated version of this python extension for windows compiled against newer version of SQLite. Why would anyone need it? Perhaps because of fixed bugs or SQLAlchemy complaining that you need a better SQLite version as it happened in my case.

Original instructions are available for discontinued MS Toolkit Compiler at http://initd.org/pub/software/pysqlite/doc/install-source-win32.html and these are for MinGW compiler. First of all you need to install MinGW somewhere and preferably add directory with gcc.exe into %PATH%. Then unpack pysqslite sources, edit setup.cfg to look like this:

[build_ext]
define=
include_dirs=sqlite3
library_dirs=sqlite3
libraries=sqlite3

That means sqlite3/ directory should contain additional includes and libraries required to compile the extension. These are sqlite3.dll and sqlite3.h available at SQLite downloads from sqlite-source-x_x_x.zip and sqlitedll-3_3_17.zip archives. Unpack them into sqlite3/subdirectory of already unpacked pysqlite source. Now issue:

setup.py build --compiler=mingw32

If everything compiles Ok patch setup.py to carry sqlite3.dll inside to avoid messages about missing .dll

--- pysqlite-2.3.3/setup_old.py Sun Jan 14 02:56:12 2007
+++ pysqlite-2.3.3/setup.py Mon May 28 15:59:05 2007
@@ -109,7 +109,9 @@
+ glob.glob("doc/*.txt") \
+ glob.glob("doc/*.css")),
("pysqlite2-doc/code",
- glob.glob("doc/code/*.py"))]
+ glob.glob("doc/code/*.py")),
+ ("Lib/site-packages/pysqlite2",
+ ["sqlite3/sqlite3.dll"])]

py_modules = ["sqlite"]
setup_args = dict(


Then build an installer to save your work for the future. I saved mine at http://rainforce.org/sqlite/bindings/

setup.py bdist_wininst

Thursday, December 22, 2005

How to compile SQLite for Windows

Here you will see how to compile SQLite for Windows with MinGW and without TCL. For bots.

How to install MINGW
  1. Download necessary files from http://sourceforge.net/project/showfiles.php?group_id=2435
    At the moment these are:
    MSYS-1.0.10.EXE
    MinGW-5.0.0.EXE
  2. Install MSYS ("minimal system" - some kind of unix emulator) to D:\MSYS (for example). If you see prompt to execute post install instructions you may say no, because you probably do not have MinGW or MSYS previously installed.
  3. Install MinGW (with GPP option necessary for SQLite) to D:\MSYS\mingw
    Installator didn't work with my proxy, so each file used by MinGW-5.0.0.EXE was downloaded separately and placed into installator's directory. Some downloaded files with newer versions than installer could understand were renamed to make it happy. List of packages required by MinGW-5.0.0.exe installer:
    binutils-2.15.91-20040904-1.tar.gz
    gcc-core-3.4.2-20040916-1.tar.gz
    mingw-runtime-3.9.tar.gz (renamed to mingw-runtime-3.8.tar.gz)
    w32api-3.5.tar.gz (renamed to w32api-3.3.tar.gz)
    gcc-g++-3.4.2-20040916-1.tar.gz (C++ preprocessor required by SQLite configure)
  4. After installer finished your environment is ready. Launch MSYS to start working. Home directory in this case ("cd ~") is located at D:\MSYS\home\yourusername if you need to find out where to place you files from windows to be found in MSYS.
  5. Optionally if you will need to run additional developer tools such as autoconf or automake, these are available through separate msysDTK-1.0.1.exe installer from the sourceforge download page above.
How to compile SQLite
  1. Download SQLite and upack it into subdirectory "sqlite" (for example) of your MSYS home directory (something like D:\MSYS\home\techtonik in my case)
  2. Launch MSYS
  3. cd sqlite
  4. mkdir bld; cd bld
  5. ../configure --disable-tcl
  6. This will create "tsrc" directory with all necessary files to compile SQLite
    make target_source
    cd tsrc
  7. Since there is no TCL libraries, tclsqlite.c should be removed to avoid errors during compilation (this is purely optional)
    mv tclsqlite.c tclsqlite.c.unused
  8. Compile sources with a some optimization (-O2 flag).
    gcc -O2 -DNDEBUG=1 -DTHREADSAFE=1 -c *.c
  9. Link compiled object (*.o) files to make sqlite3.exe (-s tell gcc to strip additional info used for debug)
    gcc -s -o sqlite3 *.o
  10. Remove shell.o object file used as entry point for command line utility. Useless for DLL.
    rm shell.o
  11. Link compiled object (*.o) files to make .dll The command below tells GCC to link all object files into one shared library (for windows it is .dll) and make functions listed in .def available for use to external programs (export). sqlite3.def file can be found in official sqlitedll-x_x_x.zip distribution
    gcc -shared -s -o sqlite3.dll *.o *.def
  12. There is one more thing you may need in future if you would like to compile other programs with SQLite support. It is import library. To make one for MinGW (called libsqlite3.dll.a) link .dll with command
    gcc -shared -s -o sqlite3.dll *.o *.def -Wl,--out-implib,libsqlite3.dll.a
  13. Wrap release.
    mkdir ../release
    cp sqlite3.dll sqlite3.exe libsqlite3.dll.a sqlite3.h ../release
    cd ../release
That's all. If you want reduce the size of .dll by several kbytes use strip --strip-unneeded command. By default strip will remove all symbols from file, but with this key it doesn't remove information needed for .dll relocation.

Ready to use files for SQLite 3.3.17 are available at http://rainforce.org/sqlite/