Abstract (version 0.44.7): This document describes the GNU build system and several non-GNU (but GPL licensed) software development tools. The main purpose is to introduce software developers to the GNU build system and present this system as a complete, flexible and well integrated development environment.
Note that this document is still in its early stages, several sections are currently incomplete or even missing.
This document uses parts of existing documentation included with each of the development tools. So it is certainly a good idea to look at the manuals and web pages for more detailed information about specific tools.
The following conventions are used in this document:
---|>means that the previous node outputs a file or a set of files. These output files can be data files or executables.
The productivity of a software developer can be increased by letting tools like Autoconf, Automake, Libtool, and other GNU tools do a lot of the (platform specific) configuration and build work for you.
Unfortunately the learning curve for some of these tools can be quite steep. However, once you get a general idea of what the GNU build system is, how it works, and start applying this knowledge to your own software projects, you'll probably understand and appreciate the GNU build system for most of its design choices and the implementation. In places where the functionality could be lacking, or where the build process is not as effective and/or efficient as it could be, this document tries to comment on workarounds or alternatives.
A large part of this document describes the tools Autoconf and Automake. By using Autoconf and Automake in the build process, you can improve the the required platform specific configuration and the quality of the makefiles.
Better makefiles and their configuration help with several common software development issues:
./configure --prefix=[somewhere] makes it very easy for your users
to customize the installation directory. AC_ARG_WITH is an execllent macro
that lets you create compilation options with ease. Two of my favorites are
creating debugging and profiling builds that can be removed in a production compile.
The underlying technologies for the advantages described above are:
In some situations you will need to do some manual tweaking to the makefiles (or sourcecode), but the GNU build system is flexible enough to not get in your way when you want to do this. And of course, if you feel you're stuck with your current solution, you can always ask for help on the mailing lists.
The work needed by the software user to build and install the software is simplified. In most cases the user just executes a standard set of a few simple commands on the source-package to get the software running on their platform (assuming all the required dependencies were satisfied).
You can even do more to help the software user. By packaging the source package
for a Package Management System, like eg. the Debian system, the user is only
a few keys/clicks away from having the package installed and maybe even
configured, by using the "dpkg" package installer and the "debconf"
configuration system.
A list of platforms (CPU architecture + Operating System) which are currently supported by GCC can be found on the GCC build status page
The support for some tools and OS functions can sometimes be incomplete or even missing however. Check the documentation of each tool for the current (functionality / operating system support) status.
A list of platforms on which GNU libc is expected to run can be found here.
Autoconf and Automake should be usable on most Un*x-like operating systems that also have support for GCC and GNU Libc.
Required developer software: (POSIX-compliant) shell, Libc, GCC
(together with any appropriate front-end and library support), Binutils,
Make, Autoconf (which also includes: aclocal), Automake, Make, M4, Perl,
Tar, Gzip.
Recommended developer software: Vim/Emacs, Gdb, Doxygen, Diff, Patch, CVS, Strace, Ltrace, Ptrace, Gettext, Xgettext, Msgmerge.
Optional developer software: GnuPG (origination integrity), cksum (content integrity).
Required user software: (POSIX-compliant) Shell, Libc, GCC (together with any appropriate front-end and library support), Binutils, Autoconf, Make, M4, Perl, Tar, Gzip.
GCC is the GNU Compiler
Collection, which is a semantic back-end and a set of front-ends for compiling
C, C++, Java, and several other high-level programming languages into platform
object code (applications or libraries). The front-ends are respectively named:
gcc, g++ and gcj.
Libc is the GNU C library for Unix-like systems. It defines the essential program runtime layer on top of a kernel, like eg. Linux, or Gnu Mach/Hurd. The GNU C library supports the ISO C and POSIX standards and is designed to be portable. Note that version 2 of this library is still in the process of being ported to other platforms.
Cpp is a macro processor which is automatically used by the compiler to transform your program before actual compilation. Macro's are one word abbreviations for longer lexical constructs. Cpp also removes the sourcecode comments. Some people dislike the use of the preprocessor in the compilation process but agree there's currently no better solution [1].
Binutils
is a software collection which contains essential system software like
an assembler, a linker, a preprocessor, and a binary object loader.
Here is a detailed list of the programs and libraries in the binutils
package:
The file in "/etc/ld.so.conf" contains a list of colon, space, tab, newline,
or comma separated directories in which to search for libraries.
The file in "/etc/ld.so.cache" contains an ordered list of libraries found in the directories specified in "/etc/ld.so.conf". This file is not in human readable format, and is not intended to be edited. The files named "lib*.so.<version>" are shared libraries.
Autoscan is a perl script to help create a "configure.ac" file for
your program package. Autoscan examines source files in the directory
tree rooted at a directory given as a command line argument, or the current
directory if none is given. It searches the source files for common
portability problems and creates a "configure.scan" file which is a
preliminary "configure.ac" for that package.
You have to manually examine "configure.scan" before renaming it to
"configure.ac", since it will probably need some adjustments.
See the example sections for more details on creating a "configure.ac" file.
Aclocal is a perl script that automatically generates "aclocal.m4" from "configure.ac" and ".m4" files. Automake includes a number of Autoconf macros which can be used in your package. Some of them are actually required by Automake in certain situations. These macros must be defined in your "aclocal.m4", otherwise they will not be seen by autoconf. At startup, aclocal scans all the ".m4" files it can find, looking for macro definitions. Then it scans "configure.ac".
Autoconf is a shell script which creates a "configure*" shell script from the "configure.ac" file. This "configure*" script is later used by the user to interpret a "Makefile.in" file and then create a platform specific "Makefile" file that can be used to build and install the software package.
Autoconf processes "configure.ac" with the "m4" macro processor, using the Autoconf macros. The Autoconf macros are defined in several files.
Some of these macro files are distributed with Autoconf. Autoconf reads them first. Then it looks for the optional file "acsite.m4" in the directory that contains the distributed Autoconf macro files, and for the optional file "aclocal.m4" in the current directory. Those files can contain your site's or the packages own Autoconf macro definitions.
The contents of "acinclude.m4", if it exists, are also automatically included in aclocal.m4. This is useful for incorporating local macros into configure.
[ autoconf.m4 ]
[ acsite.m4 ] (optional site macro file used by autoconf)
Each time you make changes to the "configure.ac" file or any other
Autoconf or Automake related configuration file, you need to
'bootstrap' the build system. Bootstrapping can be done by executing the
following 3 commands:
aclocal && automake --add-missing && autoconf
A more advanced bootstrapping script can be found here: autogen script.
Autoreconf is a Bourne shell script (included with Autoconf), that updates "configure*" scripts. If you have a lot of Autoconf generated "configure*" scripts, the autoreconf program can save you some work. It runs autoconf (and autoheader, where appropriate) repeatedly to remake the Autoconf configure scripts and configuration header templates in the directory tree rooted at the current rent directory. By default, it only re-makes those files that are older than their "configure.ac" or (if present).
Autoheader is a perl script (included with Autoconf), which runs the "m4" macro processor, to help create a template file of C/C++ #defines called "config.h".
Autoheader scans "configure.ac" and figures out which C/C++ preprocessor symbols it might define. The file that autoheader creates contains mainly "#define" and "#undef" statements and their accompanying comments. The "AM_CONFIG_HEADER(config.h)" line in "configure.ac" indicates that you want to use a "config.h" file.
You will need a "stamp-h" file in your project to ensure that Automake regenerates "config.h" from "config.h.in". Type 'touch stamp-h' to add this file to your project.
Automake is a perl script that automatically creates "Makefile.in" files from "Makefile.am" files.
The developer needs to write files named "Makefile.am", these use a simpler syntax than ordinary "Makefile"'s. The user later runs the "configure*" script and then make to build the sourcecode.
The suffix "am" means it is an Automake configuration file and the suffix "in" means it is an input file. These "in" files will later be read by autoconf to create "Makefiles". The Automake "Makefile.am" files are found by scanning the "configure.ac" file.
Make is a tool that automatically determines which sourcecode files, and other files, of a program need to be recompiled, and issues the commands to recompile them.
You need a file called "Makefile" to tell Make what to do. Most often, the "Makefile" tells Make how to compile and link a program, but other processing can also be done (eg. create typesetted documentation).
By default, Make starts with the first target (not targets whose names start with "."). This is called the "default goal". "Goals" are the targets that Make strives ultimately to update.
Libtool is a perl script that provides a standard way to generate both static and shared libraries. It hides the complexity of using shared libraries behind a consistent, portable interface. Standard scripts: libtool, libtoolize, ltmain.sh (creates ".la" files which are used to create shared libraries. Libtool currently works for library sourcecode in C, C++ and Java (libtool 1.4).
Gettext is a message catalog system for internationalisation (aka i18n).
Gettext is designed to minimise the impact of internationalisation on
program sources.
If AM_GNU_GETTEXT is seen in "configure.ac", then Automake turns on support
for gettext. The gettext support in Automake requires the addition of two
subdirectories to the package: "intl" and "po". Automake ensure that these
directories exist and are mentioned in "SUBDIRS". Furthermore, Automake checks
that the definition of "ALL_LINGUAS" in `configure.ac' corresponds to all the
valid ".po" files, and nothing more.
The letters "PO" in ".po" files stand for "Portable Object". PO files are meant to be read and edited by humans, and associate each original, translatable string of a given package with its translation in a particular target language. A single PO file is dedicated to a single target language.
(gettext*, xgettext*, msmerge*) The PO files are best created by the xgettext program, and later updated or refreshed through the msgmerge program. Xgettext extracts all marked messages from a set of sourcecode files and initialises a PO file with empty translations. msgmerge takes care of adjusting PO files between releases of the corresponding sources, commenting obsolete entries, initialising new ones, and updating all source line references. Files ending with ".pot" are kind of base translation files found in distributions, in PO file format, and ".pox" files are often temporary PO files.
Doxygen is a documentation system for C, C++, Java and IDL sourcecode. It is a flexible an easy to use 'literate programming' tool that can create HTML, Unix Manual pages, LaTeX and RTF documents from the sourcecode and its comments. There is also indirect support for Postscript and PDF output via the LaTeX format. This recommendable tool gets even better when one starts using the automatic source diagramming features of "Dot".
CVS is a document revision system, capable of handling concurrent editing sessions on the same file.
It's worth mentioning that CVS is not the perfect sourcecode management tool,
since it does not support the following:
Because of these drawbacks some alternatives have been created. Arch and Subversion are a two free alternatives to CVS.
GDB is the GNU Debugger, which make it possible to see what is going on 'inside' a program while it executes. To achieve this the sourcecode has to be compiled with debugging symbols included option "-g". Another option is to include a debugging macro, and switch this on after compile time (Fixme: show how this can be done).
Patch takes a patch file containing a difference listing produced by the diff program and applies those differences to one or more original files, producing patched versions. Normally the patched versions are put in place of the originals.
Patch is generally used like this: patch -p0 < <patch-file>
The p0 specifies the number of levels (in directories) to skip before trying to start the patching process, generally zero is the right number, though it depends on how the diff was created. You can use diff to create patches with: diff -Naur <source-dir> <changed-dir>
(Problem Definition)
^
|
|
(Requirements Analysis)
^
|
|
------------------------------------------
| Development Activities: |
------------------------------------------
| Think, Read, Learn, Understand, Sketch |
------------------------------------------
| Write - editor |
| Compile - preprocessor, compiler, |
| assembler, linker |
| Run - loader |
| Debug - debugger, tracer |
| Optimise - profiler |
| Synchronise - cvs, patch, diff |
| Communicate - voice, mailing list, irc |
| Port - compile, run, debug, ... |
------------------------------------------
| |
| | (*.diff, *.dsc, *.changes, *.orig.tar.gz)
| | <packagename-version-1_arch>.deb
| | ^
| | -
_ | |
v - |
.---> (Code) v (create a Debian package)
/ (Documentation) |
/ |
(1) / v
autoscan* (manual edit) <packagename-version>.tar.gz
| | ^
| | -
| | |
| | (10) |
| | make* distcheck
- - |
v v | make* distclean
configure.scan Makefile.am | |
^ ^ | |
| | | | (9)
| | | | make* install
| | | | |
| | | | | make* uninstall
| | | | | |
| | | | | |
| | | | | | (8)
| | | | | | make check
| | | | | | |
| | | | | | |
| (4) | v v v v v
(manual edit .------ automake* (compiled files)
and rename) / | ^
| / | -
| / | |
- / - | (7a, 7b, ...)
v / v (7) | -----------------
configure.ac <---' Makefile.in make* | Compile Tools |
^ ^ ^ COPYING / | -----------------
| | | (4a) INSTALL / | | cpp*, gcc* |
| | | libtoolize* stamp-h.in / | | g++*, gcj* |
| | | | missing* <--| | | gasp*, as* |
| | | | install-sh* <--| | | ar* |
| | | | mkinstalldirs*<--' | | ld* |
| | | - config.guess | | libtool* |
| | | v config.sub | | doxygen* |
| | | ltmain.sh* --|>ltconfig | | ... |
| | | config.guess | -----------------
| | | config.sub | ^
| | | ^ | |
| | | | | |
| | | | | |
| | | (5) | (6) v |
| | '--------- autoconf* --|> configure* ---|> Makefile ----'
| | / | \ \ | | libtool*
| | / | \ \ | | config.status*
| | / | \ \ | | config.log
| | | | \ \ | `--------> config.cache
| | v | | \ | [<FILE1>]
| | acinclude.m4 | | \ | [<FILE2>]
| | acsite.m4 | | | | ...
| | config.h.top | | | v
| | config.h.bottom | | | NEWS
| | v | | README
| aclocal*---|> aclocal.m4 | | AUTHORS
| (2) ^ | | ChangeLog
| | | v
| | | [<FILE1>.in]
| | | [<FILE2>.in]
| | | ...
| (3) | v
'------ autoheader* ---|> config.h.in
Let's now look at details of creating a "configure.ac" file for a very simple software project, which we'll call "Fibo".
Fibo is a small C++ program that takes an integer number as an argument on the command line, and recursively computes and prints all the Fibonacci numbers between 1 and the number argument, then the program exits.
Fibo uses the functions supplied in the file "fibonacci.h". These functions are defined/implemented in a file called "fibonacci.cpp". In the next chapter we'll turn these "fibonacci" source files into a shared library, but for now Fibo uses no external libraries (other than the standard C and C++ runtime libraries and their header files).
(code link: fibo.cpp, fibonacci.h, fibonacci.cpp)
We'll test that the "fibo-1.0.tar.gz" package will compile, install and be packaged (again) on the following platforms:
(Create directory structure, put source files into "src/", make bootstrap script)
If you, for whatever reason, do not want the GNU-style specific files (NEWS README AUTHORS ChangeLog) then you could add the following to your "Makefile.am" instead: AUTOMAKE_OPTIONS = foreign.
A minimal "configure.ac" file can be created by running Autoscan and editing the "configure.scan" file to look something like this for a small project:
dnl Process this file with autoconf to produce a configure script. AC_INIT() AM_INIT_AUTOMAKE(fibo, 0.1) dnl Checks for programs. AC_PROG_CC AC_PROG_CXX dnl Checks for libraries. dnl Checks for header files. AM_CONFIG_HEADER(config.h) dnl Checks for typedefs, structures, and compiler characteristics. AC_C_CONST dnl Checks for library functions. AC_OUTPUT([ Makefile src/Makefile ])
SUBDIRS = src docs EXTRA_DIST = \ src/fibo.h \ src/fibo.cpp \ docs/doxygen.conf
EXTRA_DIST = EXTRA_SUBDIRS = DIST_SUBDIRS =
The tag "dnl" means "delete to newline". It deletes everything from itself
to the next newline, so it can be used after a configuration option on the
same line. Basically its just a comment.
The other lines contain
Autoconf macro definitions, which are abbreviations for commands.
Most macro's are standard commands (included with Autoconf), but it is
also possible to create user-defined macro's.
Software libraries can be defined as being binary programs which have multiple points of entry. There are two basic types of libraries: static and shared. Static libraries are linked into the program at compile time, consequently the library will always be connected to the program.
Shared libraries have several advantages over statically linked libraries:
- Application modularity. By loading needed libraries dynamically during
runtime.
- Upgrades of a shared library can now affect every program that uses
this library, without the need for recompilation of course.
- Multiple library versions can be supported.
- Save some disk space.
However as always flexibility comes with a price called "a bit more complexity". Also some platforms don't always support shared
Here are the basic steps to creating a shared library (on a shared library
capable platform) the manual way:
1) create library objects (".lo" and ".o")
libtool gcc -g -c foo.c
libtool gcc -g -c hello.c
2) Create shared library (and indicate the install directory)
libtool gcc -g -o libhello.la foo.lo hello.lo -rpath /usr/local/lib/ -lm
Ouput:
rm -fr .libs/libhello.la .libs/libhello.* .libs/libhello.* gcc -shared foo.lo hello.lo -lm -lc -Wl,-soname -Wl,libhello.so.0 -o .libs/libhello.so.0.0.0 (cd .libs && rm -f libhello.so.0 && ln -s libhello.so.0.0.0 libhello.so.0) (cd .libs && rm -f libhello.so && ln -s libhello.so.0.0.0 libhello.so) ar cru .libs/libhello.a foo.o hello.o ranlib .libs/libhello.a creating libhello.la (cd .libs && rm -f libhello.la && ln -s ../libhello.la libhello.la)
3) linking a application against an UN-installed library
demo$ libtool gcc -g -o hell main.o libhello.la -lm
gcc -g -o .libs/hell main.o .libs/libhello.so -lm -lm -Wl,--rpath
-Wl,/usr/local/lib/
creating hell
No lets see how Libtool works with Autoconf and Automake: (Fixme)
(TODO)
Now that we've made (or fetched from another site) the source package called "fibo-0.1.tar.gz", we can begin to enjoy the fruits of our labor. Compiling, installing and repackaging the fibo package should now be quite straightforward.
Developers should run these commands:
Users should only need to run the following commands to install the package.
When the user has a PMS package the install process becomes even easier:
dpkg -i <packagename>.deb
When you have uploaded your custom Debian package to an HTTP or FTP APT-site you can do the following:
PMS's can be divided in those using (uncompiled) sourcecode packages
and those using (compiled) binary packages. Both approaches have their
advantages and disadvantages:
Required: dpkg, debmake, devscripts, dh-make,
Recommended: hello-debhelper
Here are the general steps to debianize the software package from example 1:
wget http://fibo.org/download/fibo-1.0.tar.gz
mkdir ~/fibo
cp fibo-1.0.tar.gz ~/fibo
tar xvzf fibo-1.0.tar.gz
cd ~/fibo/fibo-1.0
dh_make -e jama@debianlinux.net -f ../fibo-1.0.tar.gz
cd debian
Fill in "Section:" and "Description:" and if needed fill in "Depends:"
dsc -i, or use dch -v <version>-<revision>
cd ..
dpkg-buildpackage -rfakeroot
Enter your GPG passphrase twice, to sign these files: fibo-1.0-1_i386.dsc, fibo-1.0-1_i386.changes
lintian -i fibo-1.0-1_i386.changes
dpkg -I fibo-1.0-1_i386.deb
dpkg -i fibo-1.0-1_i386.deb
The following steps are all optional:
dpkg -r fibo
See: http://www.interq.or.jp/libra/oohara/apt-gettable/apt-gettable/ cd /var/www ln -s /var/cache/apt/archives /var/www/debian cd debian dpkg-scanpackages . /dev/null > Packages && gzip -9c Packages > Packages.gz
apt-get update
apt-get install pentium-builder
less /usr/share/doc/pentium-builderREADME.Debian
apt-get source fibo -b
Integrated Development Environment's (IDE's):
Graphical User Interface (GUI) Builders:
Internationalization (i18n):
Debugging:
Profiling:
Project Management:
CVS:
Sourcecode quality:
- Document your sourcecode via a literate programming tool, eg.
with the Doxygen tool.
- Try to fix all compile warning "-Wall" generates immediately.
Performance tuning:
- Program for correct concurrency.
- Only use optimizations "-On" for 'finished' packages and
performance benchmarking.
- Manual optimizations for your sourcecode are a lot faster than
compile-time optimizations, so don't be afraid to experiment.
However, do test optimizations for portability and readability
(comment on what you did to get this performance increase)
- Test stability by stress testing the application with scripts.
- Measure performance by profiling expensive functions.
- inline functions
- Use IO-multiplexing/multi-threading when needed.
Code and Release Management:
- CVS repository
- Version number policy and the TODO's/Milestones