Programming

Programming#

I provide some pointers for people to learn programming on the Debian system enough to trace the packaged source code. Here are notable packages and corresponding documentation packages for programming.

List of packages to help programming

Online references are available by typing "man name" after installing manpages and manpages-dev packages. Online references for the GNU tools are available by typing "info program_name" after installing the pertinent documentation packages. You may need to include the contrib and non-free archives in addition to the main archive since some GFDL documentations are not considered to be DFSG compliant.

[!WARNING] Do not use "test" as the name of an executable test file. "test" is a shell builtin.

[!CAUTION] You should install software programs directly compiled from source into "/usr/local" or "/opt" to avoid collision with system programs.

[!TIP] Code examples of creating "Song 99 Bottles of Beer" should give you good ideas of practically all the programming languages.

The shell script#

The shell script is a text file with the execution bit set and contains the commands in the following format.

#!/bin/sh
 ... command lines

The first line specifies the shell interpreter which read and execute this file contents.

Reading shell scripts is the best way to understand how a Unix-like system works. Here, I give some pointers and reminders for shell programming. See "Shell Mistakes" (http://www.greenend.org.uk/rjk/2001/04/shell.html) to learn from mistakes.

Unlike shell interactive mode (see ??? and ???), shell scripts frequently use parameters, conditionals, and loops.

POSIX shell compatibility#

Many system scripts may be interpreted by any one of POSIX shells (see ???). The default shell for the system is "/bin/sh" which is a symlink pointing to the actual program.

• bash(1) for lenny or older

• dash(1) for squeeze or newer

Avoid writing a shell script with bashisms or zshisms to make it portable among all POSIX shells. You can check it using checkbashisms(1).

List of typical bashisms

The "echo" command must be used with following cares since its implementation differs among shell builtin and external commands.

• Avoid using any command options except "-n".

• Avoid using escape sequences in the string since their handling varies.

[!NOTE] Although "-n" option is not really POSIX syntax, it is generally accepted.

[!TIP] Use the "printf" command instead of the "echo" command if you need to embed escape sequences in the output string.

Shell parameters#

Special shell parameters are frequently used in the shell script.

List of shell parameters

Basic parameter expansions to remember are as follows.

List of shell parameter expansions

Here, the colon ":" in all of these operators is actually optional.

• with ":" = operator test for exist and not null

• without ":" = operator test for exist only

List of key shell parameter substitutions

Shell conditionals#

Each command returns an exit status which can be used for conditional expressions.

• Success: 0 ("True")

• Error: non 0 ("False")

[!NOTE] "0" in the shell conditional context means "True", while "0" in the C conditional context means "False".

[!NOTE] "[" is the equivalent of the test command, which evaluates its arguments up to "]" as a conditional expression.

Basic conditional idioms to remember are the following.

• "<command> && <if_success_run_this_command_too> || true"

• "<command> || <if_not_success_run_this_command_too> || true"

• A multi-line script snippet as the following

if [ <conditional_expression> ]; then
 <if_success_run_this_command>
else
 <if_not_success_run_this_command>
fi

Here trailing "|| true" was needed to ensure this shell script does not exit at this line accidentally when shell is invoked with "-e" flag.

List of file comparison operators in the conditional expression

List of string comparison operators in the conditional expression

Arithmetic integer comparison operators in the conditional expression are "-eq", "-ne", "-lt", "-le", "-gt", and "-ge".

Shell loops#

There are several loop idioms to use in POSIX shell.

• "for x in foo1 foo2 … ; do command ; done" loops by assigning items from the list "foo1 foo2 …" to variable "x" and executing "command".

• "while condition ; do command ; done" repeats "command" while "condition" is true.

• "until condition ; do command ; done" repeats "command" while "condition" is not true.

• "break" enables to exit from the loop.

• "continue" enables to resume the next iteration of the loop.

[!TIP] The C-language like numeric iteration can be realized by using seq(1) as the "foo1 foo2 …" generator.

[!TIP] See ???.

The shell command-line processing sequence#

The shell processes a script roughly as the following sequence.

• The shell reads a line.

• The shell groups a part of the line as one token if it is within "…" or '…'.

• The shell splits other part of a line into tokens by the following.

• Whitespaces: <space> <tab> <newline>

• Metacharacters: < > | ; & ( )

• The shell checks the reserved word for each token to adjust its behavior if not within "…" or '…'.

• reserved word: if then elif else fi for in while unless do done case esac

• The shell expands alias if not within "…" or '…'.

• The shell expands tilde if not within "…" or '…'.

• "~" → current user's home directory

• "~<user>" → <user>'s home directory

• The shell expands parameter to its value if not within '…'.

• parameter: "$PARAMETER" or "${PARAMETER}"

• The shell expands command substitution if not within '…'.

• "$( command )" → the output of "command"

• "` command `" → the output of "command"

• The shell expands pathname glob to matching file names if not within "…" or '…'.

• * → any characters

• ? → one character

• […] → any one of the characters in "…"

• The shell looks up command from the following and execute it.

• function definition

• builtin command

• executable file in "$PATH"

• The shell goes to the next line and repeats this process again from the top of this sequence.

Single quotes within double quotes have no effect.

Executing "set -x" in the shell or invoking the shell with "-x" option make the shell to print all of commands executed. This is quite handy for debugging.

Utility programs for shell script#

In order to make your shell program as portable as possible across Debian systems, it is a good idea to limit utility programs to ones provided by essential packages.

• "aptitude search ~E" lists essential packages.

• "dpkg -L <package_name> |grep '/man/man.*/'" lists manpages for commands offered by <package_name> package.

List of packages containing small utility programs for shell scripts

[!TIP] Although moreutils may not exist ouside of Debian, it offers interesting small programs. Most notable one is sponge(8) which is quite useful when you wish to overwrite original file.

Shell script dialog#

The user interface of a simple shell program can be improved from dull interaction by echo and read commands to more interactive one by using one of the so-called dialog program etc.

List of user interface programs

Shell script example with zenity#

Here is a simple script which creates ISO image with RS02 data supplemented by dvdisaster(1).

#!/bin/sh -e
# gmkrs02 : Copyright (C) 2007 Osamu Aoki <osamu@debian.org>, Public Domain
#set -x
error_exit()
{
  echo "$1" >&2
  exit 1
}
# Initialize variables
DATA_ISO="$HOME/Desktop/iso-$$.img"
LABEL=$(date +%Y%m%d-%H%M%S-%Z)
if [ $# != 0 ] && [ -d "$1" ]; then
  DATA_SRC="$1"
else
  # Select directory for creating ISO image from folder on desktop
  DATA_SRC=$(zenity --file-selection --directory  \
    --title="Select the directory tree root to create ISO image") \
    || error_exit "Exit on directory selection"
fi
# Check size of archive
xterm -T "Check size $DATA_SRC" -e du -s $DATA_SRC/*
SIZE=$(($(du -s $DATA_SRC | awk '{print $1}')/1024))
if [ $SIZE -le 520 ] ; then
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is good for CD backup:\\n $SIZE MB"
elif [ $SIZE -le 3500 ]; then
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is good for DVD backup :\\n $SIZE MB"
else
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is too big to backup : $SIZE MB"
  error_exit "The data size is too big to backup :\\n $SIZE MB"
fi
# only xterm is sure to have working -e option
# Create raw ISO image
rm -f "$DATA_ISO" || true
xterm -T "genisoimage $DATA_ISO" \
  -e genisoimage -r -J -V "$LABEL" -o "$DATA_ISO" "$DATA_SRC"
# Create RS02 supplemental redundancy
xterm -T "dvdisaster $DATA_ISO" -e  dvdisaster -i "$DATA_ISO" -mRS02 -c
zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
  --text="ISO/RS02 data ($SIZE MB) \\n created at: $DATA_ISO"
# EOF

You may wish to create launcher on the desktop with command set something like "/usr/local/bin/gmkrs02 %d".

Make#

Make is a utility to maintain groups of programs. Upon execution of make(1), make read the rule file, "Makefile", and updates a target if it depends on prerequisite files that have been modified since the target was last modified, or if the target does not exist. The execution of these updates may occur concurrently.

The rule file syntax is the following.

target: [ prerequisites ... ]
 [TAB]  command1
 [TAB]  -command2 # ignore errors
 [TAB]  @command3 # suppress echoing

Here "[TAB]" is a TAB code. Each line is interpreted by the shell after make variable substitution. Use "\" at the end of a line to continue the script. Use "$$" to enter "$" for environment values for a shell script.

Implicit rules for the target and prerequisites can be written, for example, by the following.

%.o: %.c header.h

Here, the target contains the character "%" (exactly one of them). The "%" can match any nonempty substring in the actual target filenames. The prerequisites likewise use "%" to show how their names relate to the actual target name.

List of make automatic variables

List of make variable expansions

Run "make -p -f/dev/null" to see automatic internal rules.

C#

You can set up proper environment to compile programs written in the C programming language by the following.

# apt-get install glibc-doc manpages-dev libc6-dev gcc build-essential

The libc6-dev package, i.e., GNU C Library, provides C standard library which is collection of header files and library routines used by the C programming language.

See references for C as the following.

• "info libc" (C library function reference)

• gcc(1) and "info gcc"

• each_C_library_function_name(3)

• Kernighan & Ritchie, "The C Programming Language", 2nd edition (Prentice Hall)

Simple C program (gcc)#

A simple example "example.c" can compiled with a library "libm" into an executable "run_example" by the following.

$ cat > example.c << EOF
#include <stdio.h>
#include <math.h>
#include <string.h>

int main(int argc, char **argv, char **envp){
        double x;
        char y[11];
        x=sqrt(argc+7.5);
        strncpy(y, argv[0], 10); /* prevent buffer overflow */
        y[10] = '\0'; /* fill to make sure string ends with '\0' */
        printf("%5i, %5.3f, %10s, %10s\n", argc, x, y, argv[1]);
        return 0;
}
EOF
$ gcc -Wall -g -o run_example example.c -lm
$ ./run_example
        1, 2.915, ./run_exam,     (null)
$ ./run_example 1234567890qwerty
        2, 3.082, ./run_exam, 1234567890qwerty

Here, "-lm" is needed to link library "/usr/lib/libm.so" from the libc6 package for sqrt(3). The actual library is in "/lib/" with filename "libm.so.6", which is a symlink to "libm-2.7.so".

Look at the last parameter in the output text. There are more than 10 characters even though "%10s" is specified.

The use of pointer memory operation functions without boundary checks, such as sprintf(3) and strcpy(3), is deprecated to prevent buffer overflow exploits that leverage the above overrun effects. Instead, use snprintf(3) and strncpy(3).

Debug#

Debug is important part of programing activities. Knowing how to debug programs makes you a good Debian user who can produce meaningful bug reports.

Basic gdb execution#

Primary debugger on Debian is gdb(1) which enables you to inspect a program while it executes.

Let's install gdb and related programs by the following.

# apt-get install gdb gdb-doc build-essential devscripts

Good tutorial of gdb is provided by "info gdb" or found elsewhere on the web. Here is a simple example of using gdb(1) on a "program" compiled with the "-g" option to produce debugging information.

$ gdb program
(gdb) b 1                # set break point at line 1
(gdb) run args           # run program with args
(gdb) next               # next line
...
(gdb) step               # step forward
...
(gdb) p parm             # print parm
...
(gdb) p parm=12          # set value to 12
...
(gdb) quit

[!TIP] Many gdb(1) commands can be abbreviated. Tab expansion works as in the shell.

Debugging the Debian package#

Since all installed binaries should be stripped on the Debian system by default, most debugging symbols are removed in the normal package. In order to debug Debian packages with gdb(1), either corresponding *-dbg packages or *-dbgsym packages need to be installed (e.g. libc6-dbg in the case of libc6, coreutils-dbgsym in the case of coreutils).

Old-style packages would provide its corresponding *-dbg package. It is placed directly inside Debian main archive alongside of the original package itself. For newer packages, they may generate *-dbgsym packages automatically when built and those debug packages are placed separately in debian-debug archive. Please refer to articles on Debian Wiki for more information.

If a package to be debugged does not provide either its *-dbg package or its *-dbgsym package, you need to install it after rebuilding it by the following.

$ mkdir /path/new ; cd /path/new
$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo apt-get install fakeroot devscripts build-essential
$ apt-get source package_name
$ cd package_name*
$ sudo apt-get build-dep ./

Fix bugs if needed.

Bump package version to one which does not collide with official Debian versions, e.g. one appended with "+debug1" when recompiling existing package version, or one appended with "~pre1" when compiling unreleased package version by the following.

$ dch -i

Compile and install packages with debug symbols by the following.

$ export DEB_BUILD_OPTIONS="nostrip noopt"
$ debuild
$ cd ..
$ sudo debi package_name*.changes

You need to check build scripts of the package and ensure to use "CFLAGS=-g -Wall" for compiling binaries.

Obtaining backtrace#

When you encounter program crash, reporting bug report with cut-and-pasted backtrace information is a good idea.

The backtrace can be obtained by the following steps.

• Run the program under gdb(1).

• Reproduce crash.

• It causes you to be dropped back to the gdb prompt.

• Type "bt" at the gdb prompt.

In case of program freeze, you can crash the program by pressing Ctrl-C in the terminal running gdb to obtain gdb prompt.

[!TIP] Often, you see a backtrace where one or more of the top lines are in "malloc()" or "g_malloc()". When this happens, chances are your backtrace isn't very useful. The easiest way to find some useful information is to set the environment variable "$MALLOC_CHECK_" to a value of 2 (malloc(3)). You can do this while running gdb by doing the following.

 $ MALLOC_CHECK_=2 gdb hello

Advanced gdb commands#

List of advanced gdb commands

Debugging X Errors#

If a GNOME program preview1 has received an X error, you should see a message as follows.

The program 'preview1' received an X Window System error.

If this is the case, you can try running the program with "--sync", and break on the "gdk_x_error" function in order to obtain a backtrace.

Check dependency on libraries#

Use ldd(1) to find out a program's dependency on libraries by the followings.

$ ldd /bin/ls
        librt.so.1 => /lib/librt.so.1 (0x4001e000)
        libc.so.6 => /lib/libc.so.6 (0x40030000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x40153000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

For ls(1) to work in a `chroot`ed environment, the above libraries must be available in your `chroot`ed environment.

See ???.

Memory leak detection tools#

There are several memory leak detection tools available in Debian.

List of memory leak detection tools

Static code analysis tools#

There are lint like tools for static code analysis.

List of tools for static code analysis

Disassemble binary#

You can disassemble binary code with objdump(1) by the following.

$  objdump -m i386 -b binary -D /usr/lib/grub/x86_64-pc/stage1

[!NOTE] gdb(1) may be used to disassemble code interactively.

Flex — a better Lex#

Flex is a Lex-compatible fast lexical analyzer generator.

Tutorial for flex(1) can be found in "info flex".

You need to provide your own "main()" and "yywrap()". Otherwise, your flex program should look like this to compile without a library. This is because that "yywrap" is a macro and "%option main" turns on "%option noyywrap" implicitly.

%option main
%%
.|\n    ECHO ;
%%

Alternatively, you may compile with the "-lfl" linker option at the end of your cc(1) command line (like AT&T-Lex with "-ll"). No "%option" is needed in this case.

Bison — a better Yacc#

Several packages provide a Yacc-compatible lookahead LR parser or LALR parser generator in Debian.

List of Yacc-compatible LALR parser generators

Tutorial for bison(1) can be found in "info bison".

You need to provide your own "main()" and "yyerror()". "main()" calls "yyparse()" which calls "yylex()", usually created with Flex.

%%

%%

Autoconf#

Autoconf is a tool for producing shell scripts that automatically configure software source code packages to adapt to many kinds of Unix-like systems using the entire GNU build system.

autoconf(1) produces the configuration script "configure". "configure" automatically creates a customized "Makefile" using the "Makefile.in" template.

Compile and install a program#

[!WARNING] Do not overwrite system files with your compiled programs when installing them.

Debian does not touch files in "/usr/local/" or "/opt". So if you compile a program from source, install it into "/usr/local/" so it does not interfere with Debian.

$ cd src
$ ./configure --prefix=/usr/local
$ make
$ make install # this puts the files in the system

Uninstall program#

If you have the original source and if it uses autoconf(1)/automake(1) and if you can remember how you configured it, execute as follows to uninstall the program.

$ ./configure "all-of-the-options-you-gave-it"
# make uninstall

Alternatively, if you are absolutely sure that the install process puts files only under "/usr/local/" and there is nothing important there, you can erase all its contents by the following.

# find /usr/local -type f -print0 | xargs -0 rm -f

If you are not sure where files are installed, you should consider using checkinstall(8) from the checkinstall package, which provides a clean path for the uninstall. It now supports to create a Debian package with "-D" option.

Perl short script madness#

Although any AWK scripts can be automatically rewritten in Perl using a2p(1), one-liner AWK scripts are best converted to one-liner Perl scripts manually.

Let's think following AWK script snippet.

awk '($2=="1957") { print $3 }' |

This is equivalent to any one of the following lines.

perl -ne '@f=split; if ($f[1] eq "1957") { print "$f[2]\n"}' |

perl -ne 'if ((@f=split)[1] eq "1957") { print "$f[2]\n"}' |

perl -ne '@f=split; print $f[2] if ( $f[1]==1957 )' |

perl -lane 'print $F[2] if $F[1] eq "1957"' |

perl -lane 'print$F[2]if$F[1]eq+1957' |

The last one is a riddle. It took advantage of following Perl features.

• The whitespace is optional.

• The automatic conversion exists from number to the string.

See perlrun(1) for the command-line options. For more crazy Perl scripts, Perl Golf may be interesting.

Web#

Basic interactive dynamic web pages can be made as follows.

• Queries are presented to the browser user using HTML forms.

• Filling and clicking on the form entries sends one of the following URL string with encoded parameters from the browser to the web server.

• "http://www.foo.dom/cgi-bin/program.pl?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3"

• "http://www.foo.dom/cgi-bin/program.py?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3"

• "http://www.foo.dom/program.php?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3"

• "%nn" in URL is replaced with a character with hexadecimal nn value.

• The environment variable is set as: "QUERY_STRING="VAR1=VAL1 VAR2=VAL2 VAR3=VAL3"".

• CGI program (any one of "program.*") on the web server executes itself with the environment variable "$QUERY_STRING".

• stdout of CGI program is sent to the web browser and is presented as an interactive dynamic web page.

For security reasons it is better not to hand craft new hacks for parsing CGI parameters. There are established modules for them in Perl and Python. PHP comes with these functionalities. When client data storage is needed, HTTP cookies are used. When client side data processing is needed, Javascript is frequently used.

For more, see the Common Gateway Interface, The Apache Software Foundation, and JavaScript.

Searching "CGI tutorial" on Google by typing encoded URL http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial directly to the browser address is a good way to see the CGI script in action on the Google server.

The source code translation#

There are programs to convert source codes.

List of source code translation tools

Making Debian package#

If you want to make a Debian package, read followings.

• ??? to understand the basic package system

• ??? to understand basic porting process

• ??? to understand basic chroot techniques

• debuild(1), pbuilder(1) and pdebuild(1)

• Debugging the Debian package for recompiling for debugging

• Debian New Maintainers' Guide as tutorial (the maint-guide package)

• Debian Developer's Reference (the developers-reference package)

• Debian Policy Manual (the debian-policy package)

• Guide for Debian Maintainers (the debmake-doc package)

There are packages such as debmake, dh-make, dh-make-perl, etc., which help packaging.