This page presents programs and tips about log. Own your risk.

INDEX:

• Log form analizer
• Log recorder - stq
• Log reducer for common format

Many programs write several form in one log file. It cause confuse of readers. This program assists reader doing. You can get current version of source.

The program read a line from log flie, guesses token type, classified forms with sequence of token and memorize them. When reach to end, the program reports classified forms.

There are 3 report style.

1. 'short' style shows essence.
2. 'long' style shows all entry.
3. 'perl' style shows form with perl like regular expression.

example

Sample file like fstab:

% cat x # Device Mountpoint FStype Options Dump Pass# /dev/wd0s1b none swap sw 0 0 /dev/wd0s1a / ufs rw 1 1 /dev/sd0s1e /usr ufs rw 2 2 /dev/wd0s1e /var ufs rw 2 2 /dev/wcd0c /cdrom cd9660 ro,noauto 0 0

'short' style:

% ./a.out -c -rs -pe /etc/fstab # file '/etc/fstab' # forms # # # sig num count ; types.. 0 <074> 6 6 ; @word @word @word @word @number @number

'long' style:

% ./a.out -c -rl -pe /etc/fstab # file '/etc/fstab' === forms (long) 0 tokenset 0x415c, num 6, sig 074, count 6, list 0x18000 0 0x18000 3 0 1 '/dev/wd0s1b' 1 1 '/dev/wd0s1a' 2 1 '/dev/sd0s1e' 3 1 '/dev/wd0s1e' 4 1 '/dev/sd0s1f' 5 1 '/dev/wcd0c' 1 0x18400 3 0 1 'none' 1 1 '/' 2 1 '/usr' 3 1 '/var' 4 1 '/www' 5 1 '/cdrom' 2 0x18800 3 0 1 'swap' 1 4 'ufs' 2 1 'cd9660' 3 0x18c00 3 0 1 'sw' 1 4 'rw' 2 1 'ro,noauto' 4 0x1a000 4 0 2 '0' 1 1 '1' 2 3 '2' 5 0x1a400 4 0 2 '0' 1 1 '1' 2 3 '2'

'perl' style:

% ./a.out -c -rp -pe /etc/fstab # file '/etc/fstab' # perl like regular expression # 0th, count 6 /^\s*([^\d\s]+\S+)\s+([^\d\s]+\S+)\s+([^\d\s]+\S+)\s+([^\d\s]+\S+)\s+([\d\.]+)\s+([\d\.]+)$/

In monitoring or debugging, people want to view message of programs. However, message printing to terminal causes overhead under heavy load. Writing to file is more wrong because driving disk cause more overhead. Futhermore, file size often reachs several gigabytes. Another solution is required.

'stq' receives messages from stdin and stores last several thousands messages when termination itself. Overhead of running is little. and file writing ovehead occues only termination.

stq is not same as redirection to file, not 'tee' .

You can get snap-shot source of 'stq'.

% vmstat 5 | stq vmstat.log % tail -f /var/adm/messages | stq -t -a SU su.log

httpd common format

'common format' is classic and most famous format of httpd. It is used CERN httpd, NCSA httpd and Apache.

log reducer

log often be very long. This program reduces entris of log. In our trials, it reduced up to 85% (log files are compressed to 15%-24%). You will compress to 3%-5% using this program and compress program like gzip. You can get snapshot of source.

However, this program does not keep order of entities. If you want that, this program does not match such needs.

Conntact Us

If you have comments and requests, please send e-mail to <k-chinen @ is.aist-nara.ac.jp> .