Shell Programming and Scripting

Hi,

I've got a problem with awk.

On a solaris box:

SunOS 5.8 Generic_108528-23 sun4u sparc SUNW,Sun-Fire-280R,

I am running the follow awk script:

# awk -f jawk ntest.dat
awk: too many output files 10
record number 11
#

Where jawk is:

#!/usr/bin/ksh

{
printf "%-9s%8s %14s %12s %12s %12s %4s\n",$2,$3,$4,$5,$6,$7,$8 > "/tmp/db_"$1"_size.dat"
}

and ntest.dat is:

dbase01 20060105 0 222429184 8650752 3407872 589824 0
dbase02 20060105 16790 902758400 152109056 137756672 40566784 24
dbase03 20060105 4615 358940672 93650944 3473408 589824 0
dbase04 20060105 12386 2824798208 920715264 897843200 2031616 2
dbase05 20060105 20020 1939275776 381419520 637599744 56950784 0
dbase06 20060105 11546 987693056 109379584 108724224 49283072 4
dbase07 20060105 52690 6660947968 1686372352 2625372160 720896 1
dbase08 20060105 0 5020123136 25624576 3138387968 589824 3
dbase09 20060105 5916 1072562176 214040576 255655936 124583936 2
dbase10 20060105 13552 2158428160 237043712 1011941376 275578880 6
dbase11 20060105 109663 7101677568 1193672704 1851523072 40960000 0
dbase12 20060105 4425 361693184 50397184 61931520 589824 2
dbase13 20060105 215021 9385213952 1095434240 2484994048 1572864 2
dbase14 20060105 16993 1485438976 351731712 264241152 21954560 7
dbase15 20060105 11573 704905216 127008768 146800640 786432 4
dbase16 20060105 34272 1623719936 365821952 599457792 37748736 3
dbase17 20060105 60282 5028380672 617742336 2058551296 88866816 8
dbase18 20060105 36540 3106340864 1340669952 274595840 14155776 5
dbase19 20060105 0 291241984 8781824 3407872 589824 0
dbase20 20060105 128790 10186063872 1776091136 5656215552 121110528 0
dbase21 20060105 14211 1107689472 365821952 176881664 107151360 6
dbase22 20060105 26699 1458503680 113442816 361168896 2555904 8

Despite the error message when the script is run, output is produced
in /tmp for the first 10 rows in ntest.dat:

# ls -lt /tmp/*.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase01_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase02_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase03_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase04_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase05_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase06_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase07_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase08_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase09_size.dat
-rw-r--r-- 1 root other 77 Jan 6 12:12 /tmp/db_dbase10_size.dat

When I reduce the number of rows in the ntest.dat file to 9 and run:

# awk -f jawk ntest.dat

This runs succesfully, as it does for ntest.dat when there are 10 rows in ntest.dat.

However, when there are 11 rows in ntest.dat the script errors with the error:

awk: too many output files 10
record number 11

I have tried deleting the current 11th row in the file to eliminate the data in the current 11th row, but still get the same error.
I have also tried different combination of rows on to get 11+ rows in the
ntest.dat file; whenever there are more than 10 rows in the ntest.dat file
the awk script errors with same error, despite generating the correct output
for the first 10 rows in ntest.dat.

This implies that the error is not associated with the awk processing, but
that the awk script is reporting an error from the OS. There is ample
space in /tmp (the output files are 77 Kb).

On this server in /usr/include/limits.h the following parameters are defined:

#define CHILD_MAX 25 /* max # of processes per user id */

#define OPEN_MAX 256 /* max # of files a process can have open */

Running ps -efl | grep root | grep -v grep | wc -l before and after running the awk script returns 71.

Does anyone know why awk behaves in this way?

Thanks in anticipation,

Jon

Code:

#!/usr/bin/nawk -f

{
output="/tmp/db_" $1 "_size.dat"
printf "%-9s%8s %14s %12s %12s %12s %4s\n",$2,$3,$4,$5,$6,$7,$8 > output
close(output)
}

Thank-you very much, nawk has solved the problem.
Can anyone explain what the limitations of awk in the initial problem
were?

Thanks in anticipation,

J

The problem was with the number of the concurrently openned streams, i.e. openned file descriptors you're writting to:

Code:

..... > "/tmp/db_"$1"_size.dat"

Most awk's have a limit on how many files/streams you can keep openned at any given time. In the case of the Solaris' "awk" [a really old version of awk] I believe it's 10.

You kept creating output streams/files based on the value of the first column of the file. Therefore the number of streams could go OVER the maximum of 10.

The stream/file will remain openned till you close it explicitely with the 'close' call. Using the 'close' keeps the number of the concurrently openned streams/files to a minimum NOT going over the maximum numbe of 10.

Hope it helps.

Thanks for taking the time to explain that to me.
A working version of the script has now been implemented.

J