Hello. I have a scripting query that I am stumped on which I hope you can help with.
Basically, I have a ksh script that calls a process to create
n number of binary files. These files have a maximum size of 1Gb. The process can write
n number of files at once (parallel operation) based on the paralellisation parameter fed into the script at the start. Normally we would wait for this process to complete and then gzip all the files individually (gzip *.dmp for example). However, on some systems we don't have enough disk space to wait until all the 1Gb files have been produced.
I have previously written some code to gzip the files in parallel (see below), however, I now need to gzip them in parallel whilst the first process runs. I need to be careful not to attempt to gzip any files currently being written (up to
n from the parallel command), so some sort of looping will be required. And I want to maintain the option of parallel gzip if possible.
Code:
...
gzip_func() {
started=0
threads=4
for filename in `ls -1 ${EXP_DIR}/*.dmp`
do
if [[ ${started} -lt ${threads} ]]; then
let started=started+1
echo "gzip ${filename}"
( $GZIPCMD ${filename} ) &
list_of_pids="${list_of_pids} $!"
else
print "wait ${list_of_pids}"
wait ${list_of_pids}
list_of_pids=""
started=0
fi
done
}
...
my_binary_file_creation_process
...
while [ `find ${EXP_DIR} -name \*.dmp|wc -l` -gt "0" ]; do
gzip_func
print "wait ${list_of_pids}"
wait ${list_of_pids}
list_of_pids=""
done
Can anyone help me write some code for this using standard solaris 8/9/10 tools using the korn shell.
Perl commands should be possible (vers 5.6.1 installed).
Many thanks and Best Regards,
Stephen.
More UNIX and Linux Forum Topics You Might Find Helpful
Powered by 
10-13-2008
Threaded Mode
