The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
unzip particular gzip files among the normal data files thepurple Shell Programming and Scripting 4 11-30-2007 11:17 AM
gzip all the files in a directory er_ashu UNIX for Dummies Questions & Answers 2 11-06-2007 09:05 AM
Need to gzip LARGE files LordJezo UNIX for Dummies Questions & Answers 2 05-02-2005 04:18 PM
gzip, multiple files smbodnar UNIX for Dummies Questions & Answers 2 11-11-2002 04:29 PM
Two Files Created For Every One? Atama UNIX for Dummies Questions & Answers 1 04-12-2002 04:44 PM

 
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
Prev Previous Post   Next Post Next
  #1 (permalink)  
Old 10-13-2008
eisenhorn eisenhorn is offline
Registered User
  
 

Join Date: Oct 2008
Posts: 3
Gzip files as they are created

Hello. I have a scripting query that I am stumped on which I hope you can help with.

Basically, I have a ksh script that calls a process to create n number of binary files. These files have a maximum size of 1Gb. The process can write n number of files at once (parallel operation) based on the paralellisation parameter fed into the script at the start. Normally we would wait for this process to complete and then gzip all the files individually (gzip *.dmp for example). However, on some systems we don't have enough disk space to wait until all the 1Gb files have been produced.

I have previously written some code to gzip the files in parallel (see below), however, I now need to gzip them in parallel whilst the first process runs. I need to be careful not to attempt to gzip any files currently being written (up to n from the parallel command), so some sort of looping will be required. And I want to maintain the option of parallel gzip if possible.

Code:
...
gzip_func() {
started=0
threads=4
for filename in `ls -1 ${EXP_DIR}/*.dmp`
do
 if [[ ${started} -lt ${threads} ]]; then
  let started=started+1
  echo "gzip ${filename}"
  ( $GZIPCMD ${filename} ) &
  list_of_pids="${list_of_pids} $!"
 else
  print "wait ${list_of_pids}"
  wait ${list_of_pids}
  list_of_pids=""
  started=0
 fi
done
}
...
my_binary_file_creation_process
...
while [ `find ${EXP_DIR} -name \*.dmp|wc -l` -gt "0" ]; do
 gzip_func
 print "wait ${list_of_pids}"
 wait ${list_of_pids}
 list_of_pids=""
done
Can anyone help me write some code for this using standard solaris 8/9/10 tools using the korn shell. Perl commands should be possible (vers 5.6.1 installed).

Many thanks and Best Regards,
Stephen.
 

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 09:57 AM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0