Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Gzip files as they are created
Top Forums Shell Programming and Scripting Gzip files as they are created Post 302247159 by eisenhorn on Wednesday 15th of October 2008 05:02:15 AM
Old 10-15-2008
Quote:
Originally Posted by cfajohnson
As soon as a new file is created, you can gzip the previous one.
Not the most enlightening statement Smilie but I understand what you mean.

I did some playing and found that the export program will create a file of 4k initially, then stop using it while it builds up a list of objects to export. It then returns a lock on the file and fills the file up to the 1gb size.

I modified my code to use a du and fuser test to check that the file was bigger than 4k and was not being used by any user. I find that if i use parallelism on the export the program will create n number of files and then populate them, it may start with files 1,2,3, and 4, but 1,3, and 4 suddenly reach 1Gb so it creates 5,6,7 to continue the parallel tasks. file 2 is still not full (for whatever reason).

Files 1,3, and 4 are now unused but the gzip_func does not seem to want to gzip the files until file 2 is also unused - which often might not be until the end of the export. Can you please have a look at the code below and see if you can spot an obvious error? I want the code to really start gzipping when the 2 tests are passed, whether it can only gzip 1 file or up to n threads. Any ideas?

Code:
gzip_func() {
started=0
threads=4
for filename in ${EXP_DIR}/*.dmp
do
 # Check if file is bigger than 8K and is not being used
 if [ `du -sk "${filename}"|awk '{print $1}'` -gt "8" ] && [ `fuser "${filename}" 2>/dev/null | wc -m` -eq "0" ]; then
   # Loop through files until 4 are started (to match threads)
   if [ ${started} -lt ${threads} ]; then
    started=$(( $started + 1 ))
    echo "gzip ${filename}"
    $GZIPCMD "${filename}" &
    list_of_pids="${list_of_pids} $!"
   else
    print "wait ${list_of_pids}"
    wait ${list_of_pids}
    list_of_pids=""
    started=0
   fi
 else
   echo "${filename} is still being written to, trying next file..."
 fi
done
}
 
# Export creation - note done in background to allow gzip loop to run
expdp '"/ as sysdba"' directory=DPUMP_DIR_ADHOC dumpfile=ram6_full%U.dmp logfile=ram6_full.log filesize=1024m full=y parallel=4 &
 
while [ `find ${EXP_DIR} -name \*.dmp|wc -l` -gt "0" ]; do
 gzip_func
 print "out of loop wait ${list_of_pids}"
 wait ${list_of_pids}
 list_of_pids=""
 sleep 5
done

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

gzip, multiple files

Hello Everyone, Here is what I am trying to do. I have four text files, I want to gzip them under unix and mail the zipped file via outlook. I am able to do this easily enough, but using winzip or pkunzip to unzip the file, there is only one file. (In essence, all four files were... (2 Replies)
Discussion started by: smbodnar
2 Replies

2. UNIX for Dummies Questions & Answers

Need to gzip LARGE files

The windows version of gzip supports pretty much unlimited file sizes while the one we have in solaris only goes up to a set size, one or two gigs I think. Is there a new version of gzip I can put on our systems that supports massive file sizes? (2 Replies)
Discussion started by: LordJezo
2 Replies

3. UNIX for Dummies Questions & Answers

gzip all the files in a directory

Hi, There are multiple files in a directory with different names.How can they be gzipped such that the timestamp of the files is not changed. (2 Replies)
Discussion started by: er_ashu
2 Replies

4. Shell Programming and Scripting

unzip particular gzip files among the normal data files

Hello experts, I run Solaris 9. I have a below script which is used for gunzip the thousand files from a directory. ---- #!/usr/bin/sh cd /home/thousands/gzipfiles/ for i in `ls -1` do gunzip -c $i > /path/to/file/$i done ---- In my SAME directory there thousand of GZIP file and also... (4 Replies)
Discussion started by: thepurple
4 Replies

5. Shell Programming and Scripting

gzip the files with particular extension

Is there any way to compress only the files with .xml extension within a folder which in turn has many sub folders? gzip -r9 path/name/*.xml is not working This compression is done in the Windows server using Batch script. (2 Replies)
Discussion started by: Codesearcher
2 Replies

6. Shell Programming and Scripting

gzip files with extension

Hi, I have 1000 of files in a folder with the file extension as .csv In this some of the files are already zipped and its looks like filename.csv.gz Now i need to zip all the files in the folder to free some disk space. When i give gzip *.csv It prompts me to overwrite filename.csv.gz... (5 Replies)
Discussion started by: nokiak810
5 Replies

7. UNIX for Dummies Questions & Answers

Gzip files - Excluded directories

Hi, I am using the commande line find . -name "*.nc" -type f -exec gzip -v {} \; to zip all files with the extension " *.nc " in all directories. But I am looking for a way to excluded some directories as the command will recursively check all of them. If somone can help me with some... (4 Replies)
Discussion started by: Aswex
4 Replies

8. UNIX for Advanced & Expert Users

gzip vs pipe gzip: produce different file size

Hi All, I have a random test file: test.txt, size: 146 $ ll test.txt $ 146 test.txt Take 1: $ cat test.txt | gzip > test.txt.gz $ ll test.txt.gz $ 124 test.txt.gz Take 2: $ gzip test.txt $ ll test.txt.gz $ 133 test.txt.gz As you can see, gzipping a file and piping into gzip... (1 Reply)
Discussion started by: hanfresco
1 Replies

9. Shell Programming and Scripting

tar and gzip files

Hi Guys, I am using RHEL5 and Solaris 9 & 10. I want to tar and gzip my files then remove them after a successful tar command... Lets say I have files with extension .arc then I want to tar and gzip these files. After successful tar command I want to remove all these files (i.e .arc). ... (3 Replies)
Discussion started by: Phuti
3 Replies

10. Shell Programming and Scripting

Listing the file name and no of records in each files for the files created on a specific day

Hi, I want to display the file names and the record count for the files in the 2nd column for the files created today. i have written the below command which is listing the file names. but while piping the above command to the wc -l command its not working for me. ls -l... (5 Replies)
Discussion started by: Showdown
5 Replies

LEARN ABOUT XFREE86

started

started(7)						 Miscellaneous Information Manual						started(7)

NAME
       started - event signalling that a job is running

SYNOPSIS
       started JOB=JOB INSTANCE=INSTANCE [ENV]...

DESCRIPTION
       The  started  event is generated by the Upstart init(8) daemon when an instance of a job has finished starting and is now running.  The JOB
       environment variable contains the job name, and the INSTANCE environment variable contains the instance name which will be empty  for  sin-
       gle-instance jobs.

       init(8)	emits this event as an informational signal, services and tasks started or stopped by this event will do so in parallel with other
       activity.  It is typically combined with the stopping(7) event by services declaring a dependency.

       Job configuration files may use the export stanza to export environment variables from their own environment into the started  event.   See
       init(5) for more details.

EXAMPLE
       A service that wishes to depend on another service might use:

	      start on started apache
	      stop on stopping apache

       A task that must be run after another task or service has been started might use:

	      start on started postgresql

SEE ALSO
       starting(7) stopping(7) stopped(7) init(5)

Upstart 							    2009-07-09								started(7)
All times are GMT -4. The time now is 11:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy