Gzip behavior on open files? Post: 302798075

Sponsored Content

Top Forums Shell Programming and Scripting Gzip behavior on open files? Post 302798075 by dan-e on Tuesday 23rd of April 2013 07:09:26 PM

04-23-2013

Registered User

Gzip behavior on open files?

Just a quick question: How does gzip behave under linux if it's source is a file that is currently being written to by a different process? Basically, in my below code I want to make sure that there is ultimately no loss of data; ie. the gzip command runs until it "catches up" to the end of the file while the file is expanding, and then the cat /dev/null clears the file immediately, therefore the next write to the file happens when it is empty, and all prior data in the file is safely preserved in the archived gzip file. How does my code look?

Code:

CAPDIR=/data/capture
KEEPDIR=/data/capture/keep

for FILE in `find $CAPDIR -maxdepth 1 -not -type d | awk -F/ '{print $NF}'`
do
   echo Processing $CAPDIR/$FILE --\> $KEEPDIR/$FILE.GZ
   gzip -c /$CAPDIR/$FILE  > $KEEPDIR/$FILE.GZ
   cat /dev/null > $CAPDIR/$FILE
done
echo
echo Done. 
echo

I know in some OS's that when a file handle is locked for reading you get the file contents up to the EOF at the time of lock, not up to the EOF at the current time.

I guess another way to put my question would be is there a way to "atomize" these commands:

Code:

   gzip -c /$CAPDIR/$FILE  > $KEEPDIR/$FILE.GZ
   cat /dev/null > $CAPDIR/$FILE

...such that I can be guaranteed that no other process gets a chance to write data to $CAPDIR/$FILE in between the call to gzip and the call to cat /dev/null?

Last edited by dan-e; 04-23-2013 at 10:19 PM..

dan-e

View Public Profile for dan-e

Find all posts by dan-e

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

gzip, multiple files

Hello Everyone, Here is what I am trying to do. I have four text files, I want to gzip them under unix and mail the zipped file via outlook. I am able to do this easily enough, but using winzip or pkunzip to unzip the file, there is only one file. (In essence, all four files were...

2. UNIX for Dummies Questions & Answers

Need to gzip LARGE files

The windows version of gzip supports pretty much unlimited file sizes while the one we have in solaris only goes up to a set size, one or two gigs I think. Is there a new version of gzip I can put on our systems that supports massive file sizes?

3. UNIX for Dummies Questions & Answers

gzip all the files in a directory

Hi, There are multiple files in a directory with different names.How can they be gzipped such that the timestamp of the files is not changed.

4. Shell Programming and Scripting

unzip particular gzip files among the normal data files

Hello experts, I run Solaris 9. I have a below script which is used for gunzip the thousand files from a directory. ---- #!/usr/bin/sh cd /home/thousands/gzipfiles/ for i in `ls -1` do gunzip -c $i > /path/to/file/$i done ---- In my SAME directory there thousand of GZIP file and also...

5. Shell Programming and Scripting

Gzip files as they are created

Hello. I have a scripting query that I am stumped on which I hope you can help with. Basically, I have a ksh script that calls a process to create n number of binary files. These files have a maximum size of 1Gb. The process can write n number of files at once (parallel operation) based on the...

6. Shell Programming and Scripting

gzip the files with particular extension

Is there any way to compress only the files with .xml extension within a folder which in turn has many sub folders? gzip -r9 path/name/*.xml is not working This compression is done in the Windows server using Batch script.

7. Shell Programming and Scripting

gzip files with extension

Hi, I have 1000 of files in a folder with the file extension as .csv In this some of the files are already zipped and its looks like filename.csv.gz Now i need to zip all the files in the folder to free some disk space. When i give gzip *.csv It prompts me to overwrite filename.csv.gz...

8. UNIX for Advanced & Expert Users

gzip vs pipe gzip: produce different file size

Hi All, I have a random test file: test.txt, size: 146 $ ll test.txt $ 146 test.txt Take 1: $ cat test.txt | gzip > test.txt.gz $ ll test.txt.gz $ 124 test.txt.gz Take 2: $ gzip test.txt $ ll test.txt.gz $ 133 test.txt.gz As you can see, gzipping a file and piping into gzip...

9. Shell Programming and Scripting

help with a script to gzip/move files

Hi Please can you help me in writing a script to find files on a specific directory, and of extension "tap" but only of the month of september, gzip and move them to another directory. Your help will be appreciated.

10. Shell Programming and Scripting

tar and gzip files

Hi Guys, I am using RHEL5 and Solaris 9 & 10. I want to tar and gzip my files then remove them after a successful tar command... Lets say I have files with extension .arc then I want to tar and gzip these files. After successful tar command I want to remove all these files (i.e .arc). ...

LEARN ABOUT SUNOS

gzexe

GZEXE(1)						      General Commands Manual							  GZEXE(1)

NAME

       gzexe - compress executable files in place

SYNOPSIS

       gzexe [ name ...  ]

DESCRIPTION

       The  gzexe  utility  allows you to compress executables in place and have them automatically uncompress and execute when you run them (at a
       penalty in performance).  For example if you execute ``gzexe /bin/cat'' it will create the following two files:
	   -r-xr-xr-x  1 root  bin   9644 Feb 11 11:16 /bin/cat
	   -r-xr-xr-x  1 bin   bin  24576 Nov 23 13:21 /bin/cat~
       /bin/cat~ is the original file and /bin/cat is the self-uncompressing executable file.  You can remove /bin/cat~ once  you  are	sure  that
       /bin/cat works properly.

       This utility is most useful on systems with very small disks.

OPTIONS

       -d     Decompress the given executables instead of compressing them.

SEE ALSO

       gzip(1), gznew(1), gzmore(1), gzcmp(1), gzforce(1)

CAVEATS

       The  compressed	executable  is a shell script. This may create some security holes. In particular, the compressed executable relies on the
       PATH environment variable to find gzip and some other utilities (tail, chmod, ln, sleep).

BUGS

       gzexe attempts to retain the original file attributes on the compressed executable, but you may have to fix them manually  in  some  cases,
       using chmod or chown.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +--------------------+-----------------+
       |  ATTRIBUTE TYPE    | ATTRIBUTE VALUE |
       +--------------------+-----------------+
       |Availability	    | SUNWgzip	      |
       +--------------------+-----------------+
       |Interface Stability | External	      |
       +--------------------+-----------------+
NOTES

       Source for gzip is available in the SUNWgzipS package.

																	  GZEXE(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

gzip, multiple files

Discussion started by: smbodnar

2. UNIX for Dummies Questions & Answers

Need to gzip LARGE files

Discussion started by: LordJezo

3. UNIX for Dummies Questions & Answers

gzip all the files in a directory

Discussion started by: er_ashu

4. Shell Programming and Scripting

unzip particular gzip files among the normal data files

Discussion started by: thepurple