gzip vs pipe gzip: produce different file size


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users gzip vs pipe gzip: produce different file size
# 1  
Old 06-03-2011
gzip vs pipe gzip: produce different file size

Hi All,

I have a random test file: test.txt, size: 146
$ ll test.txt
$ 146 test.txt

Take 1:
$ cat test.txt | gzip > test.txt.gz
$ ll test.txt.gz
$ 124 test.txt.gz

Take 2:
$ gzip test.txt
$ ll test.txt.gz
$ 133 test.txt.gz

As you can see, gzipping a file and piping into gzip produce different sized *gz files. The problem currently is that some archivers, such as WinZip and Cygwin's gunzip have to unzip the piped gzip file twice before it can be read as a plain text file. Why is this and how can I ensure piping into gzip produces the same file as gzipping a file?

Thanks,
# 2  
Old 06-03-2011
Hi.

I did not need to unzip twice. Perhaps this will help explain the difference in lengths:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate gzip lengths.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
pe() { for i;do printf "%s" "$i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for i;do printf "%s" "$i";done; printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C spit gzip

rm -f f1 f1.gz f2 f2.gz f3 f3.gz f4 f4.gz f5 f5.gz
spit -c 10 -r 5 > f1
cp f1 f2
cp f1 f3

pl " Input data files f1, f2:"
wc f1 f2

pl " Results, re-direction:"
cat f1 | gzip > f1.gz
wc f1.gz
rm -f f1

pl " Results, by filename:"
gzip -n f2
wc f2.gz

pl " Results, by filename, plain:"
gzip f3
wc f3.gz

cp f1.gz f4.gz
cp f3.gz f5.gz

pl " All files compressed, contents:"
ls -lgG f*
pe
file f*

pl " All files uncompressed:"
for f in *.gz
do
  gunzip $f
done
ls -lgG f*

exit 0

producing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
GNU bash 3.2.39
spit - ( local: RepRev 1.3, ~/bin/spit, 2011-04-29 )
gzip 1.3.12

-----
 Input data files f1, f2:
  5  50 100 f1
  5  50 100 f2
 10 100 200 total

-----
 Results, re-direction:
 0  1 30 f1.gz

-----
 Results, by filename:
 0  1 30 f2.gz

-----
 Results, by filename, plain:
 0  1 33 f3.gz

-----
 All files compressed, contents:
-rw-r--r-- 1 30 Jun  3 21:45 f1.gz
-rw-r--r-- 1 30 Jun  3 21:45 f2.gz
-rw-r--r-- 1 33 Jun  3 21:45 f3.gz
-rw-r--r-- 1 30 Jun  3 21:45 f4.gz
-rw-r--r-- 1 33 Jun  3 21:45 f5.gz

f1.gz: gzip compressed data, from Unix, last modified: Fri Jun  3 21:45:43 2011
f2.gz: gzip compressed data, from Unix
f3.gz: gzip compressed data, was "f3", from Unix, last modified: Fri Jun  3 21:45:43 2011
f4.gz: gzip compressed data, from Unix, last modified: Fri Jun  3 21:45:43 2011
f5.gz: gzip compressed data, was "f3", from Unix, last modified: Fri Jun  3 21:45:43 2011

-----
 All files uncompressed:
-rw-r--r-- 1 100 Jun  3 21:45 f1
-rw-r--r-- 1 100 Jun  3 21:45 f2
-rw-r--r-- 1 100 Jun  3 21:45 f3
-rw-r--r-- 1 100 Jun  3 21:45 f4
-rw-r--r-- 1 100 Jun  3 21:45 f5

See man gzip for details.

Best wishes ... cheers, drl
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

MIME type for sending gzip file as attachment in email

Hello, I am trying to send a gzip file on email using below command but the zipped file received on email is corrupt. mailsend -smtp $smtpip -content-type 'application/x-gzip' -mime-type "application/x-gzip" -t $receiver -f $sender -sub "$subject" -M "$MSG" -attach $file file name is ... (1 Reply)
Discussion started by: tushar.modgil
1 Replies

2. Shell Programming and Scripting

Shell script not able to zip the file using gzip

Hi all, I am calling Temp.sh and it is has simple line $gpath=`which gzip` $gpath $FilePath/My_temp.log if I run this script, logging to server then its works fine. But when I send this script over the SSH it does not work at it. gzip is command is not execute. I am using gzip 1.6... (2 Replies)
Discussion started by: girijajoshi
2 Replies

3. Shell Programming and Scripting

File name manipulation when extracting from gzip

Hi, Below is the description of my problem. I am trying to loop through the below file names sitting in the file - FileNames.txt, and trying to decompress them FileNames.txt 20150408_MGAC_.txt.gz 20150408_MGCC_.txt.gz 20150408_MGSH__.txt.gz 20150408_MGSL__.txt.gz ... (6 Replies)
Discussion started by: Richmond_CS
6 Replies

4. Shell Programming and Scripting

Check if file is locked otherwise gzip it

Dear community, I've a continuos tcpdump on redhat that close the dumped file every 100000 captured packets. To avoid disk full I would like to gzip the closed *.cap file. But how can I check if the file is currently opened by tcpdump and skip it from gzip? Thanks! EDIT: Just to post an... (9 Replies)
Discussion started by: Lord Spectre
9 Replies

5. UNIX for Dummies Questions & Answers

gzip of a file in the same folder

I want gzip of a file in the same folder where it is kept now $filename = '/var/dev/test.txt' /opt/home/>> gzip -c $filename > test.txt.gz however command creates it in the folder in /opt/home/ How to gzip a file in the same directory where it is now , no matter from where we execute and also... (2 Replies)
Discussion started by: lalitpct
2 Replies

6. Programming

gzip on pipe error handling

Hi all... I have the following code: FILE *fp = popen(" gzip -dc /somemount/somefile.gz", "r"); while(fgets(buffer, 1024, fp)) { some code.... } "/somemount" is a mount of some network drive. Sometimes error occurs in while loop - I can see the following "Input/Output error"... (4 Replies)
Discussion started by: adm1n
4 Replies

7. Shell Programming and Scripting

Appending a GZIP File

Guys, I just want to know the difference in following (core difference) zcat a.gz b.gz c.gz |gzip >d.gz And zcat a.gz >>d.gz zcat b.gz >>d.gz zcat c.gz >>d.gz do we have 3 gzip header in 1st and only one in second case. please let me know this in detail Thanks,... (2 Replies)
Discussion started by: mohan_xunil
2 Replies

8. Solaris

gzip a file and append creation date stamp to file

I want to gzip a file and append the creation date to the end of the file. How can I accomplish this task. Basically they are log files which need a creation date stamp appended to make sure they do not overwrite other log files. -jack (3 Replies)
Discussion started by: jacktravine
3 Replies

9. Solaris

script for Gzip thousands of file

Hi experts, I have thousands of file (data file and Gziped file) in same directory like below-- bash-2.05$ pwd /home/mmc bash-2.05$ file PP023149200709270546 TT023149200709270546: gzip compressed data - deflate method bash-2.05$ file PP027443200711242320 TT027443200711242320: ... (10 Replies)
Discussion started by: thepurple
10 Replies

10. UNIX for Dummies Questions & Answers

How do I send a file as an attachment (gzip file) on a Unix system

Hi, How do I send a file as an attachment (gzip file) on a Unix system ? Using sendmail. Please help me. :confused: (3 Replies)
Discussion started by: lacca
3 Replies
Login or Register to Ask a Question