Zgrep output to another compressed file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Zgrep output to another compressed file
# 1  
Old 04-10-2016
Zgrep output to another compressed file

Hi,

I have a big (~15G) compressed file having around 170M records and I need to exclude around 4k bad records (\n in the string) .
The typical steps would have been

1. zgrep required records into new file
Code:
zgrep [^0-9] big15GFile.dat.gz > newBig64GFile.dat

2. zip back new file
Code:
gzip newBig64GFile.dat

Is there a more efficient option?
This would end up taking considerable amount of time to unzip and zip.

Any pointers would be welcome

Thanks
# 2  
Old 04-10-2016
By definition, compressing a file trades disk space for time needed to process the compressed data. You can shave a little bit of time off of the sequence:
Code:
zgrep [^0-9] big15GFile.dat.gz > newBig64GFile.dat
gzip newBig64GFile.dat

using:
Code:
zgrep [^0-9] big15GFile.dat.gz | gzip > newBig64GFile.dat.gz

Since you just write the selected uncompressed data into the pipe instead of writing it to a file and reading it again in gzip. (And, it also takes less disk space.) The time spent decompressing the file and compressing the new file won't change. But the time spent writing and reading the uncompressed data will be reduced.
This User Gave Thanks to Don Cragun For This Post:
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Size of compressed file

Hi All, Is there is any way to find the size of compressed file without doing decompression. The size should give the original uncompressed data size Thanks Arun (3 Replies)
Discussion started by: arunkumar_mca
3 Replies

2. UNIX for Dummies Questions & Answers

compressed and tar file integrity

How can I ensure the folder that I tar and compress is good to be archive in DVD or tape? Must I uncompress and untar the file, or there is any way to tell the integerity of the compressed file before send to archive? I have bad experience on this, which the archive compressed file cold not be... (2 Replies)
Discussion started by: vivien_chu
2 Replies

3. Shell Programming and Scripting

compressed file

i have a file 4d7a94d0.bbb.1292 when i do file 4d7a94d0.bbb.1292 the ouput is below 4d7a94d0.bbb.1292: gzip compressed data - deflate method and i run this command gunzip -c 4d7a94d0.bbb.1292 | awk '{gsub("\"","")}/I_ACCOUNT_ID/{print $2}' RS=":|;" FS="," i get... (3 Replies)
Discussion started by: blackzinga80
3 Replies

4. Shell Programming and Scripting

Process a compressed file

Hi i have a filename.tar.bz2 and i have to parse it with a tool that doesn't support compressed files. I have to do it for many big files, so i can't decompress and then process. I'd like to do something like: tar -jxvf namefile.tar.bz2 | parsing_tool i mean analyze it directly,... (4 Replies)
Discussion started by: Dedalus
4 Replies

5. UNIX for Dummies Questions & Answers

compressed file

I compressed a file by using gzip command gzip <<xx>> filename changed to xx.gz How to view this xx.gz file. Any idea. Thanks in advance. (7 Replies)
Discussion started by: venkatesht
7 Replies

6. Shell Programming and Scripting

check to see if a file is compressed before trying to compress

I simply need to compress all files in a directory that are not already compressed and that are older than 10 days? I have this so far. I need to add to this so I don't try and compress file that are already compressed. Or if you think this can be simplified let me know. Thx. find... (3 Replies)
Discussion started by: rstone
3 Replies

7. UNIX for Advanced & Expert Users

Is it possible to see the content of the compressed file?

How we can view the content of the file,if it compressed (or) Zipped ,without uncompress ? I have one file ,i compressed it,without uncompressing the file.Is it possible to see the content of the file? (2 Replies)
Discussion started by: bobprabhu
2 Replies

8. UNIX for Dummies Questions & Answers

How to grep / zgrep to output ONLY the matching filename and line number?

Hi all, I am trying to zgrep / grep list of files so that it displays only the matching filename:line number and does not display the whole line, like: (echo "1.txt";echo "2.txt") | xargs zgrep -no STRING If I use -o option, it displays the matching STRING and if not used, displays the... (3 Replies)
Discussion started by: vvaidyan
3 Replies

9. UNIX for Advanced & Expert Users

Search first line of compressed file

I want to read a directory full of compressed files and move the file to another directory if it meets certain criteria. I only want to look at the first line of the compressed file and if I find the string, do the move. I am currently using the following: zgrep -R -L... (1 Reply)
Discussion started by: cbreiner
1 Replies

10. Shell Programming and Scripting

Check if file compressed or not

Is there a way I can check if a file is comppressed or not? (Be it tar/gzip or compress). trying to write a generic housekeeping scrit that will delete files over 6 months old and compress any uncompressed files if less than 6 months old. But not sure if there is a clever way to check except for... (4 Replies)
Discussion started by: badg3r
4 Replies
Login or Register to Ask a Question