How to read/process a .gz file, one line at a time?

dayscripter · #1 (**permalink**) 08-06-2008

Hello

I'm stuck trying to solve this KSH issue and I'm hoping someone out there can offer some suggestions.

I want to read lots of large .gz files one line at a time in order to compare its Error entries with a list of known errors. I can't simply do "foreach ERROR do gzcat *.gz |grep ${ERROR}" because I would have to parse every single large log file for each of the known errors and the time to do that would be days. So I want to parse the log files only once, read each line, compare the error, and increment the error count for each error. I know how to do most of this except the part about processing one line at a time from a .gz file.

I came up with something like this but it's not working. It's outputing all the lines in the file, instead:

Code:

 
while read file_line
do
ERROR=`grep ^ERROR ${file_line}`
print ${ERROR}
done < `gzcat 080803.gz`

This is the output that's coming out. It should output only the lines beginning with ERROR but it's showing everything such as the AUDIT lines. I want just the ERROR lines. And there is no carriage return.

AUDIT ; WebContainer : 2008-08-04 00:11:51,554 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 337 ms.^JAUDIT ; WebContainer : 2008-08-04 00:11:58,885 ; com.at.commons:A_BeginRequest - Received request for transaction for uri '/docroot/common'.^JAUDIT ; WebContainer : 20-08-08-04 00:11:59,136 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 251ms.^JAUDIT ; WebContainer : 2008-08-04 00:12:08,686 ; com.at.commons:A_BeginRequest - Received request for transaction for uri '/docroot/common'.^JAUDIT ; WebContainer : 2008-08-04 00:12:09,078 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 392 ms.: cannot open

I wanted the output to look like this instead:

ERROR ; WebContainer : 2008-08-03 04:33:45,787 ; com.models.userlist.query:E_AggregationError
ERROR ; WebContainer : 2008-08-03 04:33:59,930 ; com.models.userlist.query:E_AggregationError
ERROR ; WebContainer : 2008-08-03 04:34:31,751 ; com.app.cdmeng.combination:E_marshalException

What am I doing wrong? Any assistance would be much appreciated.

Thanks.

cfajohnson · #2 (**permalink**) 08-06-2008

Code:

zgrep ^ERROR 080803.gz

Or:

Code:

zcat 080803.gz | grep ^ERROR

vkesineni · #3 (**permalink**) 10-16-2008

gzcat test1.gz|awk '/ERROR/{print $0}'

You can get with AWK like above

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
read the file line by line	kittusri9	Shell Programming and Scripting	3	04-24-2008 09:26 AM
How to read max of 10 file at a time?	cedrichiu	Shell Programming and Scripting	10	06-23-2007 08:52 AM
why shell scripting takes more time to read a file	brkavi_in	Shell Programming and Scripting	1	06-23-2006 09:20 AM
How to read from a file line by line and do stuff	spaceship	Shell Programming and Scripting	4	03-17-2005 09:47 PM
How to read and write files one line at a time.	s_chopra	UNIX for Dummies Questions & Answers	2	04-18-2001 10:39 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by