How to read/process a .gz file, one line at a time?
Hello
I'm stuck trying to solve this KSH issue and I'm hoping someone out there can offer some suggestions.
I want to read lots of large .gz files one line at a time in order to compare its Error entries with a list of known errors. I can't simply do "foreach ERROR do gzcat *.gz |grep ${ERROR}" because I would have to parse every single large log file for each of the known errors and the time to do that would be days. So I want to parse the log files only once, read each line, compare the error, and increment the error count for each error. I know how to do most of this except the part about processing one line at a time from a .gz file.
I came up with something like this but it's not working. It's outputing all the lines in the file, instead:
Code:
while read file_line
do
ERROR=`grep ^ERROR ${file_line}`
print ${ERROR}
done < `gzcat 080803.gz`
This is the output that's coming out. It should output only the lines beginning with ERROR but it's showing everything such as the AUDIT lines. I want just the ERROR lines. And there is no carriage return.
AUDIT ; WebContainer : 2008-08-04 00:11:51,554 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 337 ms.^JAUDIT ; WebContainer : 2008-08-04 00:11:58,885 ; com.at.commons:A_BeginRequest - Received request for transaction for uri '/docroot/common'.^JAUDIT ; WebContainer : 20-08-08-04 00:11:59,136 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 251ms.^JAUDIT ; WebContainer : 2008-08-04 00:12:08,686 ; com.at.commons:A_BeginRequest - Received request for transaction for uri '/docroot/common'.^JAUDIT ; WebContainer : 2008-08-04 00:12:09,078 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 392 ms.: cannot open
HI All,
I am aware of reading one file line by line but now my requirement is to read 2 file line by line as below explained.
I have 2 files as below file 1 with spaces and file 2 same as file1 but without space and () brackets
now i have to read both file line by line and use first line of... (13 Replies)
Hi all,
I have a log file say Test.log that gets updated continuously and it has data in pipe separated format. A sample log file would look like:
<date1>|<data1>|<url1>|<result1>
<date2>|<data2>|<url2>|<result2>
<date3>|<data3>|<url3>|<result3>
<date4>|<data4>|<url4>|<result4>
What I... (3 Replies)
Hi All,
Am trying to write wrapper shell/bash script on a utility tool for which i need to pass 2 files as arugment to execute utility tool.
Wraper script am trying is to do with above metion 2 files.
utility tool accepts :
a. userinfo file : which contains username
b. item file : which... (2 Replies)
Hello,
I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be:
SRVXPAPI001 ERRO JUN24 07:28:34 1775
REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F
... (8 Replies)
Hi ,
I am trying to write an shell, which reads a text file (from a location) having a list of numbers of strictly 5 digits only ex: 33144
Now my script will check :
1) that each entry is only 5 digits & numeric only, no alphabets, & its not empty.
2)then it executes a shell script called... (8 Replies)
Hi,
i have file which contains data as below(Only sample shown, it may contain more data similar to the one shown here)
i need to read this file line by line and generate an output file like the one below
i.e based on N value the number of MSISDNs will vary, if N=1 then the following... (14 Replies)
I am using the while-loop to read a file.
The file has lines with null-terminated strings (words, actually.)
What I have by that reading - just a first word up to '\0'!
I need to have whole string up to 'new line' - (LF, 10#10, 16#A)
What I am doing wrong?
#make file 'grb' with... (6 Replies)
Hi,
I have gps receiver log..its giving readings .like below
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GPSD,R=1
$GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)
Guys,
I've written the following to proces one line at a time and then process individual column within the line.. But it works only when there is one line. I know awk will simply fetch multiple lines when run against a file with many lines. How do i achieve this ?
cat ${PARFILE} |... (3 Replies)
Hi!
All!
I am wirting a shell script in which i want to read one line
at a time from the file and write it simultaneouly to other file one line at a time.
Please let me know about some shell utility which can help me out.
Thanx.
If further clarifications are needed then please let me know... (2 Replies)
