How to read/process a .gz file, one line at a time?
Hello
I'm stuck trying to solve this KSH issue and I'm hoping someone out there can offer some suggestions.
I want to read lots of large .gz files one line at a time in order to compare its Error entries with a list of known errors. I can't simply do "foreach ERROR do gzcat *.gz |grep ${ERROR}" because I would have to parse every single large log file for each of the known errors and the time to do that would be days. So I want to parse the log files only once, read each line, compare the error, and increment the error count for each error. I know how to do most of this except the part about processing one line at a time from a .gz file.
I came up with something like this but it's not working. It's outputing all the lines in the file, instead:
This is the output that's coming out. It should output only the lines beginning with ERROR but it's showing everything such as the AUDIT lines. I want just the ERROR lines. And there is no carriage return.
AUDIT ; WebContainer : 2008-08-04 00:11:51,554 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 337 ms.^JAUDIT ; WebContainer : 2008-08-04 00:11:58,885 ; com.at.commons:A_BeginRequest - Received request for transaction for uri '/docroot/common'.^JAUDIT ; WebContainer : 20-08-08-04 00:11:59,136 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 251ms.^JAUDIT ; WebContainer : 2008-08-04 00:12:08,686 ; com.at.commons:A_BeginRequest - Received request for transaction for uri '/docroot/common'.^JAUDIT ; WebContainer : 2008-08-04 00:12:09,078 ; com.at.commons:A_EndRequest - Done preparing response for transaction for uri '/docroot/common' in 392 ms.: cannot open
Hi!
All!
I am wirting a shell script in which i want to read one line
at a time from the file and write it simultaneouly to other file one line at a time.
Please let me know about some shell utility which can help me out.
Thanx.
If further clarifications are needed then please let me know... (2 Replies)
Guys,
I've written the following to proces one line at a time and then process individual column within the line.. But it works only when there is one line. I know awk will simply fetch multiple lines when run against a file with many lines. How do i achieve this ?
cat ${PARFILE} |... (3 Replies)
Hi,
I have gps receiver log..its giving readings .like below
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GPSD,R=1
$GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)
I am using the while-loop to read a file.
The file has lines with null-terminated strings (words, actually.)
What I have by that reading - just a first word up to '\0'!
I need to have whole string up to 'new line' - (LF, 10#10, 16#A)
What I am doing wrong?
#make file 'grb' with... (6 Replies)
Hi,
i have file which contains data as below(Only sample shown, it may contain more data similar to the one shown here)
i need to read this file line by line and generate an output file like the one below
i.e based on N value the number of MSISDNs will vary, if N=1 then the following... (14 Replies)
Hi ,
I am trying to write an shell, which reads a text file (from a location) having a list of numbers of strictly 5 digits only ex: 33144
Now my script will check :
1) that each entry is only 5 digits & numeric only, no alphabets, & its not empty.
2)then it executes a shell script called... (8 Replies)
Hello,
I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be:
SRVXPAPI001 ERRO JUN24 07:28:34 1775
REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F
... (8 Replies)
Hi All,
Am trying to write wrapper shell/bash script on a utility tool for which i need to pass 2 files as arugment to execute utility tool.
Wraper script am trying is to do with above metion 2 files.
utility tool accepts :
a. userinfo file : which contains username
b. item file : which... (2 Replies)
Hi all,
I have a log file say Test.log that gets updated continuously and it has data in pipe separated format. A sample log file would look like:
<date1>|<data1>|<url1>|<result1>
<date2>|<data2>|<url2>|<result2>
<date3>|<data3>|<url3>|<result3>
<date4>|<data4>|<url4>|<result4>
What I... (3 Replies)
HI All,
I am aware of reading one file line by line but now my requirement is to read 2 file line by line as below explained.
I have 2 files as below file 1 with spaces and file 2 same as file1 but without space and () brackets
now i have to read both file line by line and use first line of... (13 Replies)
Discussion started by: abhaydas
13 Replies
LEARN ABOUT OPENSOLARIS
fmlcut
fmlcut(1F) FMLI Commands fmlcut(1F)NAME
fmlcut - cut out selected fields of each line of a file
SYNOPSIS
fmlcut -clist [filename]...
fmlcut -flist [-dchar] [-s] [filename]...
DESCRIPTION
The fmlcut function cuts out columns from a table or fields from each line in filename; in database parlance, it implements the projection
of a relation. fmlcut can be used as a filter; if filename is not specified or is -, the standard input is read. list specifies the fields
to be selected. Fields can be fixed length (character positions) or variable length (separated by a field delimiter character), depending
on whether -c or -f is specified.
Note: Either the -c or the -f option must be specified.
OPTIONS
list A comma-separated list of integer field numbers (in increasing order), with optional - to indicate ranges. For example: 1,4,7;
1-3,8; -5,10 (short for 1-5,10); or 3- (short for third through last field).
-clist If -c is specified, list specifies character positions (for instance, -c1-72 would pass the first 72 characters of each line).
Note: No space intervenes between -c and list.
-flist If -f is specified, list is a list of fields assumed to be separated in the file by the default delimiter character, TAB, or by
char if the -d option is specified. For example, -f1,7 copies the first and seventh field only. Lines with no delimiter charac-
ters are passed through intact (useful for table subheadings), unless -s is specified. Note: No space intervenes between -f and
list. The following options can be used if you have specified -f.
-dchar If -d is specified, char is the field delimiter. Space or other characters with special meaning to FMLI must be quoted.
Note: No space intervenes between -d and char . The default field delimiter is TAB.
-s Suppresses lines with no delimiter characters. If -s is not specified, lines with no delimiters will be passed through
untouched.
EXAMPLES
Example 1 Getting login IDs and names
The following example gets the login IDs and names.
example% fmlcut -d: -f1,5 /etc/passwd
Example 2 Getting the current login name
The next example gets the current login name.
example% `who am i | fmlcut -f1 -d" "`
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
SEE ALSO fmlgrep(1F), attributes(5)DIAGNOSTICS
fmlcut returns the following exit values:
0 when the selected field is successfully cut out
2 on syntax errors
The following error messages may be displayed on the FMLI message line:
ERROR: line too long
A line has more than 1023 characters or fields, or there is no new-line character.
ERROR: bad list for c/f option
Missing -c or -f option or incorrectly specified list. No error occurs if a line has fewer fields than the list calls for.
ERROR: no fields
The list is empty.
ERROR: no delimiter
Missing char on -d option.
NOTES
fmlcut cannot correctly process lines longer than 1023 characters, or lines with no newline character.
SunOS 5.11 5 Jul 1990 fmlcut(1F)