Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Extracting data from large logs.
Top Forums Shell Programming and Scripting Extracting data from large logs. Post 302126751 by Nick31 on Thursday 12th of July 2007 07:47:24 PM
Old 07-12-2007
Well, you could do it in perl by building an array of 11 strings, and push in your lines until your array is full. Then check the 8th string for your pattern. If it doesn't match, pop the first one out, push a new one in, check again. When it does match, print out your array. It's a brute force method, but it should work.

You could do this same concept in awk or sed using buffers.

You might also try grep -p if you can identify a way to divide your lines into paragraphs. For example, if every log entry starts with a line of "++++++" and the text you're searching for is the 8th line of the log entry, do:
grep -p"++++++" "searchstring" logfile

That would be the easiest method.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awk Help for extracting report from logs

Hi I have a log file, with lines of following kind. ------------------------ 2009-05-15 07:49:42,574 INFO - SqlMapObjectDataDao - select - selectObject - 2 ms 2009-05-15 07:49:42,575 INFO - SqlMapUserDao - select - getUserSystemAdminSuperGroup - 0 ms 2009-05-15 07:49:42,576 INFO -... (3 Replies)
Discussion started by: jitendriya.dash
3 Replies

2. UNIX for Dummies Questions & Answers

restrict data from getting written to Logs

$SYBASE/bin/isql -U $DB_USERID -S $DB_SERVER << ! >> $OUTFILE `echo $DB_PASSWD` use $db go Print " The processing" go ! # Extract data to file echo $DB_PASSWD | $SYBASE/bin/bcp $WRK_DB..open out $CONV_DIR/open".csv -t\, -c -U $DB_USERID -S $DB_SERVER -b 1000 | tail -3 I am able to... (0 Replies)
Discussion started by: w020637
0 Replies

3. Shell Programming and Scripting

Extracting a portion of data from a very large tab delimited text file

Hi All I wanted to know how to effectively delete some columns in a large tab delimited file. I have a file that contains 5 columns and almost 100,000 rows 3456 f g t t 3456 g h 456 f h 4567 f g h z 345 f g 567 h j k lThis is a very large data file and tab delimited. I need... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

4. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

5. UNIX for Dummies Questions & Answers

help in extracting logs in readable format

hello everyone. newbie here in unix. I am trying to extract the logs of a certain job and would like to output it in a readable format, see below the CAT part: cat /var/opt/ctma/ctm/sysout/idwesct_sh30_eng_r6_cdcs_sh.LOG_05l0du_000* | egrep -i 'orderid:|file_name=' | sed... (1 Reply)
Discussion started by: eanne_may
1 Replies

6. UNIX for Dummies Questions & Answers

Extracting a block of text from a large file using variables?

Hi UNIX Members, I've been tasked with performing the following: Extract a block of data in column form #This data changes each time, therefore automating future procedures Please Note the following: line = reading a line from a file_list that leads to the data The filename is called... (16 Replies)
Discussion started by: Klor
16 Replies

7. Shell Programming and Scripting

Extracting logs using gunzip awk and gzip

Hi All I am trying to use a hard coded script into shell scripting but I am unable to . Kindly find the Script below along with the command Please help gunzip -c FilePath/FileName_*.gz | awk '$0 > "" && $0 < ""'|\ gzip >> FilePath/Outputfile.log.gz I Am trying to use this... (9 Replies)
Discussion started by: pulkitbahl
9 Replies

8. UNIX for Beginners Questions & Answers

Transpose large data in UNIX

Hi I have the following sample of data: my full data dimention is 900,000* 1119 rs987435 C G 1 1 1 0 2 rs345783 C G 0 0 1 0 0 rs955894 G T 1 1 2 2 1 rs6088791 ... (7 Replies)
Discussion started by: marwah
7 Replies

9. Programming

C++ help in large data set

Hi All, We are trying to replace a 3rdparty where we don't know how they handled the reader part here. The query below is getting 197 * 2038017 row in the table. In the below code we are trying to run the query and execute in the DB part and fetch and read the record. That is where it is... (1 Reply)
Discussion started by: arunkumar_mca
1 Replies

LEARN ABOUT BSD

scanlogs

SCANLOGS(8)						      System Manager's Manual						       SCANLOGS(8)

NAME
       scanlogs - summarize INN log files.

SYNOPSIS
       scanlogs [ norotate ] [ nonn ]

DESCRIPTION
       Scanlogs  summarizes  the information recorded in the INN log files (see newslog(5)).  By default, it also rotates and cleans out the logs.
       It is normally invoked by the news.daily(8) script.

KEYWORDS
       The following keywords are accepted:

       norotate
	      Using this keyword disables the rotating and cleaning aspect of the log processing: the logs files are only scanned for  information
	      and no contents are altered.

       nonn   Normally the nn log file is scanned and rotated.	Using this keyword disables this function.

       If scanlogs is invoked more than once a day, the ``norotate'' keyword should be used to prevent premature log cleaning.

HISTORY
       Written by Landon Curt Noll <chongo@toad.com> and Rich $alz <rsalz@uunet.uu.net> for InterNetNews.  This is revision 1.1, dated 1996/10/29.

SEE ALSO
       innd(8) newslog(5), news.daily(8), nnrpd(8).

																       SCANLOGS(8)
All times are GMT -4. The time now is 07:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy