Sponsored Content
Full Discussion: Incremental log parser
Top Forums Shell Programming and Scripting Incremental log parser Post 302289983 by Narcom on Saturday 21st of February 2009 04:11:06 AM
Old 02-21-2009
Incremental log parser

I have log file that is incremented every second and is rather big. I need monitor error in this file. I do not want to parse it all every time.

I want parse it first time then remember last position (as byte for example)
All consequent parsing I want to start from last saved position to the end of file.

Up to now I use this method

COUNT_OLD=`awk '$1>=0 {print $1}' file.seq`
COUNT_NEW=`cat file | wc –l` #get file lines count
cat file | grep -nf file.search | tr ':' ' ' | awk '$1 > '$COUNT_OLD' && $1 < ‘$COUNT_NEW’{print "FOUND: "$0}' > errors
echo $COUNT_NEW > file.seq

I.e. using “grep -n” feature manually select only new lines.So it does not really “incremental”.
Other approach is to use for example “head” or “tail -n” command to copy new line some where and parse only that file but I think it is inefficient…

I want to read directly from middle of the file. For example if file 1000 byte. I want to read directly from 100 byte to 999 byte.

Is it possible by standard shell commands? Or I need use some of programming languages?
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

incremental backup

Hi All.. i am trying to write a script which will give the incremental tar backup of all files with latest timestam. i tried with find -mmin -2 but if it takes half on hour or something to creat the tar itself, then no meaning in using the above command. so please help me to find the... (2 Replies)
Discussion started by: Usha Shastri
2 Replies

2. Shell Programming and Scripting

grep'ing and sed'ing chunks in bash... need help on speeding up a log parser.

I have a file that is 20 - 80+ MB in size that is a certain type of log file. It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example: The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies

3. Shell Programming and Scripting

Incremental backup

Hi, I would like to create a daily incremental backup of a directory with all of the files within and add a timestamp (year-month-day) to the tar.gz file. I have the following, but it doesn't backup the inside files of the directory. #!/bin/bash tar -czf... (1 Reply)
Discussion started by: agasamapetilon
1 Replies

4. UNIX for Dummies Questions & Answers

incremental by 1

let says, i have this number as 000002080, i want to add 1 to make it 000002081, and then i want to add 1 to 000002082, add 1 to 000002083, 84. i=000002080 TOT=$(echo "scale=9; $i + 1" | bc) echo $TOT it shows 2081, i want to retain 000002081, 000002082, 000002082, 000002084. (2 Replies)
Discussion started by: tjmannonline
2 Replies

5. Shell Programming and Scripting

FULL or INCREMENTAL ???

Hi. Can someone tell me if the following script that i have made is a script for INCREMENTAL BACKUP or FULL BACKUP. My teacher told me that is doing an FULL BACKUP. • find /etc /var /home -newer /backups/.backup_reference > /backups/.files_to_archive • touch /backups/.backup_reference • tar... (1 Reply)
Discussion started by: bender-alex
1 Replies

6. Shell Programming and Scripting

Incremental numbering?

Would it be possible for a script to duplicate a file and incrementally number it? File in: XXX_007_0580_xxxx_v0016.aep File out: XXX_007_0580_xxxx_v0017.aep If someone knows of a way I'd love to see it. Thanks! (7 Replies)
Discussion started by: scribling
7 Replies

7. UNIX for Advanced & Expert Users

Incremental extract from growing log file.

We have a log file which has 16 million row. We want to read all the lines appended from the last time we read using sed command sed -n '<START_LINE>,<LAST_LINE>p' abc.csv I can store this last line line so I can give replace that with START_LINE in my next read. The problem is wc -l which... (2 Replies)
Discussion started by: one2connect
2 Replies

8. UNIX for Beginners Questions & Answers

Incremental logic

Hi Guys, am trying to write logic to do incremental value using linux Example: a=00.00.00.01 My b should be like this b=00.00.00.02 and when it reaches 99 my b should look like this b=00.00.01.99 Appreciate your help guys Please use CODE tags when displaying sample input, sample... (14 Replies)
Discussion started by: buddi
14 Replies
cat(1)							      General Commands Manual							    cat(1)

Name
       cat - concatenate and print data

Syntax
       cat [ -b ] [ -e ] [ -n ] [ -s ] [ -t ] [ -u ] [ -v ] file...

Description
       The  command reads each file in sequence and displays it on the standard output.  Therefore, to display the file on the standard output you
       type:
       cat file
       To concatenate two files and place the result on the third you type:
       cat file1 file2 > file3
       To concatenate two files and append them to a third you type:
       cat file1 file2 >> file3
       If no input file is given, or if a minus sign (-) is encountered as an argument, reads from the standard input file.  Output is buffered in
       1024-byte blocks unless the standard output is a terminal, in which case it is line buffered.  The utility supports the processing of 8-bit
       characters.

Options
       -b   Ignores blank lines and precedes each output line with its line number.

       -e   Displays a dollar sign ($) at the end of each output line.

       -n   Precedes all output lines (including blank lines) with line numbers.

       -s   Squeezes adjacent blank lines from output and single spaces output.

       -t   Displays non-printing characters (including tabs) in output.  In addition to those representations used with the -v  option,  all  tab
	    characters are displayed as ^I.

       -u   Unbuffers output.

       -v   Displays  non-printing  characters (excluding tabs and newline) as the ^x.	If the character is in the range octal 0177 to octal 0241,
	    it is displayed as M-x. The delete character (octal 0177) displays as ^?.  For example, is displayed as ^X.

See Also
       cp(1), ex(1), more(1), pr(1), tail(1)

																	    cat(1)
All times are GMT -4. The time now is 11:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy