02-21-2009
Incremental log parser
I have log file that is incremented every second and is rather big. I need monitor error in this file. I do not want to parse it all every time.
I want parse it first time then remember last position (as byte for example)
All consequent parsing I want to start from last saved position to the end of file.
Up to now I use this method
COUNT_OLD=`awk '$1>=0 {print $1}' file.seq`
COUNT_NEW=`cat file | wc –l` #get file lines count
cat file | grep -nf file.search | tr ':' ' ' | awk '$1 > '$COUNT_OLD' && $1 < ‘$COUNT_NEW’{print "FOUND: "$0}' > errors
echo $COUNT_NEW > file.seq
I.e. using “grep -n” feature manually select only new lines.So it does not really “incremental”.
Other approach is to use for example “head” or “tail -n” command to copy new line some where and parse only that file but I think it is inefficient…
I want to read directly from middle of the file. For example if file 1000 byte. I want to read directly from 100 byte to 999 byte.
Is it possible by standard shell commands? Or I need use some of programming languages?
8 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi All..
i am trying to write a script which will give the incremental tar backup of all files with latest timestam.
i tried with find -mmin -2
but if it takes half on hour or something to creat the tar itself, then no meaning in using the above command.
so please help me to find the... (2 Replies)
Discussion started by: Usha Shastri
2 Replies
2. Shell Programming and Scripting
I have a file that is 20 - 80+ MB in size that is a certain type of log file.
It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example:
The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies
3. Shell Programming and Scripting
Hi,
I would like to create a daily incremental backup of a directory with all of the files within and add a timestamp (year-month-day) to the tar.gz file.
I have the following, but it doesn't backup the inside files of the directory.
#!/bin/bash
tar -czf... (1 Reply)
Discussion started by: agasamapetilon
1 Replies
4. UNIX for Dummies Questions & Answers
let says, i have this number as 000002080, i want to add 1 to make it 000002081, and then i want to add 1 to 000002082, add 1 to 000002083, 84.
i=000002080
TOT=$(echo "scale=9; $i + 1" | bc)
echo $TOT
it shows 2081, i want to retain 000002081, 000002082, 000002082, 000002084. (2 Replies)
Discussion started by: tjmannonline
2 Replies
5. Shell Programming and Scripting
Hi. Can someone tell me if the following script that i have made is a script for INCREMENTAL BACKUP or FULL BACKUP. My teacher told me that is doing an FULL BACKUP.
• find /etc /var /home -newer /backups/.backup_reference > /backups/.files_to_archive
• touch /backups/.backup_reference
• tar... (1 Reply)
Discussion started by: bender-alex
1 Replies
6. Shell Programming and Scripting
Would it be possible for a script to duplicate a file and incrementally number it?
File in: XXX_007_0580_xxxx_v0016.aep
File out: XXX_007_0580_xxxx_v0017.aep
If someone knows of a way I'd love to see it.
Thanks! (7 Replies)
Discussion started by: scribling
7 Replies
7. UNIX for Advanced & Expert Users
We have a log file which has 16 million row. We want to read all the lines appended from the last time we read using sed command
sed -n '<START_LINE>,<LAST_LINE>p' abc.csv
I can store this last line line so I can give replace that with START_LINE in my next read. The problem is wc -l which... (2 Replies)
Discussion started by: one2connect
2 Replies
8. UNIX for Beginners Questions & Answers
Hi Guys,
am trying to write logic to do incremental value using linux
Example:
a=00.00.00.01
My b should be like this
b=00.00.00.02
and when it reaches 99 my b should look like this
b=00.00.01.99
Appreciate your help guys
Please use CODE tags when displaying sample input, sample... (14 Replies)
Discussion started by: buddi
14 Replies
cat(1) General Commands Manual cat(1)
Name
cat - concatenate and print data
Syntax
cat [ -b ] [ -e ] [ -n ] [ -s ] [ -t ] [ -u ] [ -v ] file...
Description
The command reads each file in sequence and displays it on the standard output. Therefore, to display the file on the standard output you
type:
cat file
To concatenate two files and place the result on the third you type:
cat file1 file2 > file3
To concatenate two files and append them to a third you type:
cat file1 file2 >> file3
If no input file is given, or if a minus sign (-) is encountered as an argument, reads from the standard input file. Output is buffered in
1024-byte blocks unless the standard output is a terminal, in which case it is line buffered. The utility supports the processing of 8-bit
characters.
Options
-b Ignores blank lines and precedes each output line with its line number.
-e Displays a dollar sign ($) at the end of each output line.
-n Precedes all output lines (including blank lines) with line numbers.
-s Squeezes adjacent blank lines from output and single spaces output.
-t Displays non-printing characters (including tabs) in output. In addition to those representations used with the -v option, all tab
characters are displayed as ^I.
-u Unbuffers output.
-v Displays non-printing characters (excluding tabs and newline) as the ^x. If the character is in the range octal 0177 to octal 0241,
it is displayed as M-x. The delete character (octal 0177) displays as ^?. For example, is displayed as ^X.
See Also
cp(1), ex(1), more(1), pr(1), tail(1)
cat(1)