Shell Programming and Scripting

Hi Guys/Gals,

I have a log file that is updated once every few seconds and I am looking for a way to speed up one of my scripts.

Basically what I am trying to do is grep through a text file from start to finish once. Then each subsequent grep starts at the last line of the previous grep to the end of the file.

The log file can get as large as 200MB and rather then going through the file start to finish every time, I want to be able to have the second grep start off where the last grep finished, cutting down on processing time.

I tried to use tail to grab the last x amount of lines of the file which is much faster, but the issue is the file keeps being updated constantly so the last 100 lines is only valid for about 3 seconds, until a new line gets appended to the log file.

Is there a command to tail line 100101 to end_of_file?

I figured I could determine the last line number by using something like this
cat -n logfile.txt | tail -n1 | gawk "{print $1}" > lastline.txt

If anyone has a suggestion on how to accomplish this, please share your thoughts.

Thanks

Jerrad

Wow. 200MB?? That's a huge text file. Personally, I would first rotate that log to a more manageable size first.

... you could use the tail command with the + operand..
...so you'd do something like this ( not complete code, but you'll get the idea):

Code:

 # cat logfile | wc -l  >> marker.txt  ## where marker has the number of lines in
                                     ## your logilfe at the end of  your script

... then. next time you run your script:

Code:

# COUNT=$(cat marker)
let MARKER=${COUNT}+1
#  tail +$(MARKER} logfile

...and you'll only get the lines in the logfile that were appended after the last time you checked.

I'd still strongly suggest you rotate that log file more often first. It's easier and faster to work with closed, smaller sized files.

... quick example of the difference between tail -n and tail +n :
..say you have a file:

Code:

# cat file
111111111111
2222222222222
33333333333333
44444444444444
555555555555555
666666666666666
777777777777777
88888888888888
999999999999999
aaaaaaaaaaaaa
bbbbbbbbbbbbb
ccccccccccccc

#tail -5 file 
88888888888888
999999999999999
aaaaaaaaaaaaa
bbbbbbbbbbbbb
ccccccccccccc

# tail +5 file
555555555555555
666666666666666
777777777777777
88888888888888
999999999999999
aaaaaaaaaaaaa
bbbbbbbbbbbbb
ccccccccccccc

Code:

sed -n '100101,$p'

i want to print from xth line to second last line?
that is $p in sed will be short by 1.
so how is the syntax?

why cannot you use some thing like this.

tail -f name_of_your_file | grep 'text you want to search'