Sponsored Content
Top Forums Shell Programming and Scripting Select matches between line number and end of file? Post 302074518 by Jerrad on Tuesday 23rd of May 2006 06:41:51 PM
Old 05-23-2006
Select matches between line number and end of file?

Hi Guys/Gals,

I have a log file that is updated once every few seconds and I am looking for a way to speed up one of my scripts.

Basically what I am trying to do is grep through a text file from start to finish once. Then each subsequent grep starts at the last line of the previous grep to the end of the file.

The log file can get as large as 200MB and rather then going through the file start to finish every time, I want to be able to have the second grep start off where the last grep finished, cutting down on processing time.

I tried to use tail to grab the last x amount of lines of the file which is much faster, but the issue is the file keeps being updated constantly so the last 100 lines is only valid for about 3 seconds, until a new line gets appended to the log file.

Is there a command to tail line 100101 to end_of_file?

I figured I could determine the last line number by using something like this
cat -n logfile.txt | tail -n1 | gawk "{print $1}" > lastline.txt

If anyone has a suggestion on how to accomplish this, please share your thoughts.

Thanks

Jerrad
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to select a column from particular line number

The awk command awk -F: '{print $1}' test1 gives the first columns of all the lines in file ,is there some command to get a particular column from particular line . Any help is appreciated. thanks arif (4 Replies)
Discussion started by: mab_arif16
4 Replies

2. Shell Programming and Scripting

Number a list at end of line using 'sed'

Hi All I have a script which has produced a list, I have used 'sed' to number my list, but i want to list at end of line with the first line starting at zero (0) and brackets round it ie My List i want Hello (0) this (1) day (2) can (3) be (4) sed '/./=' filename | sed '/./N; s/\n/) /'... (5 Replies)
Discussion started by: chassis
5 Replies

3. Shell Programming and Scripting

How to Select or cut from the certain filed to the end of the line

Hi: I have few rows in file..Like suppose... 9063C0 44 00051363603253033253347 3333 070248 06 9063C0 5G PAN00013 9063C0 44 00061030305040404250724 0506 100248 08 9063C0 43 01 00000089 I need to cut the row starting after... (5 Replies)
Discussion started by: grajesh_955
5 Replies

4. Shell Programming and Scripting

Capturing a number at the end of line and store it as variable

Hello, Would someone guide me on how to write a shell script the would search for a phone no using at the end text file using sed or awk and store it in a varaible or print it. The text file is in this form text or numbers in first line text or numbers in second line . . . Firsname... (6 Replies)
Discussion started by: amuthiga
6 Replies

5. Shell Programming and Scripting

To add a number at the end of the line

Hi Folks, Using the Vi, how can I add a numbers at the end of the line. For eg: I have the numbers in the file as: 58.125.33 22.58.68 25.144.225 114.25.38 I need to add .0/8 at the end of all the line. So, it should be like 58.125.33.0/8 22.58.68.0/8 25.144.225.0/8 114.25.38.0/8 (6 Replies)
Discussion started by: gsiva
6 Replies

6. Shell Programming and Scripting

Get line number when matches a string

If I have a file something like as shown below, ARM*187878*hjhj BAG*88778*jjjj COD*7777*kkkk BAG*87878*kjjhjk DEF*65656*89989*khjkk I need the line numbers to be added with a colon when it matches the string "BAG". Here in my case, I need something like ARM*187878*hjhj... (4 Replies)
Discussion started by: Muthuraj K
4 Replies

7. Shell Programming and Scripting

AWK-grep from line number to the end of file

Does anyone know how to use awk to act like grep from a particular line number to the end of file? I am using Solaris 10 and I don't have any GNU products installed. Say I want to print all occurrences of red starting at line 3 to the end of file. EXAMPLE FILE: red green red red... (1 Reply)
Discussion started by: thibodc
1 Replies

8. Shell Programming and Scripting

adding line number to *end* of records in file

Given a file like this: abc def ghi I need to get to somestandardtext abc1 morestandardtext somestandardtext def2 morestandardtext somestandardtext ghi3 morestandardtext Notice that in addition to the standard text there is the line number added in as well. What I conceived is... (4 Replies)
Discussion started by: edstevens
4 Replies

9. Shell Programming and Scripting

Count number of pattern matches per line for all files in directory

I have a directory of files, each with a variable (though small) number of lines. I would like to go through each line in each file, and print the: -file name -line number -number of matches to the pattern /comp/ for each line. Two example files: cat... (4 Replies)
Discussion started by: pathunkathunk
4 Replies

10. Shell Programming and Scripting

Print line if values in fields matches number and text

datafile: 2017-03-24 10:26:22.098566|5|'No Route for Sndr:RETEK RMS 00040 /ZZ Appl:PF Func:PD Txn:832 Group Cntr:None ISA CntlNr:None Ver:003050 '|'2'|'PFI'|'-'|'EAI_ED_DeleteAll'|'EAI_ED'|NULL|NULL|NULL|139050594|ActivityLog| 2017-03-27 02:50:02.028706|5|'No Route for... (7 Replies)
Discussion started by: SkySmart
7 Replies
LOGTOP(1)						      General Commands Manual							 LOGTOP(1)

NAME
logtop - Realtime log line rate analyser SYNOPSIS
logtop [OPTIONS] DESCRIPTION
logtop is a System Administrator tool analyzing line rate on stdin. It reads on stdin and print a constantly updated result displaying, in columns: Line number, count, frequency, and the actual line. $ tail -f FILE | logtop is the friendly version of: $ watch 'tail FILE | sort | uniq -c | sort -gr' OPTIONS
-s, --size=K Only keep K lines in memory, instead of 10000. -q, --quiet Do not display a live view of the data, only display a top at exit. -l, --line-by-line=K Print result line by line, in a machine friendly format, K is the number of result to print per line. Line by line format is : [%d %f %s ]* %d : Number of occurences %f : Frequency of apparition %s : String (Control chars replaced by dots. -i, --interval=K Interval between graphical updates, in seconds. Defaults to 1. -h, --help Show summary of options. -v, --version Show version of program. EXAMPLES
Here are some logtop usage examples. tail -f cache.log | grep -o "HIT|MISS" | logtop Realtime hit / miss ratio on some caching software log file. tail -f access.log | cut -d' ' -f1 | logtop -s 10000 Realtime most querying IPs on your server, as long as log lines in access.log starts with the client IP. tail -f access.log | cut -d' ' -f7 | logtop -s 10000 Realtime most requested web pages in a NCSA like log file. cat auth.log | grep -v "CRON" | grep -o ": .*" | logtop -q -s 100000 Display a one-shot simple analyse of your auth.log. SEE ALSO
watch(1) AUTHOR
logtop was written by Julien Palard. This manual page was written by Julien Palard <julien@palard.fr>, for the Debian project (and may be used by others). April 16, 2011 LOGTOP(1)
All times are GMT -4. The time now is 06:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy