Compare last 90 logs and print repeating lines with >20


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare last 90 logs and print repeating lines with >20
# 1  
Old 03-14-2012
Compare last 90 logs and print repeating lines with >20

*log files are in date order

sample logs...

Code:
ciscoresets_20120314
ciscoresets_20120313
ciscoresets_20120312
ciscoresets_20120311
ciscoresets_20120310

Code:
cat ciscoresets_20120314
SYDGRE04,10,9
SYDGRE04,10,10
SYDGRE04,10,11
SYDGRE04,10,12
SYDGRE04,10,13
SYDGRE04,10,14
SYDGRE04,10,15
NEPGRE01,10,1
NEPGRE01,10,2
NEPGRE01,10,3
NEPGRE01,10,4
NEPGRE01,10,5
NEPGRE01,10,6
NEPGRE01,10,7
NEPGRE01,10,8
NEPGRE01,10,9
NEPGRE01,10,10
NEPGRE01,10,11
NEPGRE01,10,12
NEPGRE01,10,13
NEPGRE01,10,14
NEPGRE01,10,15
NEPGRE01,10,16


cat ciscoresets_20120313
SYDGRE04,10,1
SYDGRE04,10,2
SYDGRE04,10,3
SYDGRE04,10,4
SYDGRE04,10,5
SYDGRE04,10,6
SYDGRE04,10,7
SYDGRE04,10,8
SYDGRE04,10,9
SYDGRE04,10,10
SYDGRE04,10,11
SYDGRE04,10,12
SYDGRE04,10,13
SYDGRE04,10,14
SYDGRE04,10,15
NEPGRE01,10,4
NEPGRE01,10,5
NEPGRE01,10,6
NEPGRE01,10,7
NEPGRE01,10,8
NEPGRE01,10,9

# 2  
Old 03-14-2012
Try this (not tested)
Code:
cat `ls -1 ciscoresets_* | sort -r | head -90` | sort | uniq -c | awk '$1>20 {print $2}'

# 3  
Old 03-15-2012
Hi, try this:

Code:
awk '
  function fdate( fd ) {
    sub(/^[^0-9]*/, nothing, fd ); return fd
  }
  function pattern( fd ) {
    sub(/[0-9]*$/, "*" , fd ); return fd
  }
  {
    MATCH[$1]=0
  }
  END {
    end=fdate(FILENAME)
    while ( "printf \"%s\n\" " pattern(FILENAME) | getline file ) {
      if ( fdate(file) >= start && fdate(file) < end ) list=list FS file
    }
    while( "cat "list | getline record ) {
      if ( record in MATCH ) MATCH[record]++
    }
    for (record in MATCH){
      if ( ! MATCH[record]  ) print "NEW:" record
      if ( MATCH[record] > treshold ) print record, MATCH[record]
    }
  }
' treshold=20 start=20111214 ciscoresets_20120314 | sort -k2


Last edited by Scrutinizer; 03-15-2012 at 06:54 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count and print the most repeating string in each line

Hi all, I have a file in which each string from column 1 is associated with one or multiple strings from column 2. For an example, in the sample input below, Gene1 from column1 is associated with two different strings from column 2 (BP1 and BP2).For every unique string from column 1, I need to... (9 Replies)
Discussion started by: AshwaniSharma09
9 Replies

2. Shell Programming and Scripting

Compare file1 for matching line in file2 and print the difference in matching lines

Hello, I have two files file 1 and file 2 each having result of a query on certain database tables and need to compare for Col1 in file1 with Col3 in file2, compare Col2 with Col4 and output the value of Col1 from File1 which is a) not present in Col3 of File2 b) value of Col2 is different from... (2 Replies)
Discussion started by: RasB15
2 Replies

3. Shell Programming and Scripting

compare & print lines with 2 requirements

1.compare 90 logs >20 print off with correct figure 2.compare latest log i.e ciscoresets_20120314 against all records not just the 90 (as above) and any lines not matching print as: NEW:SYDGRE04,10,9 1 (note this number could be 2 or whatever not necessary 1 it could be 2,3 or even 10... (2 Replies)
Discussion started by: slashbash
2 Replies

4. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

5. Shell Programming and Scripting

Compare Tab Separated Field with AWK to all and print lines of unique fields.

Hi. I have a tab separated file that has a couple nearly identical lines. When doing: sort file | uniq > file.new It passes through the nearly identical lines because, well, they still are unique. a) I want to look only at field x for uniqueness and if the content in field x is the... (1 Reply)
Discussion started by: rocket_dog
1 Replies

6. Shell Programming and Scripting

Count and print all repeating words in a line

Gurus, I have a file containing lines like this : Now, number of words in each line varies. My need is, if a word repeats in a line get it printed. Also total number of repeats. So, the output would be : Any help would be highly appreciated. Thanks & Regards (5 Replies)
Discussion started by: AshwaniSharma09
5 Replies

7. Shell Programming and Scripting

Compare two strings, and print lines containing mismatches

pls help me on this... and im really sorry because i really don't know where to start here... FILE1 ABC DEF 10 2 DEF GHI 11 3 GHI JKL 12 5 JKL MNO 13 7 MNO PQR 14 5 requirements: 1. The third string should only be 10 or 12 2. The fourth string should only be 2 or 3 3. Prinnt... (1 Reply)
Discussion started by: kingpeejay
1 Replies

8. Shell Programming and Scripting

Compare two files and print the two lines with difference

I have two files like this: #FILE 1 ABCD 4322 26485 JMTJ 5311 97248 XMPJ 4321 58978 #FILE 2 ABCD 4321 26485 JMTJ 5311 97248 XMPJ 4321 68978 What to do: Compare the two files and find those lines that doesn't match. And have a new file like this: #FILE 3 "from file 1" ABCD 4322 26485... (11 Replies)
Discussion started by: kingpeejay
11 Replies

9. Shell Programming and Scripting

awk to compare lines of two files and print output on screen

hey guys, I have two files both with two columns, I have already created an awk code to ignore certain lines (e.g lines that start with 963) as they wou ld begin with a certain string, however, the rest I have added together and calculated the average. At the moment the code also displays... (3 Replies)
Discussion started by: chlfc
3 Replies

10. UNIX for Dummies Questions & Answers

Omit repeating lines

Can someone help me with the following 2 objectives? 1) The following command is just an example. It gets a list of all print jobs. From there I am trying to extract the printer name. It works with the following command: lpstat -W "completed" -o | awk -F- '{ print $1}' Problem is, I want... (6 Replies)
Discussion started by: TheCrunge
6 Replies
Login or Register to Ask a Question