Visit Our UNIX and Linux User Community


Awk/sed : help on:Filtering multiple lines to one:


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk/sed : help on:Filtering multiple lines to one:
# 1  
Old 05-31-2013
Awk/sed : help on:Filtering multiple lines to one:

Experts Good day,

I want to filter multiple lines of same error of same day , to only 1 error of each day, the first line from the log.

Here is the file:


Code:
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:31 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:34 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 10:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 10:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 10:01:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 10:02:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 10:03:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)



The output should be:
Code:
May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 11:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 10:00:02 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)

may be this is quite simple, but not able to figure out:
Thanks ,
# 2  
Old 05-31-2013
Code:
awk '{E=$0;gsub(/.*..:..:.. /,X,E)}!A[$1,$2,E]++' file

This User Gave Thanks to Yoda For This Post:
# 3  
Old 05-31-2013
Yoda Thanks,
Correction. I saw the lines were different so it is working as per the logic I had given.

---------- Post updated at 12:46 PM ---------- Previous update was at 12:39 PM ----------

In an another set of data : I found a different out put so I thought initially that it is not working but realized that the lines are not identical in the data , so reporting multiple times:

Code:
May 26 11:29:33 cmihpx02 vmunix: NFS getattr failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:33 cmihpx02 vmunix: NFS getattr failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 26 11:29:35 cmihpx02 vmunix: NFS getattr failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:34:30 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:34:30 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:52 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:52 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:34:31 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)

Code:
awk '{E=$0;gsub(/.*..:..:.. /,X,E)}!A[$1,$2,E]++' file2
May 26 11:29:33 cmihpx02 vmunix: NFS getattr failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 26 11:29:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 29 14:33:35 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 30 15:45:40 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.
May 31 09:33:43 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS write error on host cmiauxe1: error 238.


- I think I should be more specific about the string I am looking here. I am trying to filter the lines based on the string :
Code:
vmunix: NFS

Thanks a lot..

---------- Post updated at 12:50 PM ---------- Previous update was at 12:46 PM ----------

Hi Yoda, I made a slight change in the code: and it is working .


Code:
$ awk '{E=$5 " " $6;gsub(/.*..:..:.. /,X,E)}!A[$1,$2,E]++' file2
May 26 11:29:33 cmihpx02 vmunix: NFS getattr failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)

It is nice..
Thanks a lot,
This User Gave Thanks to rveri For This Post:
# 4  
Old 05-31-2013
Code:
$ sort -u -k2n -t' ' file
May 26 11:29:33 cmihpx02 vmunix: NFS getattr failed for server cmiauxe1: error 5 (RPC: Timed out)
May 29 14:33:35 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 30 15:45:40 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)
May 31 09:33:43 cmihpx02 vmunix: NFS commit failed for server cmiauxe1: error 5 (RPC: Timed out)

This User Gave Thanks to anbu23 For This Post:
# 5  
Old 05-31-2013
anbu23 , Thanks this too works, However noticed if -t' ' is really needed since the data already contains space separators.

Code:
sort -u -k2n -t' '

Thanks,

Previous Thread | Next Thread
Test Your Knowledge in Computers #22
Difficulty: Easy
RAM is considered volatile memory in a computer system.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to find=grep or maybe sed/awk for multiple lines of text?

Hi, I am running the following: PASS="username/password" sqlplus -s << EOF | grep -v "^$" $PASS set feedback off set heading off set termout off select name from v\$database ; exit EOF Which gives ERROR: ORA-28002: the password will expire within 5 days PSMP1 (1 Reply)
Discussion started by: newbie_01
1 Replies

2. Shell Programming and Scripting

File filtering using awk or sed

Hello Members, I have a file, having below contents: <KEYVALUE>$4,0,1,4,966505098999--&gt;RemoteSPC: 13 SSN: 146</KEYVALUE> <KEYVALUE>$4,123,1,4,966505050198--&gt;RemoteSPC: 1002 SSN: 222,Sec:RemoteSPC: 1004 SSN: 222</KEYVALUE> <KEYVALUE>$4,123,1,4,966505050598--&gt;RemoteSPC: 1002 SSN:... (9 Replies)
Discussion started by: umarsatti
9 Replies

3. Shell Programming and Scripting

Sed/awk/perl substitution with multiple lines

OSX I have been grinding my teeth on a portion of code. I am building a bash script that edits a html email template. In the template, I have place holders for SED (or whatever program is appropriate) to use as anchors for find and replace, with user defined corresponding html code. The HTML code... (3 Replies)
Discussion started by: sudo
3 Replies

4. Shell Programming and Scripting

Sed/awk/perl command to replace pattern in multiple lines

Hi I know sed and awk has options to give range of line numbers, but I need to replace pattern in specific lines Something like sed -e '1s,14s,26s/pattern/new pattern/' file name Can somebody help me in this.... I am fine with see/awk/perl Thank you in advance (9 Replies)
Discussion started by: dani777
9 Replies

5. Shell Programming and Scripting

sed filtering lines by range fails 1-line-ranges

The following is part of a larger project and sed is (right now) a given. I am working on a recursive Korn shell function to "peel off" XML tags from a larger text. Just for context i will show the complete function (not working right now) here: function pGetXML { typeset chTag="$1" typeset... (5 Replies)
Discussion started by: bakunin
5 Replies

6. Shell Programming and Scripting

Print lines between two strings multiple occurencies (with sed, awk, or grep)

Hello, I can extract lines in a file, between two strings but only one time. If there are multiple occurencies, my command show only one block. Example, monfichier.txt contains : debut_sect texte L1 texte L2 texte L3 texte L4 fin_sect donnees inutiles 1 donnees inutiles 2 ... (8 Replies)
Discussion started by: theclem35
8 Replies

7. Shell Programming and Scripting

Sed or Awk for lines between two strings multiple times and keep the last one

Hi, I am trying to get lines between the last occurrences of two patterns. I have files that have several occurrences of “Standard” and “Visual”. I will like to get the lines between “Standard” and “Visual” but I only want to retain only the last one e.g. Standard Some words Some words Some... (4 Replies)
Discussion started by: damanidada
4 Replies

8. Shell Programming and Scripting

Filtering Issues Using sed and awk

Hi, I am currently using the sed and awk commands to filter a file that has multiple sets of data in different columns. An example of part of the file i am filtering is as follows; Sat Oct 2 07:42:45 2010 01:33:46 R1_CAR_12.34 Sun Oct 3 13:09:53 2010 00:02:34 R2_BUS_56.78 Sun... (4 Replies)
Discussion started by: crunchie
4 Replies

9. Shell Programming and Scripting

sed/awk to insert multiple lines before pattern

I'm attempting to insert multiple lines before a line matching a given search pattern. These lines are generated in a separate function and can either be piped in as stdout or read from a temporary file. I've been able to insert the lines from a file after the pattern using: sed -i '/pattern/... (2 Replies)
Discussion started by: zksailor534
2 Replies

10. Shell Programming and Scripting

awk and sed filtering

Goo afternoon Sir'sould like to ask your help reagrding in this scenario using sed and awk. ******************************************************** Host:CDRMSAPPS1 Operating System:Linux 2.6.9-42.ELsmp Machine Type:UNIX Host Type:Client Version:5.1... (2 Replies)
Discussion started by: invinzin21
2 Replies

Featured Tech Videos