Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sort & uniq on specific fields problem

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sort & uniq on specific fields problem
# 1  
Old 01-31-2012
sort & uniq on specific fields problem
Hello;

I have the output data set from:

Code:
egrep -i 'warning| error| fail' /var/adm/syslog/syslog.log

Code:
Jan 31 12:02:18 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Jan 31 12:02:23 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 12:02:23 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Jan 31 12:02:28 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 12:02:28 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Jan 31 12:02:33 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 12:02:33 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Jan 31 12:02:38 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 12:02:38 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Jan 31 12:02:43 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 12:11:32 fidsrv sshd[24848]: fatal: Read from socket failed: Connection reset by peer
Jan 31 12:11:34 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 12:48:11 fidsrv sshd[19048]: fatal: Read from socket failed: Connection reset by peer
Jan 31 12:48:15 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 13:07:26 fidsrv sshd[22050]: error: PAM: Authentication failed for little from 192.168.172.69
Jan 31 13:07:27 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)
Jan 31 13:45:50 fidsrv sshd[17513]: Failed password for dslawil from 10.158.154.51 port 4651
Jan 31 13:45:53 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)

My desired output:

Code:
Jan 31 12:02:38 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Jan 31 13:45:50 fidsrv sshd[17513]: Failed password for dslawil from 10.158.154.51 port 4651
Jan 31 13:45:53 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)

In other words, I just want LAST uniq occurence of an event.

My feeble attempt with :

Code:
egrep -i 'warning| error| fail' /var/adm/syslog/syslog.log |sort -k 1,2| uniq -f 4

did not work ..

Any ideas appreciated. Thank you
# 2  
Old 01-31-2012
try this:
Code:
egrep -i 'warning| error| fail' /var/adm/syslog/syslog.log |
 awk -F ':'  '{arr[$4 " " $5 " " $6 " "$7]= $1 " " $2 " " $3; next} 
               END {for(i in arr) {print i, arr[i]}} '

# 3  
Old 01-31-2012
This might work as well:
Code:
egrep -i 'warning| error| fail' /var/adm/syslog/syslog.log| sort -rk6 | uniq -f6 | egrep -v "PAM|socket"

# 4  
Old 02-01-2012
Thank you, Jim and bartus11

Jim, your code gave the following ouput ..how do I switch around the field pacements ??

Code:
 Asynchronous write failed on LUN (dev=0x100000f)    Feb  1 07 50 51 fidsrv vmunix
 LVM  WARNING  VG 128 0x001000  LV 5 Feb  1 07 12 22 fidsrv vmunix

bartus11, your solution worked great !!

Code:
Feb  1 07:12:22 fidsrv vmunix: LVM: WARNING: VG 128 0x001000: LV 5: Some I/O requests to this LV are waiting
Feb  1 07:51:26 fidsrv vmunix: Asynchronous write failed on LUN (dev=0x100000f)

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk sort based on difference of fields and print all fields

Hi I have a file as below <field1> <field2> <field3> ... <field_num1> <field_num2> Trying to sort based on difference of <field_num1> and <field_num2> in desceding order and print all fields. I tried this and it doesn't sort on the difference field .. Appreciate your help. cat... (9 Replies)
Discussion started by: newstart
9 Replies

2. Shell Programming and Scripting

Sort & Uniq -u

Hi All, Below the actual file which i like to sort and Uniq -u /opt/oracle/work/Antony/Shell_Script> cat emp.1st 2233|a.k. shukula |g.m. |sales |12/12/52 |6000 1006|chanchal singhvi |director |sales |03/09/38 |6700... (8 Replies)
Discussion started by: Antony Ankrose
8 Replies

3. UNIX for Dummies Questions & Answers

Uniq and sort -u

Hello all, Need to pick your brains, I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters. So I want to do a sort -u file Will it be considerably faster or slower to use a uniq before piping it to sort... (3 Replies)
Discussion started by: senhia83
3 Replies

4. Shell Programming and Scripting

Uniq or sort -u or similar only between { }

Hi ! I am trying to remove doubbled entrys in a textfile only between delimiters. Like that example but i dont know how to do that with sort or similar. input: { aaa aaa } { aaa aaa } output: { aaa } { (8 Replies)
Discussion started by: fugitivus
8 Replies

5. Shell Programming and Scripting

Sort uniq or awk

Hi again, I have files with the following contents datetime,ip1,port1,ip2,port2,number How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up? Please mind the file may contain 100k lines. (8 Replies)
Discussion started by: LDHB2012
8 Replies

6. Shell Programming and Scripting

sort | uniq question

Hello, I have a large data file: 1234 8888 bbb 2745 8888 bbb 9489 8888 bbb 1234 8888 aaa 4838 8888 aaa 3977 8888 aaa I need to remove duplicate lines (where the first column is the duplicate). I have been using: sort file.txt | uniq -w4 > newfile.txt However, it seems to keep the... (11 Replies)
Discussion started by: palex
11 Replies

7. Shell Programming and Scripting

Help with Uniq and sort

The key is first field i want only uniq record for the first field in file. I want the output as or output as Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies

8. UNIX for Dummies Questions & Answers

Finding & Replacing specific Fields

All I have a very large file (aproximately 150,000) as shown below separated by pipe "|". I need to replace data in 2, 16, 17, 23 fields that are of time stamp format. My goal is to look in those fields and it ends with "000000|" then replace it with "000|". In other words, make it as 6 digit... (2 Replies)
Discussion started by: ddraj2015
2 Replies

9. Shell Programming and Scripting

sort and uniq in perl

Does anyone have a quick and dirty way of performing a sort and uniq in perl? How an array with data like: this is bkupArr BOLADVICE_VN this is bkupArr MLT6800PROD2A this is bkupArr MLT6800PROD2A this is bkupArr BOLADVICE_VN_7YR this is bkupArr MLT6800PROD2A I want to sort it... (4 Replies)
Discussion started by: reggiej
4 Replies

10. UNIX for Dummies Questions & Answers

sort/uniq

I have a file: Fred Fred Fred Jim Fred Jim Jim If sort is executed on the listed file, shouldn't the output be?: Fred Fred Fred Fred Jim Jim Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies
Login or Register to Ask a Question