Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove lines, Sorted with Time based columns using AWK & SORT
# 1  
Old 05-08-2008
Remove lines, Sorted with Time based columns using AWK & SORT
Hi

having a file as follows

MediaErr.log
Code:
84   Server1     Policy1       Schedule1 master1   05/08/2008 02:12:16
84   Server1     Policy1       Schedule1 master1   05/08/2008 02:22:47
84   Server1     Policy1       Schedule1 master1   05/08/2008 03:41:26
84   Server1     Policy1       Schedule1 master1   05/08/2008 03:42:26

using below command to print duplicate line with latest time
Code:
nawk '!arr[$1]++ || arr[$3]++ || arr[$7]>=1' MediaErr.log

Current Output
Code:
84   Server1     Policy1       Schedule1 master1   05/08/2008 02:12:16
84   Server1     Policy1       Schedule1 master1   05/08/2008 03:42:26

but could not get exact output

it should be
Code:
84   Server1     Policy1       Schedule1 master1   05/08/2008 02:22:47
84   Server1     Policy1       Schedule1 master1   05/08/2008 03:42:26

Smilie any suggestions
# 2  
Old 05-10-2008
I guess you're looking for the most recent timestamp within an hour for your log file (?).

If sorting is not an issue use:

Code:
 
awk '{split($NF,a,":"); b[a[1]]=$0}END{for (i in b) print b[i]}' MediaErr.log

Otherwise sort your file first:

Code:
sort -k 7.1,7.2n -k 7.4,7.5n -k 7.7,7.8n  MediaErr.log | awk '{split($NF,a,":"); b[a[1]]=$0}END{for (i in b) print b[i]}'

Last edited by rubin; 05-11-2008 at 01:58 AM.. Reason: typo, replaced sort keys 7.6,7.7 --> 7.7,7.8 in the last code.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies

2. Shell Programming and Scripting

Sort based on Multiple Columns in UNIX

Hi, I would like to sort a list in different ways: 1> Unique based on Field 1 with highest Field 4 For Instance Input: 1678923450;11112222333344;11-1x;2_File.xml 1678923450;11112222333344;11-1x;5_File.xml 1234567890;11113333222244;11-1x;3_File.xml Output: ... (7 Replies)
Discussion started by: DevendraG
7 Replies

3. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

4. UNIX for Dummies Questions & Answers

sort comma separated lines by specific columns

Hello, I have a file which lines' words are comma separated: aa, bb, cc, uu b, ee, ff bb, cc, zz, ee, ss, kk oo, bb, hh, uu a, xx, ww tt, aa, dd, yy aa, gg I want to sort first by second column and in case of tie by fourth column with sort command. So the output would be: ... (4 Replies)
Discussion started by: asanchez
4 Replies

5. Shell Programming and Scripting

sort the files based on timestamp and execute sorted files in order

Hi I have a requirement like below I need to sort the files based on the timestamp in the file name and run them in sorted order and then archive all the files which are one day old to temp directory My files looks like this PGABOLTXML1D_201108121235.xml... (1 Reply)
Discussion started by: saidutta123
1 Replies

6. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

7. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

8. Shell Programming and Scripting

shell script to remove all lines which exceeds a particular date & time

I have a text file which got 6th coloumn as date and 7th coloumn as time. The text contains data for last one week. I need to remove all the data whose date & time is after 03/08/2011 06:00:00 and save it on another file TEXT FILE ======== 6 dbclstr-b IXT_Web Memphis_Prod_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

9. Shell Programming and Scripting

sort by based on multiple columns

Hi, Is there any way to sort a file in cshell by sort command, sorting it by multiple fields, like to sort it first by the second column and then by the first column. Thanks forhead (1 Reply)
Discussion started by: Takeeshe
1 Replies

10. Shell Programming and Scripting

awk : extracting unique lines based on columns

Hi, snp.txt CHR_A SNP_A BP_A_st BP_A_End CHR_B BP_B SNP_B R2 p-SNP_A p-SNP_B 5 rs1988728 74904317 74904318 5 74960646 rs1427924 0.377333 0.000740085 0.013930081 5 ... (12 Replies)
Discussion started by: genehunter
12 Replies
Login or Register to Ask a Question