Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove lines of the same time stamp leaving the highest

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove lines of the same time stamp leaving the highest
# 1  
Old 02-11-2010
Remove lines of the same time stamp leaving the highest
Hi guys,

I have a log that looks like that below. Columns 2 3 4 5 6 7 is the date/time stamp separated by comma.
Code:
UUU,02,06,2010,10,00,00,00,0000000000000000,0000000000000000,0000000000001224
UUU,02,06,2010,10,05,00,00,0000000000000000,0000000000000000,0000000000001502
UUU,02,06,2010,10,10,00,00,0000000000000000,0000000000000000,0000000000001514
UUU,02,06,2010,10,10,00,00,0000000000000000,0000000000000000,0000000000001305
UUU,02,06,2010,10,10,00,00,0000000000000000,0000000000000000,0000000000001456
UUU,02,06,2010,10,15,00,00,0000000000000000,0000000000000000,0000000000001324

The problem is it can often contain entries for the same time stamp, I want to remove entries with the same time and just leave one entry that has the highest number in column 11.

So the above log would be outputed as:

Code:
UUU,02,06,2010,10,00,00,00,0000000000000000,0000000000000000,0000000000001224
UUU,02,06,2010,10,05,00,00,0000000000000000,0000000000000000,0000000000001502
UUU,02,06,2010,10,10,00,00,0000000000000000,0000000000000000,0000000000001514
UUU,02,06,2010,10,15,00,00,0000000000000000,0000000000000000,0000000000001324

Anyone know how to do it, it would be nice to be done in perl.
# 2  
Old 02-11-2010
Dunno perl, but hope this awk script helps. I tested it against your input and it worked.
In the code below, replace the /tmp/input.data file to where your input data file points to.

Code:
awk -F',' '{
  ts=$2","$3","$4","$5","$6","$7;
  if( x[ts] == "" ) {
    x[ts] = $0
  }
  else  {
    split(x[ts],f11)
    if( $11 > f11[11] ) {
      x[ts] = $0
    }
  }
}
END {
  for(i in x)  {
    print x[i]
  }
}' /tmp/input.data

# 3  
Old 02-11-2010
Let's try another solution:
Code:
awk -F, 'function X(){return substr($0,5,19)};function Y(){z=$NF;Z[y]=$0}END{print Z[y]}NR==1{y=X();Y()}{x=X();if(x!=y){print Z[y];y=x;Y()}}'  infile

Use gawk, nawk or /usr/xpg4/bin/awk on Solaris.

---------- Post updated at 10:51 PM ---------- Previous update was at 08:15 PM ----------

linuxpenguin you are +54
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove lines having older end time

Hi, In my bash script I want to add a code which remove all entries older than x days. To simplify this problem, I have divided into 3 parts. (2 parts are done looking answer for 3rd part.) To find the latest log date - Done Latest_Time=`find . -name '*.tps' -exec sed '/endTime/!d; s/{//;... (3 Replies)
Discussion started by: Agoyals1986
3 Replies

2. Shell Programming and Scripting

Collecting all lines between two time stamp from the log

Can you help me to collect the entire logs between two time stamp. The below awk command collecting the logs only if the line has time stamp. awk '$0>=from && $0<=to' from="150318 23:19:04" to="150318 23:55:04" log file 150318 23:19:04 logentries 150318 23:29:04 logentries 150318... (11 Replies)
Discussion started by: zenkarthi
11 Replies

3. Shell Programming and Scripting

Leaving last 30 lines

If we want to display lines from file leaving last 30 lines. i dont know the count of lines in file (14 Replies)
Discussion started by: mirwasim
14 Replies

4. Shell Programming and Scripting

Delete lines containing and remove the blank line at the same time

Is there a way to delete a line containing something and the blank line at the same time? If you do this it leaves a blank line behind. sed '/yum/d' .bash_historyI know this works but I would think there would be a way to do it with one command sed '/yum/d' .bash_history | sed '/^$/d'In... (2 Replies)
Discussion started by: cokedude
2 Replies

5. Solaris

System time and Cron time stamp not matching

On Solaris 10 server the system date won't match with the timestamp on files created by a cron jobs, Please help here is what i get when i check for system date infodba-ie10ux014:/tcpdv1_ie10/tcadmin/bin\n\r-> date Tue Apr 24 15:27:43 GMT 2012at same time i executed a cron job, and checked... (4 Replies)
Discussion started by: karghum
4 Replies

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

7. Shell Programming and Scripting

How to get time duration between two human readable time stamp in Unix?

Here is two time I have: Jul 12 16:02:01 Jul 13 01:02:01 and how can I do a simple match to get difference between two time which is 09:00:00 Thanks in advance. (3 Replies)
Discussion started by: ford99
3 Replies

8. Shell Programming and Scripting

shell script to remove all lines which exceeds a particular date & time

I have a text file which got 6th coloumn as date and 7th coloumn as time. The text contains data for last one week. I need to remove all the data whose date & time is after 03/08/2011 06:00:00 and save it on another file TEXT FILE ======== 6 dbclstr-b IXT_Web Memphis_Prod_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

9. Shell Programming and Scripting

remove a colon and number and leaving the rest

just have a file 1:2333 2:-09393 ]3:45453 4:-09999 5:-09933 6:93939 question is to get output by removing colons as well as number before each colon (in bold) 2333 -09393 45453 -09999 09933 93939 (5 Replies)
Discussion started by: cdfd123
5 Replies

10. UNIX for Dummies Questions & Answers

Copy all the files with time stamp and remove header,trailer from file

All, I am new to unix and i have the following requirement. I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file.. Could... (35 Replies)
Discussion started by: ksrams
35 Replies
Login or Register to Ask a Question