Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Log file - Delete duplicate line & keep last date

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Log file - Delete duplicate line & keep last date
# 1  
Old 02-08-2016
Code Log file - Delete duplicate line & keep last date
Hello All !

I need your help on this case,

I have a csv file with this:

Code:
ITEM105;ARI FSR;2016-02-01 08:02;243 
ITEM101;ARI FSR;2016-02-01 06:02;240  
ITEM032;RNO TLE;2016-02-01 11:03;320  
ITEM032;RNO TLE;2016-02-02 05:43;320 
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO TLE;2016-01-01 14:03;320

And I want to delete duplicate line (ITEMxxx) and keep only the last date, in this example the result must be:

Code:
ITEM105;ARI FSR;2016-02-01 08:02;243 
ITEM101;ARI FSR;2016-02-01 06:02;240  
ITEM032;RNO TLE;2016-02-02 05:43;320

I try this:
Code:
awk '!x[$1,$2]++' FS=";" FILE

but the result is not good:
Code:
ITEM032;RNO TLE;2016-02-01 11:03;320

Do you have an idea for this problem ?
Thanks !
Moderator's Comments:
Mod Comment Please use CODE tags for sample input, sample output, AND code segments.
Last edited by Don Cragun; 02-08-2016 at 12:26 PM..
# 2  
Old 02-08-2016
Please use code tags as required by forum rules!

---------- Post updated at 15:21 ---------- Previous update was at 15:18 ----------

How about
Code:
sort -t";" -k3,3r file | awk -F\; '!L[$1]++'
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240

This User Gave Thanks to RudiC For This Post:
vadim-bzh
# 3  
Old 02-08-2016
Thank you rudiC, It works well !!!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need Script to ZIP/SAVE & then DELETE Log file & send a mail conformation for any error

ENVIROMENT Linux: RHEL 6.4 Log Path: /usr/iplanet/servers/https-company/logs Log Format: user.log.03-15-2015 I have log4j log rotation enabled rotating files on a daily basis. The rotated logs are NOT compressed & are taking up too much space. I need a script that will run daily that... (1 Reply)
Discussion started by: admin_job_admin
1 Replies

2. Red Hat

Need Script to ZIP/SAVE & then DELETE Log file & DELETE ZIPS older than 12 months

ENVIROMENT Linux: Fedora Core release 1 (Yarrow) iPlanet: iPlanet-WebServer-Enterprise/6.0SP1 Log Path: /usr/iplanet/servers/https-company/logs I have iPlanet log rotation enabled rotating files on a daily basis. The rotated logs are NOT compressed & are taking up too much space. I... (7 Replies)
Discussion started by: zachs
7 Replies

3. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd... (2 Replies)
Discussion started by: sreekumarhari
2 Replies

4. Shell Programming and Scripting

Delete Duplicate line (not really) from the file

I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size). Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies

5. UNIX for Dummies Questions & Answers

Removing duplicate rows & selecting only latest date

Gurus, From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column). Ex: Input File: Output File: I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Discussion started by: shash
5 Replies

6. Shell Programming and Scripting

Delete lines in a file by date on the line

Hi I have a file with lines ending with a date in format dd/mm/yyyy see example below: a|b|c|08/01/2011 d|a|e|31/11/2010 e|d|f|20/11/2010 f|s|r|18/01/2011 What I would like to do is delete all lines with a date older than 30 days. With above example I should be left with a file... (5 Replies)
Discussion started by: fas1
5 Replies

7. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

8. Shell Programming and Scripting

How to find duplicate line in log file?

Hi guys, I'm really happy to find this forum I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated? I already... (2 Replies)
Discussion started by: wax_light
2 Replies

9. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

10. Shell Programming and Scripting

To remove date and duplicate rows from a log file using unix commands

Hi, I have a log file having size of 48mb. For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages. The following things to be done : 1) To remove all the date and time from the log file 2) To remove all the... (1 Reply)
Discussion started by: Pank10
1 Replies
Login or Register to Ask a Question