Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

identifying duplicate entries

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting identifying duplicate entries
# 1  
Old 06-20-2010
identifying duplicate entries
hi all,
have got a large log file and was wondering if there is a easy way on solaris box to grep out duplicate entries based on email address ??

sample log file :

Code:
2010-06-19,04:08:12,235632470,2010-06-18T00:00:00.000+12:00,zinny123@hotmail.com
2010-06-19,04:09:57,235632470,2010-06-18T00:00:00.000+12:00,zinny123@hotmail.com
2010-06-19,04:28:36,223906214,2010-06-18T00:00:00.000+12:00,zkml123@xtra.co.nz
2010-06-19,04:01:51,101427641,2010-06-18T00:00:00.000+12:00,zl2t890@orcon.net.nz
2010-06-19,04:03:40,101427641,2010-06-18T00:00:00.000+12:00,zl2890@orcon.net.nz


thanks in advance.
# 2  
Old 06-20-2010
Remove consecutive duplicate
Code:
awk -F, '$NF!=f{print}{f=$NF}' file

# 3  
Old 06-21-2010
Hi

Code:
sort -u -t, -k5 file

Guru.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove columns with duplicate entries

I have a 13gb file. It has the following columns: The 3rd column is basically correlation values. I want to delete those rows which are repeated between the columns: A B 0.04 B C 0.56 B B 1 A A 1 C D 1 C C 1 Desired Output: (preferably in a .csv format A,B,0.04 B,C,0.56 C,D,1... (3 Replies)
Discussion started by: Sanchari
3 Replies

2. Shell Programming and Scripting

How to keep the last 2 records from duplicate entries?

Gents, Please how I can get only the last 2 records from repetead values, from column 2 input 1 1011 1 1011 1 1012 1 1012 1 5001 1 5001 1 5002 1 5002 1 5003 1 5003 1 7001 1 7001 1 7002 1 7002 (2 Replies)
Discussion started by: jiam912
2 Replies

3. Shell Programming and Scripting

Append values of duplicate entries

My input file is: LOC_Os01g01870 GO:0006139 LOC_Os01g01870 GO:0009058 LOC_Os01g02570 GO:0006464 LOC_Os01g02570 GO:0009987 LOC_Os01g02570 GO:0008152 LOC_Os01g04380 GO:0006950 LOC_Os01g04380 GO:0009628 I want to append the duplicate values in a tab/space... (2 Replies)
Discussion started by: Sanchari
2 Replies

4. Shell Programming and Scripting

How to find duplicate entries

I have a file contails as below I/P: 123456 123456 234567 987654 678905 678905 Like above i have 1000's of entries I need output as below O/P: 123456 678905 I'm using uniq -d filename it is showing results but it is missing few duplicate entries and i dont know why.Please... (9 Replies)
Discussion started by: buzzme
9 Replies

5. Shell Programming and Scripting

Identifying entries based on 2 fields in a string.

Hi Guys, I’m struggling to use two fields to do a duplicate/ unique by output. I want to look IP addresses assigned to more than one account during a given period in the logs. So duplicate IP and account > 1 then print all the logs for that IP. I have been Using AWK (just as its installed... (3 Replies)
Discussion started by: wabbit02
3 Replies

6. Solaris

duplicate PATH entries

I noticed we have duplicate (or even triple) entries in PATH env variable. Is there any reason to have them all or is it safe to leave only one? thanks. (2 Replies)
Discussion started by: orange47
2 Replies

7. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi, I forgot how to start a new thread. :( Can somebody please guide me? I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like wcars42g... (2 Replies)
Discussion started by: akash_mahakode
2 Replies

8. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi All, I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like Code: wcars42g wcars89j wcars42g wcars42b wcars42b Will duplicate entries... (1 Reply)
Discussion started by: akash_mahakode
1 Replies

9. Shell Programming and Scripting

Removal of Duplicate Entries from the file

I have a file which consists of 1000 entries. Out of 1000 entries i have 500 Duplicate Entires. I want to remove the first Duplicate Entry (i,e entire Line) in the File. The example of the File is shown below: 8244100010143276|MARISOL CARO||MORALES|HSD768|CARR 430 KM 1.7 ... (1 Reply)
Discussion started by: ravi_rn
1 Replies

10. Solaris

Duplicate crontab entries

hi guys. can someone tell me what will happen if there are two identical crontab entry for an application. For example 03 23 * * 1 /usr/vt/crondemo 03 23 * * 1 /usr/vt/crondemo will the file crondemo run twice, once or wont run at all?? (2 Replies)
Discussion started by: vikashtulsiyan
2 Replies
Login or Register to Ask a Question