Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Request to check:Remove duplicates

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Request to check:Remove duplicates
# 1  
Old 07-18-2012
Request to check:Remove duplicates
Hi all

I have a file with following kind of data


Quote:
abc 123
cdf 456
abc 123
abc 3455
xyz 3456
def 04923
xyz 3456


I want to remove duplicates according to first column so that output contains

Quote:
abc 123
cdf 456
xyz 3456
def 04923
Kindly let me scripting regading this.
# 2  
Old 07-18-2012
Code:
perl -lane 'if(exists $x{$F[0]}){next}else{$x{$F[0]}=$F[1]} END {for(sort keys %x){print "$_ $x{$_}"}}' inputfile

# 3  
Old 07-18-2012
Code:
 
$ awk '{if (!a[$1]++){print}}'  inputfile

Last edited by raj_saini20; 07-18-2012 at 09:33 AM..
# 4  
Old 07-18-2012
Code:
$ awk '!a[$1]{a[$1]=$2;next}END{for(i in a){print i,a[i]}}' input.txt
def 04923
abc 123
cdf 456
xyz 3456

---------- Post updated at 05:50 PM ---------- Previous update was at 05:49 PM ----------

Code:
$ awk '!a[$1]++' input.txt
abc 123
cdf 456
xyz 3456
def 04923

# 5  
Old 07-18-2012
Try.....
Code:
awk -F" " '!_[$1]++' test1.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request to check:remove duplicates only in first column

Hi all, I have an input file like this Now I have to remove duplicates only in first column and nothing has to be changed in second and third column. so that output would be Please let me know scripting regarding this (20 Replies)
Discussion started by: manigrover
20 Replies

2. Shell Programming and Scripting

Request to check:remove entries with duplicate numbers in first row

Hi I have a file 1 xyz 456 1 xyz 456 1 xyz 456 2 abc 8459 3 gfd 657 4 ghf 658 4 ghf 658 I want the output 1 xyz 456 2 abc 8459 3 gfd 657 4 ghf 658 (3 Replies)
Discussion started by: manigrover
3 Replies

3. Shell Programming and Scripting

Kindly check:remove duplicates with similar data in front of it

Hi all, I have 2 files containing data like this: so if there is same entry repeated in the column like1,2,3,4 I have to check if there is different entries column like 2,4 but similar entries for duplicatein column 2 like1,3 the output shuld be like this for first file ... (5 Replies)
Discussion started by: manigrover
5 Replies

4. Shell Programming and Scripting

Request to check:remove duplicates and write sytematically

Hi all I have a file with following input It contains 5 columns gene name drug drug ID disease approved Now the same gene is repeated many times with different data in column2,3 ,4,5 I want to arrange dat in such a way that there shuld be one entry in the column(no... (2 Replies)
Discussion started by: manigrover
2 Replies

5. Shell Programming and Scripting

Request to check remove duplicates but write before it

Hi alll I have a file with following kind input I want in output duplicates should not be there but there should be numbering mentioned before that like (4 Replies)
Discussion started by: manigrover
4 Replies

6. Shell Programming and Scripting

Request to check:remove entries more than once in different column

Hi I have a file 12m 345693460 12 1234 12 1234 34 345 34 345 And I want output fiel as 12m 345693460 12 1234 34 345 hw can it be done Thanks (1 Reply)
Discussion started by: manigrover
1 Replies

7. Shell Programming and Scripting

Request to check:remove entries with N/A mentioned

Hi I have a file with following entries 122 N/A 123 5654656 123423 43534543 4544 45435 435454 N/A i Have to remove entries with N/A so that only 123 5654656 123423 43534543 4544 45435 remain in output file can anybody guide for a code/unix/perl (2 Replies)
Discussion started by: manigrover
2 Replies

8. Shell Programming and Scripting

Request to check:remove entries with blank spaces

Hi I want to remove entries with blank spaces in my input file: 123 234 456 678 56789 345346456 589 3454 345456 3454566............................ (2 Replies)
Discussion started by: manigrover
2 Replies

9. Shell Programming and Scripting

Request to check:remove entries more than once

Hi I have a file like this 1234 2345 567889 567889 2345 234899420 83743 2345 67890 67890 ................ so on I want to delete entries which are more than once like 2345, 567889 and 67890 so that these appear once (4 Replies)
Discussion started by: manigrover
4 Replies

10. Shell Programming and Scripting

Request to check:remove entries with N/A entries

Hi I have a file with numerous entries some entries are 1 mani 2 kavya 3 N/A 4 Praveeen 5 N/A and so on How to remove entries with N/A so the result will be 1 mani 2 kavya 3 Praveeen (6 Replies)
Discussion started by: manigrover
6 Replies
Login or Register to Ask a Question