Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extracting duplicates from a desired field

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extracting duplicates from a desired field
# 1  
Old 11-23-2010
Question Extracting duplicates from a desired field
Hello,

I have a file of group names and GID's (/etc/group) and I want to find the duplicate group names and put them in a file. So there are 2 fields, i.e.:
Code:
audit 10
avahi 70
avahi-autoipd 103
bellrpi 605
bin 1
bin 2
bord 512
busobj 161
bwadm 230
cali81 202
card 323
cardiff 901
cbm 165
cdas 500
cdas 600
...

So, I want to find the duplicates from field 1, group name. In this example I want to end up with:
Code:
bin 1
bin 2
cdas 500
cdas 600
...

Any takers? Googling is not cutting it.

Thanks,

mgb
Last edited by Franklin52; 11-23-2010 at 02:36 PM.. Reason: Please use code tags
# 2  
Old 11-23-2010
Code:
awk 'NR==FNR{a[$1]++;next}a[$1]>1' file file

# 3  
Old 11-23-2010
With one pass:
Code:
awk 'a[$1]==1{print x};!a[$1]++{x=$0};a[$1]>1' file

This User Gave Thanks to bartus11 For This Post:
mgb
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

2. Shell Programming and Scripting

Deleting only 2nd and third duplicates in field 2

(7 Replies)
Discussion started by: newbie2010
7 Replies

3. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

4. UNIX for Dummies Questions & Answers

remove duplicates based on a field and criteria

Hi, I have a file with fields like below: A;XYZ;102345;222 B;XYZ;123243;333 C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 desired output: C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 ie, if the 4rth field is a duplicate.. i need only those records where... (5 Replies)
Discussion started by: wanderingmind16
5 Replies

5. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies

6. Shell Programming and Scripting

Remove the partial duplicates by checking the length of a field

Hi Folks - I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below: aaaaaa|a12|45|56 abbbbaaa|a12|45|56 bbaabb|b1|51|45 bbbbbabbb|b2|51|45 aaabbbaaaa|a11|45|56 ... (3 Replies)
Discussion started by: asyed
3 Replies

7. Shell Programming and Scripting

need Shell script for Sort BASED ON FIRST FIELD and PRINT THE WHOLE FILE WITHOUT DUPLICATES

Can some one provide me a shell script. I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique. Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies

8. Shell Programming and Scripting

extracting desired value from cmd o/p

Hi I m using vxassist cmd to get avaialbe space in DG o/p is like this, wat I need is only numeric value of the o/p , I have a solution is anyone can provide better x1# vxassist -g appdg1 maxsize Maximum volume size: 90406912 (44144Mb) I need only 44144 value command I m using rite... (7 Replies)
Discussion started by: tarunn.dubeyy
7 Replies

9. Shell Programming and Scripting

Awk to find duplicates in 2nd field

I want to find duplicates in file on 2nd field i wrote this code: nawk '{a++} END{for i in a {if (a>1) print}}' temp Could not find whats wrong with this. Appreciate help (5 Replies)
Discussion started by: pinnacle
5 Replies

10. Shell Programming and Scripting

Removing duplicates in a sorted file by field.

I have data like this: It's sorted by the 2nd field (TID). envoy,90000000000000634600010001,04/11/2008,23:19:27,RB00266,0015,DETAIL,ERROR, envoy,90000000000000634600010001,04/12/2008,04:23:45,RB00266,0015,DETAIL,ERROR,... (1 Reply)
Discussion started by: kinksville
1 Replies
Login or Register to Ask a Question