Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
google site



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Closed Thread
English Japanese Spanish French German Portuguese Italian Powered by Powered by Google
 
Thread Tools Search this Thread Display Modes
  #1  
Old 03-11-2009
Registered User
 

Join Date: Dec 2008
Posts: 11
Thanks: 0
Thanked 0 Times in 0 Posts
find out duplicate records in file?

Dear All,

I have one file which looks like :

account1asswd1
account2asswd2
account3asswd3
account1asswd4
account5asswd5
account6asswd6

you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in file?

Thanks.
Sponsored Links
  #2  
Old 03-11-2009
Registered User
 

Join Date: Jun 2007
Location: Beijing China
Posts: 1,177
Thanks: 0
Thanked 3 Times in 3 Posts

Code:
awk -F":" '{_[$1]++}
END{
  for (i in _)
   print "Duplicated: _[i]
}' filename

  #3  
Old 03-11-2009
Registered User
 

Join Date: Dec 2008
Posts: 11
Thanks: 0
Thanked 0 Times in 0 Posts
i guess there's one missing " in 4th line?
but after appending the " , it still doesn't work...
  #4  
Old 03-12-2009
cfajohnson's Avatar
Shell programmer, author
 

Join Date: Mar 2007
Location: Toronto, Canada
Posts: 2,642
Thanks: 0
Thanked 15 Times in 15 Posts
Quote:
Originally Posted by tiger2000 View Post
Dear All,

I have one file which looks like :

account1asswd1
account2asswd2
account3asswd3
account1asswd4
account5asswd5
account6asswd6

you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in file?


Code:
awk -F: 'x[$1]++ { print $1 " is duplicated"}' FILENAME

Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
compare fields in a file with duplicate records rleal Shell Programming and Scripting 1 01-29-2009 01:55 AM
find duplicate records... again rleal Shell Programming and Scripting 4 01-28-2009 05:30 PM
How to find Duplicate Records in a text file G.Aavudai Shell Programming and Scripting 1 10-17-2008 04:59 AM
Duplicate records from oracle to text file. shilendrajadon UNIX for Advanced & Expert Users 1 01-10-2008 10:21 AM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 06:50 AM



All times are GMT -4. The time now is 02:32 AM.