Flag duplicate enteries | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Flag duplicate enteries

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 08-19-2013
Junes Junes is offline
Registered User
 
Join Date: Nov 2005
Last Activity: 7 January 2014, 9:25 AM EST
Posts: 56
Thanks: 4
Thanked 0 Times in 0 Posts
Flag duplicate enteries

Morning Unix Guru's,
I have written a script which output the contents of a DB. No the DB is not clever to check this or flag and there is no command for the tool to flag this, which human intervention.

But i need to check to see if there are duplicate enteries in the file output and flag them.

So the example below there is duplicate enteries for actlni02 and allocd01.

How do i flag this?



Code:
actlni02              LDN_CAM_UX_PRD
actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
allocd01              LDN_WAT_UX_DEV

Any advice will be greatly appericated.
Sponsored Links
    #2  
Old 08-19-2013
Jotne's Avatar
Jotne Jotne is offline
Registered User
 
Join Date: Dec 2010
Last Activity: 25 July 2014, 9:30 AM EDT
Posts: 1,038
Thanks: 62
Thanked 216 Times in 204 Posts
Simple solution.

Code:
awk '{a[$1]++;if (a[$1]>1) f="* ";print f$0;f=x}'
actlni02              LDN_CAM_UX_PRD
* actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
* allocd01              LDN_WAT_UX_DEV

The Following User Says Thank You to Jotne For This Useful Post:
RavinderSingh13 (08-20-2013)
Sponsored Links
    #3  
Old 08-19-2013
Junes Junes is offline
Registered User
 
Join Date: Nov 2005
Last Activity: 7 January 2014, 9:25 AM EST
Posts: 56
Thanks: 4
Thanked 0 Times in 0 Posts
Thanks.. So simple
    #4  
Old 08-19-2013
Jotne's Avatar
Jotne Jotne is offline
Registered User
 
Join Date: Dec 2010
Last Activity: 25 July 2014, 9:30 AM EDT
Posts: 1,038
Thanks: 62
Thanked 216 Times in 204 Posts
More complex. This version change color to red on all duplicated lines

Code:
awk '{
	a[$1]++
	b[NR]=$0
	}
END {
	for (i=1;i<=NR;i++) {
		split(b[i],q," +")
		s=(a[q[1]]>1)?"\033[1;31m":"\033[0m"
		print s b[i] "\033[0m"
		}
	}' file


actlni02              LDN_CAM_UX_PRD
actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
allocd01              LDN_WAT_UX_DEV

Edit: some shorter

Code:
awk '{a[$1]++;b[NR]=$0;c[NR]=$1} END {for (i=1;i<=NR;i++) print ((a[c[i]]>1)?"\033[1;31m":"\033[0m") b[i] "\033[0m"}' file


Last edited by Jotne; 08-19-2013 at 07:39 AM..
Sponsored Links
    #5  
Old 08-20-2013
rdcwayx rdcwayx is offline Forum Advisor  
Use nawk in Solaris
 
Join Date: Jun 2006
Last Activity: 22 March 2014, 12:27 PM EDT
Posts: 2,759
Thanks: 44
Thanked 418 Times in 406 Posts
if you need mark all duplicate enteries, try this:


Code:
awk 'NR==FNR{a[$1]++;next} a[$1]>1 {$0="* " $0}1' infile infile

* actlni02              LDN_CAM_UX_PRD
* actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
* allocd01              LDN_CAM_UX_DEV
* allocd01              LDN_WAT_UX_DEV

Sponsored Links
    #6  
Old 08-20-2013
Jotne's Avatar
Jotne Jotne is offline
Registered User
 
Join Date: Dec 2010
Last Activity: 25 July 2014, 9:30 AM EDT
Posts: 1,038
Thanks: 62
Thanked 216 Times in 204 Posts
Nice. It will be much faster to read file twice compare to add file to array than loop trough every element end test.
Here is a color version of rdcwayxs script

Code:
awk 'NR==FNR{a[$1]++;next} a[$1]>1 {$0="\033[1;31m" $0 "\033[0m"}1' file file


Last edited by Jotne; 08-20-2013 at 03:49 AM.. Reason: Shorten it some
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
'h' flag in du sudon't UNIX for Dummies Questions & Answers 2 07-22-2012 01:29 PM
Find duplicate based on 'n' fields and mark the duplicate as 'D' machomaddy Shell Programming and Scripting 7 01-28-2012 05:26 PM
vfstab enteries automatically added during live upgrade chidori Solaris 0 01-23-2012 11:10 AM
du flag -x Casey AIX 6 09-16-2009 03:56 AM
LOST my crontab enteries of root wakhan UNIX for Advanced & Expert Users 2 09-04-2008 06:55 AM



All times are GMT -4. The time now is 11:16 AM.