Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Flag duplicate enteries

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Flag duplicate enteries
# 1  
Old 08-19-2013
Flag duplicate enteries
Morning Unix Guru's,
I have written a script which output the contents of a DB. No the DB is not clever to check this or flag and there is no command for the tool to flag this, which human intervention.

But i need to check to see if there are duplicate enteries in the file output and flag them.

So the example below there is duplicate enteries for actlni02 and allocd01.

How do i flag this?


Code:
actlni02              LDN_CAM_UX_PRD
actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
allocd01              LDN_WAT_UX_DEV

Any advice will be greatly appericated.
# 2  
Old 08-19-2013
Simple solution.
Code:
awk '{a[$1]++;if (a[$1]>1) f="* ";print f$0;f=x}'
actlni02              LDN_CAM_UX_PRD
* actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
* allocd01              LDN_WAT_UX_DEV

# 3  
Old 08-19-2013
Thanks.. So simple
# 4  
Old 08-19-2013
More complex. This version change color to red on all duplicated lines
Code:
awk '{
	a[$1]++
	b[NR]=$0
	}
END {
	for (i=1;i<=NR;i++) {
		split(b[i],q," +")
		s=(a[q[1]]>1)?"\033[1;31m":"\033[0m"
		print s b[i] "\033[0m"
		}
	}' file


actlni02              LDN_CAM_UX_PRD
actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
allocd01              LDN_WAT_UX_DEV

Edit: some shorter
Code:
awk '{a[$1]++;b[NR]=$0;c[NR]=$1} END {for (i=1;i<=NR;i++) print ((a[c[i]]>1)?"\033[1;31m":"\033[0m") b[i] "\033[0m"}' file

Last edited by Jotne; 08-19-2013 at 08:39 AM..
# 5  
Old 08-20-2013
if you need mark all duplicate enteries, try this:

Code:
awk 'NR==FNR{a[$1]++;next} a[$1]>1 {$0="* " $0}1' infile infile

* actlni02              LDN_CAM_UX_PRD
* actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
* allocd01              LDN_CAM_UX_DEV
* allocd01              LDN_WAT_UX_DEV

# 6  
Old 08-20-2013
Nice. It will be much faster to read file twice compare to add file to array than loop trough every element end test.
Here is a color version of rdcwayxs script
Code:
awk 'NR==FNR{a[$1]++;next} a[$1]>1 {$0="\033[1;31m" $0 "\033[0m"}1' file file

Last edited by Jotne; 08-20-2013 at 04:49 AM.. Reason: Shorten it some
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Get the id with specify flag

.... means multi line ddd,bug fgdrg dfdfsdfdfsd fsdfdfdfd fdfdsfdsfsd ....... flag2 ...... aaa,bug sfsfsfsfs dfdfsdfdfsd fsdfdfdfd fdfdsfdsfsd ...... flag1 ...... ddd,bug fgdrg dfdfsdfdfsd (9 Replies)
Discussion started by: yanglei_fage
9 Replies

2. Shell Programming and Scripting

Scripting Question - getting lines above ane below certain enteries

Hello - I have a file in the followig format LINE TEXT 1 Line TEXT 2 TIMESTARTED=Fri Aug 16 15:20:23 EDT 2013 START-OF-DATA 123 123444 23232323 END-OF-DATA Line TEXT 9 I need to get all the lines between the Start of Data and End of Data and pipe it to a file so output should be ... (5 Replies)
Discussion started by: mnassiri
5 Replies

3. UNIX for Dummies Questions & Answers

'h' flag in du

Hey, all! Why is the "human readable" flag changing the behavior of du? And while I'm at it, can you make du only look at files, not directories. I often find myself wanting to find the largest file(s) in a dir or vol. Using 'find' itself, it seems you have to at least be able to guess the size of... (2 Replies)
Discussion started by: sudon't
2 Replies

4. AIX

Flag issue

Hi all, We dont have access to aix source code and i have a doubt. The flag SC_NO_RESERVE, is it got to do anything with the failover? If the flag is set the paths are going to failed state. If flag is not set everything comes up fine after failover. Thanks in advance for helping ... (2 Replies)
Discussion started by: gnakul
2 Replies

5. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

6. Solaris

vfstab enteries automatically added during live upgrade

i am trying to patch a solaris 10 server using live upgrade. issue is, when i create a new BE and activate it during reboot the file system that are mounted but doesnt have an entry in vfstab are automatically added in vfstab of new BE. looks like live upgrade uses df -h output as reference... (0 Replies)
Discussion started by: chidori
0 Replies

7. Shell Programming and Scripting

Add the flag

#!/bin/bash while : do ./abc.sh PB sleep 60 ./abc.sh RA sleep 60 ./abc.sh GS sleep 68400 done Instead of making the script sleep for sometime, it doesn't work all the time as time may shift over a period. How to make a script wake up every 30 seconds and check the current time, if... (2 Replies)
Discussion started by: sandy1028
2 Replies

8. AIX

du flag -x

Hi, I would like to know if there's any option to use with the du command so that I can list only the files/directories on the current filesystem... I usually use du -gs *But I'd like to see only the directories in the filesystem I am on, and not the mount point directory of other fss... ... (6 Replies)
Discussion started by: Casey
6 Replies

9. UNIX for Advanced & Expert Users

LOST my crontab enteries of root

Hello All, I did very stupid mistake and lost all my crontab enteries on root prompt. Please help me to solve this issue. i did entry on root prompt like this. crontab /home/back.cron and enters but when i enter crontab -l so its only showing 1 entry defined in back.cron and all... (2 Replies)
Discussion started by: wakhan
2 Replies

10. Shell Programming and Scripting

the flag -c in sh command

I need your help please. In a production system, i've seen many running process as follow: sh -c ./pathname/shellname what exactly the flag option -c is used for? ive tried to look at the man page, but it doesnt say much. ill appreciate yor help. Thanks (4 Replies)
Discussion started by: alexcol
4 Replies
Login or Register to Ask a Question