Sponsored Content
Top Forums Shell Programming and Scripting Find duplicates in 2 & 3rd column and their ID Post 303000201 by zaxxon on Thursday 6th of July 2017 07:59:14 AM
Old 07-06-2017
You are building a key with $3 and $4 where there exists only fields $1 $2 $3 with an unaltered field separator.

If I understood it correct, you might want something like this?
Code:
$ awk '{a[$2 FS $3]++; b[$2 FS $3]=$1} END{for(e in a){if(a[e] > 1){print a[e], b[e], e}}}' infile| sort -r
3 0.910094 Aahan Aabid
3 0.893134 Aahva Aaban
3 0.810667 Aaditya Aadhan
3 0.491736 Aagneya Aadam
3 0.484934 Aadvik Aadavan
3 0.366954 Aagney Aadan
3 0.265232 Aahlaad Aabheer
3 0.193443 Aaditeya Aadhira
3 0.173545 Aadrian Aadesh
3 0.151862 Aagam Aadarsh
2 0.845814 Aadinath Aadil
2 0.585537 Aadit Aadi
2 0.291066 Aadil Aadinath
2 0.152208 Aadir Aadidev

I am not sure if it makes sense to process $1 this way, since for example "Aahan Aabid" has 3 entries with 2 different numbers in front. If $1 makes a difference and needs to be printed, you might include $1 into the key instead of using it in a second array.

Last edited by zaxxon; 07-06-2017 at 09:54 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find duplicates in the first column of text file

Hello, My text file has input of the form abc dft45.xml ert rt653.xml abc ert57.xml I need to write a perl script/shell script to find duplicates in the first column and write it into a text file of the form... abc dft45.xml abc ert57.xml Can some one help me plz? (5 Replies)
Discussion started by: gameboy87
5 Replies

2. Shell Programming and Scripting

need to remove duplicates based on key in first column and pattern in last column

Given a file such as this I need to remove the duplicates. 00060011 PAUL BOWSTEIN ad_waq3_921_20100826_010517.txt 00060011 PAUL BOWSTEIN ad_waq3_921_20100827_010528.txt 0624-01 RUT CORPORATION ad_sade3_10_20100827_010528.txt 0624-01 RUT CORPORATION ... (13 Replies)
Discussion started by: script_op2a
13 Replies

3. Shell Programming and Scripting

AWK script to create max value of 3rd column, grouping by first column

Hi, I need an awk script (or whatever shell-construct) that would take data like below and get the max value of 3 column, when grouping by the 1st column. clientname,day-of-month,max-users ----------------------------------- client1,20120610,5 client2,20120610,2 client3,20120610,7... (3 Replies)
Discussion started by: ckmehta
3 Replies

4. Shell Programming and Scripting

Find duplicates in column 1 and merge their lines (awk?)

Hi, I have a file (sorted by sort) with 8 tab delimited columns. The first column contains duplicated fields and I need to merge all these identical lines. My input file: comp100002 aaa bbb ccc ddd eee fff ggg comp100003 aba aba aba aba aba aba aba comp100003 fff fff fff fff fff fff fff... (5 Replies)
Discussion started by: falcox
5 Replies

5. UNIX for Dummies Questions & Answers

Search word in 3rd column and move it to next column (4th)

Hi, I have a file with +/- 13000 lines and 4 column. I need to search the 3rd column for a word that begins with "SAP-" and move/skip it to the next column (4th). Because the 3rd column need to stay empty. Thanks in advance.:) 89653 36891 OTR-60 SAP-2 89653 36892 OTR-10 SAP-2... (2 Replies)
Discussion started by: AK47
2 Replies

6. Shell Programming and Scripting

Find smallest & largest in every column

Dear All, I have input like this, J_15TEST_ASH05_33A22.13885.txt: $$ 1 MAKE SP1501 1 1 4 6101 7392 2 2442 2685 18 3201 4008 20 120 4158 J_15TEST_ASH05_33A22.13885.txt: $$ 1 MAKE SP1502 1 1 4 5125 6416 2 ... (4 Replies)
Discussion started by: attila
4 Replies

7. Shell Programming and Scripting

Changing values only in 3rd column and 4th column

#cat file testing test! nipw asdkjasjdk ok! what !ok host server1 check_ssh_disk!102.56.1.101!30!50!/ other host server 2 des check_ssh_disk!192.6.1.10!40!30!/ #grep check file| awk -F! '{print $3,$4}'|awk '{gsub($1,"",$1)}1' 50 30 # Output: (6 Replies)
Discussion started by: kenshinhimura
6 Replies

8. Shell Programming and Scripting

Solution for replacement of 4th column with 3rd column in a file using awk/sed preserving delimters

input "A","B","C,D","E","F" "S","T","U,V","W","X" "AA","BB","CC,DD","EEEE","FFF" required output: "A","B","C,D","C,D","F" "S", T","U,V","U,V","X" "AA","BB","CC,DD","CC,DD","FFF" tried using awk but double quotes not preserving for every field. any help to solve this is much... (5 Replies)
Discussion started by: khblts
5 Replies

9. Shell Programming and Scripting

awk to Sum columns when other column has duplicates and append one column value to another with Care

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (1 Reply)
Discussion started by: as7951
1 Replies

10. UNIX for Beginners Questions & Answers

UNIX script to compare 3rd column value with first column and display

Hello Team, My source data (INput) is like below EPIC1 router EPIC2 Targetdefinition Exp1 Expres rtr1 Router SQL SrcQual Exp1 Expres rtr1 Router EPIC1 Targetdefinition My output like SQL SrcQual Exp1 Expres Exp1 Expres rtr1 Router rtr1 Router EPIC1 Targetdefinition... (5 Replies)
Discussion started by: sekhar.lsb
5 Replies
makedbm(1M)															       makedbm(1M)

NAME
makedbm - make a Network Information System database SYNOPSIS
nis_input_file] nis_output_name] nis_domain_name] nis_master_name] infile outfile database_name Remarks The Network Information Service (NIS) was formerly known as Yellow Pages (yp). Although the name has changed, the functionality of the service remains the same. DESCRIPTION
generates databases (maps) for the Network Information System (NIS) from infile. A database created by consists of two files: and A data- base contains records called dbm records composed of key-value pairs. Each line of infile is converted to a single dbm record; all characters up to the first tab or space form the key, and the remainder of the line is the value. If a value read from infile ends with the value for that record is continued onto the next line. The NIS clients must interpret the character (which means that does not treat the as if it precedes a comment). If infile is a hyphen reads standard input. always generates a special dbm record with the key whose value is the time of last modification of infile (or the current time, if infile is This value is also known as the order number of a map, and prints it for a specified NIS map (see yppoll(1M)). Another special dbm record created by has the key Its value is usually the host name retrieved by however, the option can be used to spec- ify a different value (see gethostname(2)). If the option is used, another special dbm record with the key is created. When this key exists in the NIS host.by* maps or ipnodes.by* maps and the NIS host name resolution fails, the process will query the Internet domain name server, to provide the host name resolution. Before using the option, it is recommended that the name services switch, be set to allow NIS host name resolution first. (Note that, since the process only checks hosts.by* and ipnodes.by* for the existence of the key, using the option on any other NIS map will have no effect. Also, the option should be used on both the *.byname and *.byaddr maps, not one exclusively.) If the option is used, another special dbm record created is the key. If this key exists in an NIS map, will only allow privileged pro- cesses (applications that can create reserved ports) to access the data within the map. Options recognizes the following options and command-line arguments. Create a special dbm record with the key This key, which is in the hosts.byname, hosts.byaddr, ipnodes.byname, and ipnodes.byaddr maps, allows the process to query the Internet domain name server (see named(1M)). Convert the keys of the given map to lowercase. This command option allows host name matches to work independent of character-case distinctions. Accept connections from secure NIS networks only. Create a special dbm record with the key and the value If the option is used, another special dbm record created is the key. If this key exists in an NIS map, will only allow privileged processes to access the data within the map (that is, applications that can create reserved ports). Create a special dbm record with the key and the value nis_output_name. Create a special dbm record with the key and the value nis_domain_name. Replace the value of the special dbm record whose key is with nis_master_name. Undo the database_name (that is, write the contents of database_name to the standard output) one dbm record per line. A single space sepa- rates each key from its value. EXAMPLES
Shell scripts can be written to convert ASCII files such as to the key-value form used by For example, #!/usr/bin/sh /usr/bin/awk 'BEGIN { FS = ":" } { print $1, $0 }' /etc/netgroup | makedbm - netgroup converts the file to a form that is read by to make the NIS map The keys in the database are names, and the values are the remainders of the lines in the file. AUTHOR
was developed by Sun Microsystems, Inc. SEE ALSO
domainname(1), named(1M), ypinit(1M), ypmake(1M), yppoll(1M), gethostname(2), netgroup(4), ypfiles(4). makedbm(1M)
All times are GMT -4. The time now is 05:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy