Sponsored Content
Top Forums Shell Programming and Scripting awk - Remove duplicates during array build Post 302976721 by RudiC on Monday 4th of July 2016 01:42:41 PM
Old 07-04-2016
Make (and test) the combination of $1, $2, and $3 unique:
Code:
awk  '
FNR == 1        {next
                }
!T[$1,$2,$3]++  {v_array[$1 OFS $2]=v_array[$1 OFS $2] ? v_array[$1 OFS $2] "," $3 : $3
                }
END             {for (v in v_array) print v, v_array[v]
                }
' file
abc def xyz,qrs
abc efg pqr
stu vwx yz

While some awk implementations provide a sort function, and you could build one yourself in others, piping through sort might be the easiest way to get what you want:

Code:
. . . | sort -k3
abc efg pqr
abc def xyz,qrs
stu vwx yz

This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Arranging an array so that duplicates will turn up first

Hi All, I have an array that contains duplicates as well unique numbers. ex- (21, 33, 35, 21, 33, 70, 33, 35, 50) I need to arrange it in such a way that all the duplicates will come up first followed by unique numbers. Result for the given example should be: (21, 21, 33, 33, 35, 35, 70,... (4 Replies)
Discussion started by: ashim
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

4. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates in C shell Array?

Please help me on this My script name is uniqueArray.csh #!/bin/csh set ARRAY = ( one teo three one three ) set ARRAY = ( $ARRAY one five three five ) How to remove the duplicates in this array ,sort and save those in the same variable or different variable. Thanks in the advance ... (3 Replies)
Discussion started by: SA_Palani
3 Replies
subst(n)						       Tcl Built-In Commands							  subst(n)

__________________________________________________________________________________________________________________________________________________

NAME
subst - Perform backslash, command, and variable substitutions SYNOPSIS
subst ?-nobackslashes? ?-nocommands? ?-novariables? string _________________________________________________________________ DESCRIPTION
This command performs variable substitutions, command substitutions, and backslash substitutions on its string argument and returns the fully-substituted result. The substitutions are performed in exactly the same way as for Tcl commands. As a result, the string argument is actually substituted twice, once by the Tcl parser in the usual fashion for Tcl commands, and again by the subst command. If any of the -nobackslashes, -nocommands, or -novariables are specified, then the corresponding substitutions are not performed. For example, if -nocommands is specified, command substitution is not performed: open and close brackets are treated as ordinary characters with no special interpretation. Note that the substitution of one kind can include substitution of other kinds. For example, even when the -novariables option is speci- fied, command substitution is performed without restriction. This means that any variable substitution necessary to complete the command substitution will still take place. Likewise, any command substitution necessary to complete a variable substitution will take place, even when -nocommands is specified. See the EXAMPLES below. If an error occurs during substitution, then subst will return that error. If a break exception occurs during command or variable substi- tution, the result of the whole substitution will be the string (as substituted) up to the start of the substitution that raised the excep- tion. If a continue exception occurs during the evaluation of a command or variable substitution, an empty string will be substituted for that entire command or variable substitution (as long as it is well-formed Tcl.) If a return exception occurs, or any other return code is returned during command or variable substitution, then the returned value is substituted for that substitution. See the EXAMPLES below. In this way, all exceptional return codes are "caught" by subst. The subst command itself will either return an error, or will complete successfully. EXAMPLES
When it performs its substitutions, subst does not give any special treatment to double quotes or curly braces (except within command sub- stitutions) so the script set a 44 subst {xyz {$a}} returns "xyz {44}", not "xyz {$a}" and the script set a "p} q {r" subst {xyz {$a}} returns "xyz {p} q {r}", not "xyz {p} q {r}". When command substitution is performed, it includes any variable substitution necessary to evaluate the script. set a 44 subst -novariables {$a [format $a]} returns "$a 44", not "$a $a". Similarly, when variable substitution is performed, it includes any command substitution necessary to retrieve the value of the variable. proc b {} {return c} array set a {c c [b] tricky} subst -nocommands {[b] $a([b])} returns "[b] c", not "[b] tricky". The continue and break exceptions allow command substitutions to prevent substitution of the rest of the command substitution and the rest of string respectively, giving script authors more options when processing text using subst. For example, the script subst {abc,[break],def} returns "abc,", not "abc,,def" and the script subst {abc,[continue;expr {1+2}],def} returns "abc,,def", not "abc,3,def". Other exceptional return codes substitute the returned value subst {abc,[return foo;expr {1+2}],def} returns "abc,foo,def", not "abc,3,def" and subst {abc,[return -code 10 foo;expr {1+2}],def} also returns "abc,foo,def", not "abc,3,def". SEE ALSO
Tcl(n), eval(n), break(n), continue(n) KEYWORDS
backslash substitution, command substitution, variable substitution Tcl 7.4 subst(n)
All times are GMT -4. The time now is 05:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy