Sponsored Content
Top Forums Shell Programming and Scripting awk - Remove duplicates during array build Post 302976721 by RudiC on Monday 4th of July 2016 01:42:41 PM
Old 07-04-2016
Make (and test) the combination of $1, $2, and $3 unique:
Code:
awk  '
FNR == 1        {next
                }
!T[$1,$2,$3]++  {v_array[$1 OFS $2]=v_array[$1 OFS $2] ? v_array[$1 OFS $2] "," $3 : $3
                }
END             {for (v in v_array) print v, v_array[v]
                }
' file
abc def xyz,qrs
abc efg pqr
stu vwx yz

While some awk implementations provide a sort function, and you could build one yourself in others, piping through sort might be the easiest way to get what you want:

Code:
. . . | sort -k3
abc efg pqr
abc def xyz,qrs
stu vwx yz

This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Arranging an array so that duplicates will turn up first

Hi All, I have an array that contains duplicates as well unique numbers. ex- (21, 33, 35, 21, 33, 70, 33, 35, 50) I need to arrange it in such a way that all the duplicates will come up first followed by unique numbers. Result for the given example should be: (21, 21, 33, 33, 35, 35, 70,... (4 Replies)
Discussion started by: ashim
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

4. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates in C shell Array?

Please help me on this My script name is uniqueArray.csh #!/bin/csh set ARRAY = ( one teo three one three ) set ARRAY = ( $ARRAY one five three five ) How to remove the duplicates in this array ,sort and save those in the same variable or different variable. Thanks in the advance ... (3 Replies)
Discussion started by: SA_Palani
3 Replies
readproctitle(8)					      System Manager's Manual						  readproctitle(8)

NAME
readproctitle - maintains an automatically rotated log in memory for inspection by ps(1). readproctitle is available in daemontools 0.75 and above. SYNOPSIS
readproctitle L D DESCRIPTION
L consists of any number of arguments. D is one argument consisting of at least five dots. readproctitle reads data into the end of D, shifting D to the left to make room. This means that the most recent data is visible to process-listing tools such as ps(1). readproctitle always leaves three dots at the left of D. For example, if readproctitle io errors: .................... reads the data fatal error xyz warning abc then its command-line arguments change to readproctitle io errors: ... xyz!warning abc! with a newline character in place of each !. Process-listing tools typically show the newline character as ? or . readproctitle exits when it reaches the end of input. Beware that most implementations of ps(1) have small argument-length limits. These limits apply to the total length of readproctitle L D. I have not seen a system with a limit below 512 bytes. SEE ALSO
supervise(8), svc(8), svok(8), svstat(8), svscanboot(8), svscan(8), fghack(8), pgrphack(8), multilog(8), tai64n(8), tai64nlocal(8), setu- idgid(8), envuidgid(8), envdir(8), softlimit(8), setlock(8), ps(1) http://cr.yp.to/daemontools.html http://cr.yp.to/slashcommand.html readproctitle(8)
All times are GMT -4. The time now is 03:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy