Sponsored Content
Top Forums Shell Programming and Scripting awk script to find duplicate values Post 302910712 by RudiC on Sunday 27th of July 2014 03:30:57 PM
Old 07-27-2014
Try also
Code:
awk     'NR==1  {print; next}
         function Z (P) {X=$2","$3","P; T[X]=T[X] D[X] sprintf ("%s|%s|%s|%s|%s|", $1, $2, $3, $4, P); C[X]++; D[X]="\n"}
                {Z ($5)}
         NF>6   {Z ($6)}
         END    {for (i in T) if (C[i]>1) print T[i]}
        ' FS="[,|]" file
ID|Class|SubClass|Prop|Value
2|XYZ|MNF|PXM1|456|
2|XYZ|MNF|PXM4|456|
1|ABC|DEF|PAD1|123|
1|ABC|DEF|PAD4|123|
3|ABC|DEF|PAD2|123|
1|ABC|DEF|PAD2|234|
3|ABC|DEF|PAD1|234|

 

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find and replace duplicate column values in a row

I have file which as 12 columns and values like this 1,2,3,4,5 a,b,c,d,e b,c,a,e,f a,b,e,a,h if you see the first column has duplicate values, I need to identify (print it to console) the duplicate value (which is 'a') and also remove duplicate values like below. I could be in two... (5 Replies)
Discussion started by: nuthalapati
5 Replies

2. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

3. Shell Programming and Scripting

[Solved] Find duplicate and add pattern in sed/awk

<Update> I have the solution: sed 's/\{3\}/&;&;---;4/' The thread can be marked as solved! </Update> Hi There, I'm working on a script processing some data from a website into cvs format. There is only one final problem left I can't find a solution. I've processed my file... (0 Replies)
Discussion started by: lolworlds
0 Replies

4. Shell Programming and Scripting

How to find the X highest values in a list depending on the values of another list with bash/awk?

Hi everyone, This is an exemple of inpout.txt file (a "," delimited text file which can be open as csv file): ID, Code, Value, Store SP|01, AABBCDE, 15, 3 SP|01, AABBCDE, 14, 2 SP|01, AABBCDF, 13, 2 SP|01, AABBCDE, 16, 3 SP|02, AABBCED, 15, 2 SP|01, AABBCDF, 12, 3 SP|01, AABBCDD,... (1 Reply)
Discussion started by: jeremy589
1 Replies

5. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

6. Shell Programming and Scripting

Sum duplicate values in text file through awk between dates

I need to sum values in text file in case duplicate row are present with same name and different value below is example of data in file i have and format i need. Data in text file 20170308 PM,U,2 PM,U,113 PM,I,123 DA,U,135 DA,I,113 DA,I,1 20170309 PM,U,2 PM,U,1 PM,I,123 PM,I,1... (3 Replies)
Discussion started by: Adfire
3 Replies

7. Shell Programming and Scripting

Do replace operation and awk to sum multiple columns if another column has duplicate values

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (12 Replies)
Discussion started by: as7951
12 Replies

8. UNIX for Beginners Questions & Answers

Find lines with duplicate values in a particular column

I have a file with 5 columns. I want to pull out all records where the value in column 4 is not unique. For example in the sample below, I would want it to print out all lines except for the last two. 40991764 2419 724 47182 Cand A 40992936 3591 724 47182 Cand B 40993016 3671 724 47182 Cand C... (5 Replies)
Discussion started by: kaktus
5 Replies
Class::Accessor::Fast(3pm)				User Contributed Perl Documentation				Class::Accessor::Fast(3pm)

NAME
Class::Accessor::Fast - Faster, but less expandable, accessors SYNOPSIS
package Foo; use base qw(Class::Accessor::Fast); # The rest is the same as Class::Accessor but without set() and get(). DESCRIPTION
This is a faster but less expandable version of Class::Accessor. Class::Accessor's generated accessors require two method calls to accompish their task (one for the accessor, another for get() or set()). Class::Accessor::Fast eliminates calling set()/get() and does the access itself, resulting in a somewhat faster accessor. The downside is that you can't easily alter the behavior of your accessors, nor can your subclasses. Of course, should you need this later, you can always swap out Class::Accessor::Fast for Class::Accessor. Read the documentation for Class::Accessor for more info. EFFICIENCY
"EFFICIENCY" in Class::Accessor for an efficiency comparison. AUTHORS
Copyright 2007 Marty Pauley <marty+perl@kasei.com> This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. That means either (a) the GNU General Public License or (b) the Artistic License. ORIGINAL AUTHOR Michael G Schwern <schwern@pobox.com> SEE ALSO
Class::Accessor perl v5.10.0 2009-09-15 Class::Accessor::Fast(3pm)
All times are GMT -4. The time now is 07:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy