Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate values in a column(not in the file) Post 302980004 by Don Cragun on Monday 22nd of August 2016 06:10:08 PM
Old 08-22-2016
Does the order of the resulting strings in fields 4 and 5 matter?
In other words, does it matter if the last line of the output shown above would be:
Code:
 twe|tez|555|dddddd,agentcode=sample3 code ddddddddd|agentadd=tttt stereet ttttt

instead of:
Code:
 twe|tez|555|agentcode=sample3 code ddddddddd,dddddd|agentadd=tttt stereet ttttt

It is easier and faster if the output order can be random; but it isn't hard to keep the input order if it matters.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

2. UNIX for Dummies Questions & Answers

[SOLVED] remove lines that have duplicate values in column two

Hi, I've got a file that I'd like to uniquely sort based on column 2 (values in column 2 begin with "comp"). I tried sort -t -nuk2,3 file.txtBut got: sort: multi-character tab `-nuk2,3' "man sort" did not help me out Any pointers? Input: Output: (5 Replies)
Discussion started by: pathunkathunk
5 Replies

3. Shell Programming and Scripting

Check to identify duplicate values at first column in csv file

Hello experts, I have a requirement where I have to implement two checks on a csv file: 1. Check to see if the value in first column is duplicate, if any value is duplicate script should exit. 2. Check to verify if the value at second column is between "yes" or "no", if it is anything else... (4 Replies)
Discussion started by: avikaljain
4 Replies

4. Shell Programming and Scripting

Remove the values from a certain column without deleting the Column name in a .CSV file

(14 Replies)
Discussion started by: dhruuv369
14 Replies

5. Shell Programming and Scripting

Get the average from column, and eliminate the duplicate values.

Dear Experts, Kindly help me please, I have a big file where there is duplicate values in col 11 till col 23, every 2 rows appers a new numbers, but in each row there is different coordinates x and y in col 57 till col 74. Please i will like to get a single value and average of the x and y... (8 Replies)
Discussion started by: jiam912
8 Replies

6. Shell Programming and Scripting

Remove duplicate values with condition

Hi Gents, Please can you help me to get the desired output . In the first column I have some duplicate records, The condition is that all need to reject the duplicate record keeping the last occurrence. But the condition is. If the last occurrence is equal to value 14 or 98 in column 3 and... (2 Replies)
Discussion started by: jiam912
2 Replies

7. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

8. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

9. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies
Perl::Critic::Policy::ControlStructures::ProhibitUnreachUserCContributed Perl DPerl::Critic::Policy::ControlStructures::ProhibitUnreachableCode(3)

NAME
Perl::Critic::Policy::ControlStructures::ProhibitUnreachableCode - Don't write code after an unconditional "die, exit, or next". AFFILIATION
This Policy is part of the core Perl::Critic distribution. DESCRIPTION
This policy prohibits code following a statement which unconditionally alters the program flow. This includes calls to "exit", "die", "return", "next", "last" and "goto". Due to common usage, "croak" and "confess" from Carp are also included. Code is reachable if any of the following conditions are true: o Flow-altering statement has a conditional attached to it o Statement is on the right side of an operator "&&", "||", "//", "and", "or", or "err". o Code is prefixed with a label (can potentially be reached via "goto") o Code is a subroutine EXAMPLES
# not ok exit; print "123 "; # ok exit if !$xyz; print "123 "; # not ok for ( 1 .. 10 ) { next; print 1; } # ok for ( 1 .. 10 ) { next if $_ == 5; print 1; } # not ok sub foo { my $bar = shift; return; print 1; } # ok sub foo { my $bar = shift; return if $bar->baz(); print 1; } # not ok die; print "123 "; # ok die; LABEL: print "123 "; # not ok croak; do_something(); # ok croak; sub do_something {} CONFIGURATION
This Policy is not configurable except for the standard options. SEE ALSO
Perl::Critic::Policy::ControlStructures::ProhibitPostfixControls AUTHOR
Peter Guzis <pguzis@cpan.org> COPYRIGHT
Copyright (c) 2006-2011 Peter Guzis. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of this license can be found in the LICENSE file included with this module. perl v5.16.3 2014-06-09 Perl::Critic::Policy::ControlStructures::ProhibitUnreachableCode(3)
All times are GMT -4. The time now is 08:08 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy