Sponsored Content
Top Forums Shell Programming and Scripting Replace duplicate columns with values from first occurrence Post 302695523 by asyed on Monday 3rd of September 2012 05:18:32 AM
Old 09-03-2012
Replace duplicate columns with values from first occurrence

I've a text file with below values viz. multiple rows with same values in column 3, 4 and 5, which need to be considered as duplicates. For all such cases, the rows from second occurrence onwards should be modified in a way that their values in first two columns are replaced with values as in first occurrence. See the sample input and output copied below

Input file:
Code:
AAA,AAA,1,2,99,981,901,11,14
AAA,AAA,1,2,99,987,907,17,13
AAA,AAA,1,2,99,987,902,12,15
ABC,ABC,1,2,99,983,903,13,16
ABC,ABC,1,2,99,984,904,14,17
ABC,ABC,1,2,99,985,905,15,18
ABC,ABC,1,2,99,986,906,16,19

Output file:
Code:
AAA,AAA,1,2,99,981,901,11,14
AAA,AAA,1,2,99,987,907,17,13
AAA,AAA,1,2,99,987,902,12,15
AAA,AAA,1,2,99,983,903,13,16
AAA,AAA,1,2,99,984,904,14,17
AAA,AAA,1,2,99,985,905,15,18
AAA,AAA,1,2,99,986,906,16,19

Can you help me with a awk command to achieve this?

Last edited by Franklin52; 09-03-2012 at 06:25 AM.. Reason: Please use code tags for data and code samples
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Search and replace to first occurrence of string

Hi all, I have a very large; delimited file. In vi I would like to replace: CSACT_DY;AVG_UEACT1;uesPerActiveLinkSetSize_1;#;A CSACT_DY;AVG_UEACT2;uesPerActiveLinkSetSize_2;#;A CSACT_DY;AVG_UEACT3;uesPerActiveLinkSetSize_3;#;A with: CSACT_DY;AVG_UEACT1;Average... (7 Replies)
Discussion started by: gilmord
7 Replies

2. Shell Programming and Scripting

Replace second occurrence only

HPUX /bin/sh (posix) I have a file as such cat dog mouse deer elk rabbit mouse rat pig I would like to replace the second occurrence of mouse in this file with mouse2. The rest of the file has to stay exactly as is. I'm not sure exactly where mouse might be (could be first,second,third... (5 Replies)
Discussion started by: lyoncc
5 Replies

3. Shell Programming and Scripting

Find and replace duplicate column values in a row

I have file which as 12 columns and values like this 1,2,3,4,5 a,b,c,d,e b,c,a,e,f a,b,e,a,h if you see the first column has duplicate values, I need to identify (print it to console) the duplicate value (which is 'a') and also remove duplicate values like below. I could be in two... (5 Replies)
Discussion started by: nuthalapati
5 Replies

4. UNIX for Dummies Questions & Answers

How to replace particular occurrence of character in between a delimiter?

Hi, Hi, I have a file with following format 1|" "text " around " |" fire "guest"|" " 2| "xyz"" | "no guest"|"3" 3| """ test3""| "one" guest"|"4" My requirement is to replace all occurrences of " to ' which are occurring between |" "|delimiter so my output should look like this 1|"... (3 Replies)
Discussion started by: H_bansal
3 Replies

5. Programming

awk to count occurrence of strings and loop for multiple columns

Hi all, If i would like to process a file input as below: col1 col2 col3 ...col100 1 A C E A ... 3 D E G A 5 T T A A 6 D C A G how can i perform a for loop to count the occurences of letters in each column? (just like uniq -c ) in every column. on top of that, i would also like... (8 Replies)
Discussion started by: iling14
8 Replies

6. Shell Programming and Scripting

Replace values in columns

I have the following input format: AA00000712000 -0.17 0.90 -1.04 -0.37 -1.45 -1.13 -0.22 -0.34 -0.55 2.37 0.67 -0.48 -0.48 AA00000712001 0.15 0.03 0.47 0.62 2.04 1.14 0.29 -0.81 0.85 0.53 1.00 -0.10 -0.48 BB00000712000 1.32 -0.47 0.44 0.00 0.98 ... (4 Replies)
Discussion started by: ncwxpanther
4 Replies

7. Shell Programming and Scripting

Multiple columns replace with null values.

I am trying to replace the partcular columns(Col3,col5,col20,col44,col55,co56,col59,col60,col61,col62,col74,col75,col88,col90,col91,col93,col94,col95) with empty Input file Col1,col2,col3,col4,col5------,col100 1,2,3,4,5,---------,100 3,4,5,6,7,---------,300 Output : ... (3 Replies)
Discussion started by: onesuri
3 Replies

8. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

9. Shell Programming and Scripting

Do replace operation and awk to sum multiple columns if another column has duplicate values

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (12 Replies)
Discussion started by: as7951
12 Replies

10. Shell Programming and Scripting

Match value in two files and replace values in selected columns

The purpose is to check if values for column 3 and 4 in file1 match with column 1 in file2. If any value match do: 1) Replace values in file2 for column 2 and 3 using the information of file1 columns 5 and 6 2) Replace string ($1,1,5) and string ($1,6,5) in file2 with values of columns 7... (8 Replies)
Discussion started by: jiam912
8 Replies
Net::Ifconfig::Wrapper(3pm)				User Contributed Perl Documentation			       Net::Ifconfig::Wrapper(3pm)

NAME
Net::Ifconfig::Wrapper - provides a unified way to configure network interfaces on FreeBSD, OpenBSD, Solaris, Linux, OS X, and WinNT (from Win2K). Version 0.11 SYNOPSIS
#!/usr/local/bin/perl -w # uni-ifconfig.pl # The unified ifconfig command. # Works the same way on FreeBSD, OpenBSD, Solaris, Linux, OS X, WinNT (from Win2K). # Note: due of Net::Ifconfig::Wrapper limitations 'inet' and 'down' commands # are not working on WinNT. +/-alias are working, of course. use strict; use Net::Ifconfig::Wrapper; my $Usage = << 'EndOfText'; uni-ifconfig.pl # Print this notice uni-ifconfig.pl -a # Print info about all interfaces uni-ifconfig.pl <iface> # Print info obout specified interface uni-ifconfig.pl <iface> down # Bring specified interface down uni-ifconfig.pl <iface> inet <AAA.AAA.AAA.AAA> mask <MMM.MMM.MMM.MMM> # Set the specified address on the specified interface # and bring this interface up uni-ifconfig.pl <iface> inet <AAA.AAA.AAA.AAA> mask <MMM.MMM.MMM.MMM> [+]alias # Set the specified alias address # on the specified interface uni-ifconfig.pl <iface> inet <AAA.AAA.AAA.AAA> [mask <MMM.MMM.MMM.MMM>] -alias # Remove specified alias address # from the specified interface EndOfText my $Info = Net::Ifconfig::Wrapper::Ifconfig('list', '', '', '') or die $@; scalar(keys(%{$Info})) or die "No one interface found. Something wrong? "; if (!scalar(@ARGV)) { print $Usage; exit 0; } if ($ARGV[0] eq '-a') { defined($ARGV[1]) and die $Usage; foreach (sort(keys(%{$Info}))) { print IfaceInfo($Info, $_); }; exit 0; }; $Info->{$ARGV[0]} or die "Interface '$ARGV[0]' is unknown "; if (!defined($ARGV[1])) { print IfaceInfo($Info, $ARGV[0]); exit 0; } my $CmdLine = join(' ', @ARGV); my $Result = undef; if ($CmdLine =~ m/As*([w{}-]+)s+downs*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('down', $1, '', ''); } elsif ($CmdLine =~ m/As*([w{}-]+)s+inets+(d{1,3}(?:.d{1,3}){3})s+masks+(d{1,3}(?:.d{1,3}){3})s*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('inet', $1, $2, $3); } elsif ($CmdLine =~ m/As*([w{}-]+)s+inets+(d{1,3}(?:.d{1,3}){3})s+masks+(d{1,3}(?:.d{1,3}){3})s++?aliass*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('+alias', $1, $2, $3); } elsif ($CmdLine =~ m/As*([w{}-]+)s+inets+(d{1,3}(?:.d{1,3}){3})s+(:?masks+(d{1,3}(?:.d{1,3}){3})s+)?-aliass*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('-alias', $1, $2, ''); } else { die $Usage; }; $Result or die $@; exit 0; sub IfaceInfo { my ($Info, $Iface) = @_; my $Res = "$Iface: ".($Info->{$Iface}{'status'} ? 'UP' : 'DOWN')." "; while (my ($Addr, $Mask) = each(%{$Info->{$Iface}{'inet'}})) { $Res .= sprintf(" inet %-15s mask $Mask ", $Addr); }; $Info->{$Iface}{'ether'} and $Res .= " ether ".$Info->{$Iface}{'ether'}." "; $Info->{$Iface}{'descr'} and $Res .= " descr '".$Info->{$Iface}{'descr'}."' "; return $Res; }; DESCRIPTION
This module provides a unified way to configure the network interfaces on FreeBSD, OpenBSD, Solaris, Linux, OS X, and WinNT (from Win2K) systems. Only "inet" (IPv4) and "ether" (MAC) addresses are supported at the moment On Unixes this module calls the system "ifconfig" command to perform the actions. On Windows the functions from IpHlpAPI.DLL are called. For all supported Unixes "Net::Ifconfig::Wrapper" expect "ifconfig" command to be "/sbin/ifconfig". Module was tested on FreeBSD 4.7,4.8,5.3 (Intel), RedHat 6.2,7.3,8.0 (Intel), Win2000 Pro (Intel), OpenBSD 3.1 (SPARC), Solaris 7 (SPARC), OS X 10.3 (aka Panther), OS X 10.4 (aka Tiger). In MSWin32 family only WinNT is supported. In WinNT family only Win2K or later is supported. The Net::Ifconfig::Wrapper methods "Ifconfig(Command, Interface, Address, Netmask);" The first and the last method of the "Net::Ifconfig::Wrapper" module. Do all the job. The particular action is described by the $Command parameter. $Command could be: 'list' "Ifconfig('list', '', '', '')" will return the reference to the hash contains the information about interfaces. The structure of this hash is the following: {IfaceName => {'status' => 0|1 # The status of the interface. 0 means down, 1 means up 'ether' => MACaddr, # The ethernet address of the interface if available 'descr' => Description, # The description of the interface if available 'inet' => {IPaddr1 => NetMask, # The IP address and his netmask, both are in AAA.BBB.CCC.DDD notation IPaddr2 => NetMask, ... }, ... }; Interface, Address, Netmask parameters are ignored. The following programs are called: FreeBSD "/sbin/ifconfig -a" Solaris "/sbin/ifconfig -a" OpenBSD "/sbin/ifconfig -A" Linux "/sbin/ifconfig -a" OS X "/sbin/ifconfig -a" MSWin32 "GetAdaptersInfo" function from "IpHlpAPI.DLL" Limitations: OpenBSD: "/sbin/ifconfig -A" command is not returning information about MAC addresses so we are trying to get it from '/usr/sbin/arp -a' command (first 'static' entry). If no one present the 'ff:ff:ff:ff:ff' address is returned. MSWin32: "GetAdaptersInfo" function is not returning information about interface which have address 127.0.0.1 binded so "Net::Ifconfig::Wrapper" have no ability to display it. Not limitation but little problem: MSWin32 interface names are not human-readable, they looks like "{843C2077-30EC-4C56-A401-658BB1E42BC7}" (on Win2K at least). 'inet' This function is used to set IPv4 address on interface. It have to be called as Ifconfig('inet', $IfaceName, $Addr, $Mask); $IfaceName is an interface name as displayed by 'list' command $Addr is an IPv4 address in the "AAA.AAA.AAA.AAA" notation $Mask is an IPv4 subnet mask in the "MMM.MMM.MMM.MMM" notation The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" Solaris "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" OpenBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" Linux "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" OS X "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" MSWin32: nothing :( Limitations: MSWin32: I did not find the relaible way to recognize the "main" address on the Win32 network interface, so I have disabled this functionality. If you know the way please let me know. 'up' Just a synonym for 'inet' 'down' This function is used to bring specified interface down. It have to be called as Ifconfig('inet', $IfaceName, '', ''); $IfaceName is an interface name as displayed by 'list' command Address and Netmask are ignored. The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% down" Solaris "/sbin/ifconfig %Iface% down" OpenBSD "/sbin/ifconfig %Iface% down" Linux "/sbin/ifconfig %Iface% down" OS X "/sbin/ifconfig %Iface% down" MSWin32 nothing :( Limitations: MSWin32: I did not find the way to implement the 'up' command so I did not implement 'down'. '+alias' This function is used to set IPv4 alias address on interface. It have to be called as Ifconfig('+alias', $IfaceName, $Addr, $Mask); $IfaceName is an interface name as displayed by 'list' command $Addr is an IPv4 address in the "AAA.AAA.AAA.AAA" notation $Mask is an IPv4 subnet mask in the "MMM.MMM.MMM.MMM" notation The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% alias" Solaris "/sbin/ifconfig %Iface%:%Logic% inet %Addr% netmask %Mask% up" OpenBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% alias" Linux "/sbin/ifconfig %Iface%:%Logic% inet %Addr% netmask %Mask% up" OS X "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% alias" MSWin32 "AddIPAddress" function from "IpHlpAPI.DLL" First available logic interface is taken automatically for Solaris and Linux 'alias' Just a synonim for '+alias' '-alias' This function is used to remove IPv4 alias address from interface. It have to be called as Ifconfig('-alias', $IfaceName, $Addr, ''); $IfaceName is an interface name as displayed by 'list' command $Addr is an IPv4 address in the "AAA.AAA.AAA.AAA" notation Netmask> parameter is ignored The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% inet %Addr% -alias" Solaris "/sbin/ifconfig %Iface%:%Logic% down" OpenBSD "/sbin/ifconfig %Iface% inet %Addr% -alias" Linux "/sbin/ifconfig %Iface%:%Logic% down" OS X "/sbin/ifconfig %Iface% inet %Addr% -alias" MSWin32 "DeleteIPAddress" function from "IpHlpAPI.DLL" Appropriate logic interface is obtained automatically for Solaris and Linux On success "Ifconfig(...)" returns the defined value. Actually, it is a reference to the array contains the output of the actual "ifconfig" program called. In case of troubles "Ifconfig(...)" returns 'undef' value, $@ variable contains the error message. EXPORT None by default. AUTHOR
Daniel Podolsky, <tpaba@cpan.org> SEE ALSO
ifconfig(8), Internet Protocol Helper in Platform SDK. perl v5.14.2 2012-01-19 Net::Ifconfig::Wrapper(3pm)
All times are GMT -4. The time now is 01:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy