Sponsored Content
Top Forums Shell Programming and Scripting Replace duplicate columns with values from first occurrence Post 302695523 by asyed on Monday 3rd of September 2012 05:18:32 AM
Old 09-03-2012
Replace duplicate columns with values from first occurrence

I've a text file with below values viz. multiple rows with same values in column 3, 4 and 5, which need to be considered as duplicates. For all such cases, the rows from second occurrence onwards should be modified in a way that their values in first two columns are replaced with values as in first occurrence. See the sample input and output copied below

Input file:
Code:
AAA,AAA,1,2,99,981,901,11,14
AAA,AAA,1,2,99,987,907,17,13
AAA,AAA,1,2,99,987,902,12,15
ABC,ABC,1,2,99,983,903,13,16
ABC,ABC,1,2,99,984,904,14,17
ABC,ABC,1,2,99,985,905,15,18
ABC,ABC,1,2,99,986,906,16,19

Output file:
Code:
AAA,AAA,1,2,99,981,901,11,14
AAA,AAA,1,2,99,987,907,17,13
AAA,AAA,1,2,99,987,902,12,15
AAA,AAA,1,2,99,983,903,13,16
AAA,AAA,1,2,99,984,904,14,17
AAA,AAA,1,2,99,985,905,15,18
AAA,AAA,1,2,99,986,906,16,19

Can you help me with a awk command to achieve this?

Last edited by Franklin52; 09-03-2012 at 06:25 AM.. Reason: Please use code tags for data and code samples
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Search and replace to first occurrence of string

Hi all, I have a very large; delimited file. In vi I would like to replace: CSACT_DY;AVG_UEACT1;uesPerActiveLinkSetSize_1;#;A CSACT_DY;AVG_UEACT2;uesPerActiveLinkSetSize_2;#;A CSACT_DY;AVG_UEACT3;uesPerActiveLinkSetSize_3;#;A with: CSACT_DY;AVG_UEACT1;Average... (7 Replies)
Discussion started by: gilmord
7 Replies

2. Shell Programming and Scripting

Replace second occurrence only

HPUX /bin/sh (posix) I have a file as such cat dog mouse deer elk rabbit mouse rat pig I would like to replace the second occurrence of mouse in this file with mouse2. The rest of the file has to stay exactly as is. I'm not sure exactly where mouse might be (could be first,second,third... (5 Replies)
Discussion started by: lyoncc
5 Replies

3. Shell Programming and Scripting

Find and replace duplicate column values in a row

I have file which as 12 columns and values like this 1,2,3,4,5 a,b,c,d,e b,c,a,e,f a,b,e,a,h if you see the first column has duplicate values, I need to identify (print it to console) the duplicate value (which is 'a') and also remove duplicate values like below. I could be in two... (5 Replies)
Discussion started by: nuthalapati
5 Replies

4. UNIX for Dummies Questions & Answers

How to replace particular occurrence of character in between a delimiter?

Hi, Hi, I have a file with following format 1|" "text " around " |" fire "guest"|" " 2| "xyz"" | "no guest"|"3" 3| """ test3""| "one" guest"|"4" My requirement is to replace all occurrences of " to ' which are occurring between |" "|delimiter so my output should look like this 1|"... (3 Replies)
Discussion started by: H_bansal
3 Replies

5. Programming

awk to count occurrence of strings and loop for multiple columns

Hi all, If i would like to process a file input as below: col1 col2 col3 ...col100 1 A C E A ... 3 D E G A 5 T T A A 6 D C A G how can i perform a for loop to count the occurences of letters in each column? (just like uniq -c ) in every column. on top of that, i would also like... (8 Replies)
Discussion started by: iling14
8 Replies

6. Shell Programming and Scripting

Replace values in columns

I have the following input format: AA00000712000 -0.17 0.90 -1.04 -0.37 -1.45 -1.13 -0.22 -0.34 -0.55 2.37 0.67 -0.48 -0.48 AA00000712001 0.15 0.03 0.47 0.62 2.04 1.14 0.29 -0.81 0.85 0.53 1.00 -0.10 -0.48 BB00000712000 1.32 -0.47 0.44 0.00 0.98 ... (4 Replies)
Discussion started by: ncwxpanther
4 Replies

7. Shell Programming and Scripting

Multiple columns replace with null values.

I am trying to replace the partcular columns(Col3,col5,col20,col44,col55,co56,col59,col60,col61,col62,col74,col75,col88,col90,col91,col93,col94,col95) with empty Input file Col1,col2,col3,col4,col5------,col100 1,2,3,4,5,---------,100 3,4,5,6,7,---------,300 Output : ... (3 Replies)
Discussion started by: onesuri
3 Replies

8. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

9. Shell Programming and Scripting

Do replace operation and awk to sum multiple columns if another column has duplicate values

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (12 Replies)
Discussion started by: as7951
12 Replies

10. Shell Programming and Scripting

Match value in two files and replace values in selected columns

The purpose is to check if values for column 3 and 4 in file1 match with column 1 in file2. If any value match do: 1) Replace values in file2 for column 2 and 3 using the information of file1 columns 5 and 6 2) Replace string ($1,1,5) and string ($1,6,5) in file2 with values of columns 7... (8 Replies)
Discussion started by: jiam912
8 Replies
funtablerowget(3)						SAORD Documentation						 funtablerowget(3)

NAME
FunTableRowGet - get Funtools rows SYNOPSIS
#include <funtools.h> void *FunTableRowGet(Fun fun, void *rows, int maxrow, char *plist, int *nrow) DESCRIPTION
The FunTableRowGet() routine retrieves rows from a Funtools binary table or raw event file, and places the values of columns selected by FunColumnSelect() into an array of user structs. Selected column values are automatically converted to the specified user data type (and to native data format) as necessary. The first argument is the Fun handle associated with this row data. The second rows argument is the array of user structs into which the selected columns will be stored. If NULL is passed, the routine will automatically allocate space for this array. (This includes proper allocation of pointers within each struct, if the "@" pointer type is used in the selection of columns. Note that if you pass NULL in the second argument, you should free this space using the standard free() system call when you are finished with the array of rows.) The third maxrow argument specifies the maximum number of rows to be returned. Thus, if rows is allocated by the user, it should be at least of size maxrow*sizeof(evstruct). The fourth plist argument is a param list string. Currently, the keyword/value pair "mask=transparent" is supported in the plist argument. If this string is passed in the call's plist argument, then all rows are passed back to the user (instead of just rows passing the filter). This is only useful when FunColumnSelect() also is used to specify "$region" as a column to return for each row. In such a case, rows found within a region have a returned region value greater than 0 (corresponding to the region id of the region in which they are located), rows passing the filter but not in a region have region value of -1, and rows not passing any filter have region value of 0. Thus, using "mask=transparent" and the returned region value, a program can process all rows and decide on an action based on whether a given row passed the filter or not. The final argument is a pointer to an int variable that will return the actual number of rows returned. The routine returns a pointer to the array of stored rows, or NULL if there was an error. (This pointer will be the same as the second argument, if the latter is non-NULL). /* get rows -- let routine allocate the row array */ while( (buf = (Ev)FunTableRowGet(fun, NULL, MAXROW, NULL, &got)) ){ /* process all rows */ for(i=0; i<got; i++){ /* point to the i'th row */ ev = buf+i; /* rearrange some values. etc. */ ev->energy = (ev->pi+ev->pha)/2.0; ev->pha = -ev->pha; ev->pi = -ev->pi; } /* write out this batch of rows */ FunTableRowPut(fun2, buf, got, 0, NULL); /* free row data */ if( buf ) free(buf); } As shown above, successive calls to FunTableRowGet() will return the next set of rows from the input file until all rows have been read, i.e., the routine behaves like sequential Unix I/O calls such as fread(). See evmerge example code for a more complete example. Note that FunTableRowGet() also can be called as FunEventsGet(), for backward compatibility. SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funtablerowget(3)
All times are GMT -4. The time now is 08:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy