Adding Column Values Using Pattern Match


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Adding Column Values Using Pattern Match
# 1  
Old 03-22-2013
Adding Column Values Using Pattern Match

Hi All,

I have a file with data as below:

HTML Code:
A,FILE1_MYFILE_20130309_1038,80,25.60
B,FILE1_MYFILE_20130309_1038,24290,18543.38
C,FILE1_dsc_dlk_MYFILE_20130309_1038,3,10.10
A,FILE2_MYFILE_20130310_1039,85,110.10
B,FILE2_MYFILE_20130310_1039,10,12.10
C,FILE2_err_dlk_MYFILE_20130310_1039,3,10.10
I am using following command to sum values of 3 column based on the value in second column.
HTML Code:
for i in `cat OUTPUT_FILE|awk -F"," '{print $2}'|sort -u`;[U]do grep $i [/U]OUTPUT_FILE|awk -F"," '{c+=$3}END{print $2".edr""|"c}';done
However the above command will definitely exclude third row as the value of second column in third row is not matching with that for other two rows. However, I want to include the third rwo as well. Hence I need to perform a pattern matching in the underlined part so that my output looks as below:

HTML Code:
FILE1_MYFILE_20130309_1038,24373
FILE2_MYFILE_20130310_1039,98
How do I perform a pattern matching here.

Thanks and Regards
Angshuman
# 2  
Old 03-22-2013
One way:

Code:
awk -F, '{x=$2;sub(/_.*/,"",x);if(!a[x])a[x]=$2;b[x]+=$3;}END{for (i in a)print a[i]","b[i];}' file

Guru.
# 3  
Old 03-22-2013
Hi Guru,

Thank you for your reply.

Why is it not working agaisnt following data set. Can you please explain a bit on how is the sub function working in awk.

Code:
SUSPREL,ICP_MED_DEL_SEM_20130309_1038,80,25.60
REL,ICP_MED_DEL_SEM_20130309_1038,24290,18543.38
ERROR_ALLRATE_DSC_DLK,ICP_dsc_dlk_MED_DEL_SEM_20130309_1038,3,10.10
SUSPREL,ICP_MED_DEL_SEM_20130309_1039,80,25.60
REL,ICP_MED_DEL_SEM_20130309_1039,24290,18543.38
ERROR_ALLRATE_DSC_DLK,ICP_dsc_dlk_MED_DEL_SEM_20130309_1039,3,10.10

Another point is that the data between ICP and MED can be of any leth.

---------- Post updated at 01:15 PM ---------- Previous update was at 12:30 PM ----------

Hi Rudic,

Thank you for your reply. However, the output that you have showed is not what I am expecting

Let me explain a bit more. If you check the value in column 2, you will see that the values are like FILE1_MYFILE_20130309_1038, FILE1_MYFILE_20130309_1038, FILE1_dsc_dlk_MYFILE_20130309_1038. These are actually from the same group. The only difference is that there are some additional values between FILE1 and MYFILE. Hence I want to add all the corresponsing column 3 values. If this is not possible in awk, I was thinking if I remove the values between FILE1 and MYFILE first using awk and then add the values in column 3. What is your input on that and how do I remove those values?

Thanks and Regards
Angshuman
# 4  
Old 03-22-2013
A crude approach

Code:
awk -F"," '{a["ICP_"substr($2,index($2,"MED"))]+=$3} END{ for(i in a) print i,a[i]}' OFS="," file


Last edited by panyam; 03-22-2013 at 05:39 AM.. Reason: Added new code
# 5  
Old 03-22-2013
Quote:
Originally Posted by angshuman
. . .
Hi Rudic,

Thank you for your reply. However, the output that you have showed is not what I am expecting

Let me explain a bit more. If you check the value in column 2, you will see that the values are like FILE1_MYFILE_20130309_1038, FILE1_MYFILE_20130309_1038, FILE1_dsc_dlk_MYFILE_20130309_1038. These are actually from the same group. The only difference is that there are some additional values between FILE1 and MYFILE. Hence I want to add all the corresponsing column 3 values. If this is not possible in awk, I was thinking if I remove the values between FILE1 and MYFILE first using awk and then add the values in column 3. What is your input on that and how do I remove those values?

Thanks and Regards
Angshuman
Hi Angshuman,

I figured that out and deleted my post, as it was irrelevant. In order to do what you want, we need to know what parts of the file names are to be retained, either by counting the subfields between separators (here: "_"), or by identifying static substrings (here: FILEn and MYFILE). If you can't give any hint on anchoring the to-be-deleted substring, things become difficult.
# 6  
Old 03-22-2013
Hi Panyam,

Sorry your code does not provide the output that I am expecting. I get the following output:

Code:
MED_DEL_SEM_20130309_1038 24373
MED_DEL_SEM_20130309_1039 24373

and my expected result is

Code:
ICP_MED_DEL_SEM_20130309_1038,24373
ICP_MED_DEL_SEM_20130309_1039,24373

# 7  
Old 03-22-2013
Quote:
Originally Posted by angshuman
Hi Panyam,

Sorry your code does not provide the output that I am expecting. I get the following output:

Code:
MED_DEL_SEM_20130309_1038 24373
MED_DEL_SEM_20130309_1039 24373

and my expected result is

Code:
ICP_MED_DEL_SEM_20130309_1038,24373
ICP_MED_DEL_SEM_20130309_1039,24373

Check my earlier post now. I edited it.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If pattern match in other column, modify column 3.

My command sed will modify everything in column 3 if i will use the command below. I want to search for a pattern then modify everything in column 3. sed -i 's/\|165\|/server1/g' file.txt Input: 01-31-2019 19:14:05|device|165|1548962040165|5c5348f9-0804-1111|file_attach|7271|587|smtp|... (6 Replies)
Discussion started by: invinzin21
6 Replies

2. UNIX for Dummies Questions & Answers

Match sum of values in each column with the corresponding column value present in trailer record

Hi All, I have a requirement where I need to find sum of values from column D through O present in a CSV file and check whether the sum of each Individual column matches with the value present for that corresponding column present in the trailer record. For example, let's assume for column D... (9 Replies)
Discussion started by: tpk
9 Replies

3. Shell Programming and Scripting

Adding values of a column based on another column

Hello, I have a data such as this: ENSGALG00000000189 329 G A 4 2 0 ENSGALG00000000189 518 T C 5 1 0 ENSGALG00000000189 1104 G A 5 1 0 ENSGALG00000000187 3687 G T 5 1 0 ENSGALG00000000187 4533 A T 4 2 0 ENSGALG00000000233 5811 T C 4 2 0 ENSGALG00000000233 5998 C A 5 1 0 I want to... (3 Replies)
Discussion started by: Homa
3 Replies

4. Shell Programming and Scripting

awk Print New Column For Every Two Lines and Match On Multiple Column Values to print another column

Hi, My input files is like this axis1 0 1 10 axis2 0 1 5 axis1 1 2 -4 axis2 2 3 -3 axis1 3 4 5 axis2 3 4 -1 axis1 4 5 -6 axis2 4 5 1 Now, these are my following tasks 1. Print a first column for every two rows that has the same value followed by a string. 2. Match on the... (3 Replies)
Discussion started by: jacobs.smith
3 Replies

5. Shell Programming and Scripting

Adding of two column values

Hi cat /tmp/xx.txt 1 4 1 5 1 6 2 1 2 1 2 1 i want to add the values of 2nd column resepect to 1st column values..for 1 in 1st column i need sum of all the values in 2nd column ..pls tell me hw to do it?? (8 Replies)
Discussion started by: Aditya.Gurgaon
8 Replies

6. Shell Programming and Scripting

Adding column values in a file

Hi, I am having a file in the following format. for aaaa 1111 1234 2222 3434 for bbbb 1111 3434.343 2222 2343 for cccc 3333 2343.343 4444 89000 for dddd 1111 5678.343 2222 890.3 aaaa 2343.343 bbbb 34343.343 (5 Replies)
Discussion started by: jpkumar10
5 Replies

7. Shell Programming and Scripting

Compare values for a pattern match and execute script

Here in the input file 23:59:13,devicename,21,server1,700 23:59:13,devicename,22,server2,200 23:59:13,devicename,23,server3,200 23:59:13,devicename,24,server4,200 23:59:13,devicename,25,server5,200 23:59:13,devicename,26,server6,200 23:59:13,devicename,27,server7,200... (6 Replies)
Discussion started by: necro98
6 Replies

8. UNIX for Dummies Questions & Answers

Adding column with values

Dear all, I need your help for my question please I have without header (space separated) and need to add two colomns at the beginning with values my file look like : rs1 a t 0.6 rs2 a c 0.3 rs3 t g 0.8 I need to a new file like: 1 100 rs1 a t 0.6 1 100 rs2 a c 0.3 1 100 rs3 t g... (3 Replies)
Discussion started by: biopsy
3 Replies

9. Shell Programming and Scripting

Need help in adding the column to set of lines that match the certain criteria

Hi, I need help to add a certain values to end of each line in the same file. Here is my input file name:aaa id :123 salary job qulification name:bbb id :124 salary job qulification name:ccc id :345 salary job qulification (2 Replies)
Discussion started by: jpkumar10
2 Replies

10. Shell Programming and Scripting

problem while adding column values in awk

Hi, I have a file "input.txt" with the following content : 5312,0,,,1,8,141.2,20090727 3714,0,,,1,8,285.87,20090727 5426,0,,,1,8,3.9,20090727 3871,0,,,1,8,30.4,20090727 9071,0,,,1,8,146.2,20090727 5141,0,,,1,8,2.8,20090727 0460,0,,,1,8,-0.1,20090727 7918,0,,,1,8,-0.1,20090727... (3 Replies)
Discussion started by: valokv
3 Replies
Login or Register to Ask a Question