Combine Columns


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combine Columns
# 1  
Old 12-09-2015
Combine Columns

Input
Code:
 
NJ090237_0263_GRP,NJ090237_0263_VIEW,NJ090237_0263_PSGRP,NJ090237_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE5
NJ090237_0264_GRP,NJ090237_0263_VIEW,NJ090237_0264_PSGRP,NJ090237_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE5
NJ090233_0263_GRP,NJ090233_0263_VIEW,NJ090233_0263_PSGRP,NJ090233_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE6
NJ090233_0263_GRP,NJ090233_0263_VIEW,NJ090233_0264_PSGRP,NJ090233_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE6

Basically when column 6 is same in the input file , combine $1,$2,$3,$4,$5 using a ";"
but if the value in any of the columns 1-5 is similar , just use unique value .

Code:
gawk '
  {
    i=$6
    p=(i in A)
  } 
  NR==FNR {
    A[i]=A[i] (p?";":x) $1
    B[i]=B[i] (p?";":x) $2
    C[i]=C[i] (p?";":x) $3
    D[i]=D[i] (p?";":x) $4
    E[i]=E[i] (p?";":x) $5
    next
  } 
  p {
    $1=A[i]
    $2=B[i]
    $3=C[i]
    $4=D[i]
    $5=E[i]
    delete A[i]
    delete B[i]
    delete C[i]
    delete D[i]
    delete E[i]
    print
  }
' FS=, OFS=, input1 input1

I am getting this output .. its combining unique values also

Code:
NJ090237_0263_GRP;NJ090237_0264_GRP,NJ090237_0263_VIEW;NJ090237_0263_VIEW,NJ090237_0263_PSGRP;NJ090237_0264_PSGRP,NJ090237_0263_GOLD_CSGRP;NJ090237_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0;06E:0_08E:0_09E:0_11E:0,0CE5
NJ090233_0263_GRP;NJ090233_0263_GRP,NJ090233_0263_VIEW;NJ090233_0263_VIEW,NJ090233_0263_PSGRP;NJ090233_0264_PSGRP,NJ090233_0263_GOLD_CSGRP;NJ090233_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0;06E:0_08E:0_09E:0_11E:0,0CE6

but output needed is
Code:
NJ090237_0263_GRP;NJ090237_0264_GRP,NJ090237_0263_VIEW,NJ090237_0263_PSGRP;NJ090237_0264_PSGRP,NJ090237_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE5
NJ090233_0263_GRP,NJ090233_0263_VIEW,NJ090233_0263_PSGRP;NJ090233_0264_PSGRP,NJ090233_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE6

# 2  
Old 12-09-2015
Are all lines with the same values in the last field always adjacent in your input file (as they are in your example)?

Does the output order matter?

What operating system and shell are you using?

Last edited by Don Cragun; 12-09-2015 at 06:38 PM.. Reason: Add OS & shell question.
# 3  
Old 12-09-2015
They may or may not be adjacent .
The output order does not matter .
Linux ,bourne ,gawk ...

Thanks
# 4  
Old 12-09-2015
Try:
Code:
gawk '
{	A[i = $6]
	for(j = 1; j <= 5; j++)
		if(!((i, j, $j) in d)) {
			d[i, j, $j]
			outd[i, j] = outd[i, j] ? outd[i, j] ";" $j : $j
		}
}
END {	for(i in A)
		print outd[i,1], outd[i,2], outd[i,3], outd[i,4], outd[i,5], i
}' FS=, OFS=, input1

If someone else wants to try this on a Solaris system and gawk isn't loaded, change gawk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 12-09-2015
Thanks Don ! .. worked as expected ...
I have another question for u .. , not related to above though

Basically wherever there is a ";" in front of or after a field , change it to "[Field]"

Code:
 
 Example 
  
 F1;F2;F3,F4,F5,F6;F7,F8,F9;F10;F11

etc


to

Code:
[F1][F2][F3],F4,F5,[F6][F7],F8,[F9][F10][F11]

Thanks
# 6  
Old 12-09-2015
Do you want to take the output produced by my earlier script and transform it into this new output format?

Or, do you want to change my earlier script to produce this new output format instead of the old output format?
# 7  
Old 12-10-2015
The first one if you please .... thx
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Combine columns - awk

Need some help with this ... please 60644,NJ090237_0263_GRP,NJ090237_0263_VIEW,NJ090237_0263_PSGRP,NJ090237_0263_GOLD_CSGRP,,06E:0_08E:0_09E:0_11E:0,0CE5,TDEV,34,VP_TIER... (3 Replies)
Discussion started by: greycells
3 Replies

2. Shell Programming and Scripting

Compare 2 csv files by columns, then extract certain columns of matcing rows

Hi all, I'm pretty much a newbie to UNIX. I would appreciate any help with UNIX coding on comparing two large csv files (greater than 10 GB in size), and output a file with matching columns. I want to compare file1 and file2 by 'id' and 'chain' columns, then extract exact matching rows'... (5 Replies)
Discussion started by: bkane3
5 Replies

3. Shell Programming and Scripting

Combine columns from many files but keep them aligned in columns-shorter left column issue

Hello everyone, I searched the forum looking for answers to this but I could not pinpoint exactly what I need as I keep having trouble. I have many files each having two columns and hundreds of rows. first column is a string (can have many words) and the second column is a number.The files are... (5 Replies)
Discussion started by: isildur1234
5 Replies

4. UNIX for Dummies Questions & Answers

Combine columns from 100 files with same structure

Hi, I have a bunch of files with the following format. PUR.1.9 30910 0.024 0.926 0.050 36587 0.024 0.927 0.049 91857 0.023 0.928 0.049 105797 0.024 0.927 0.049 146659 0.024 0.927 0.049 152695 0.024 0.927 0.049 192118 0.022 0.930 0.048 193310 0.018 0.936 0.046 PUR.2.9 30910 0.028... (6 Replies)
Discussion started by: genehunter
6 Replies

5. Shell Programming and Scripting

Combine columns from multiple files

Can anybody help on the script to combine/concatenate columns from multiple files input1 4 135 5 185 6 85 11 30 16 72 17 30 21 52 22 76 input2 2 50 4 50 6 33 8 62 10 25 12 46 14 42 15 46output (2 Replies)
Discussion started by: sdf
2 Replies

6. UNIX for Dummies Questions & Answers

How to combine 2 files with 6 columns?

This may seem obvious but I am having problems doing this as columns get converted to rows when i try to write a script. I have 2 files text1.txt and text2.txt each of which have 6 columns of numbers separated by a space. I need to combine the 2 files so that the output file text3.txt maintains... (2 Replies)
Discussion started by: tgoldstone
2 Replies

7. Shell Programming and Scripting

How to combine 2 files into 1 file with 2 columns

Hi Guys, I want to combine 2 files and and put together in 1 file and make two columns out it. See below desired output. Any help will be much appreciated. inputfile1.txt 12345 67890 24580 inputfile2.txt AAAAA BBBBB CCCCC (11 Replies)
Discussion started by: pinpe
11 Replies

8. UNIX for Dummies Questions & Answers

combine the values from the first two columns within a file

Hello everybody, I have a text file containing 10,000 rows and 5000 columns. The values are separated by a tab. Ex. file_ex.ped 1 mike 0 0 2 1 A A G G C T A G 1 jack 0 0 2 2 T A G T C A A C 1 Mary 0 0 1 2 A T G C A T G C ... I would like a out put file 1 mike 0 0 2 1 AA GG CT AG 1... (7 Replies)
Discussion started by: Unilearn
7 Replies

9. Shell Programming and Scripting

Single command for add 2 columns and remove 2 columns in unix/performance tuning

Hi all, I have created a script which adding two columns and removing two columns for all files. Filename: Cust_information_1200_201010.txt Source Data: "1","Cust information","123","106001","street","1-203 high street" "1","Cust information","124","105001","street","1-203 high street" ... (0 Replies)
Discussion started by: onesuri
0 Replies

10. Shell Programming and Scripting

Combine multiple columns from multiple files

Hi there, I was wondering if someone can help me with this. I am trying the combine multiple columns from multiple files into one file. Example file 1: c0t0d0 c0t2d0 # hostname vgname c0t0d1 c0t2d1 # hostname vgname c0t0d2 c0t2d2 # hostname vgname c0t1d0 c0t3d0 # hostname vgname1... (5 Replies)
Discussion started by: martva
5 Replies
Login or Register to Ask a Question