break from a single list into multiple columns


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting break from a single list into multiple columns
# 1  
Old 09-12-2012
break from a single list into multiple columns

Hi Guys,

I am prety new to the hell scripting world. I am running some grep/cut commands and extracting from a csv file into a list. But the final product I need is that the whole list that I now have has to be broken and separated into columns.

Say what I now have extracted is a list of currencies and values:

Code:
USD
10
15
20
GBP
11
12
13
FRC
14
18
24

Instead of the above I can also extract each currency list into separate files (worst case scenario).

The next step where I am stuck is ... I want to see in a csv file (to be finally read in excel) the above in separate columns:

Code:
USD GBP FRC
10   11   14
15   12   18
20   13   24

Hope you got the idea. The original file I am extracting from is about 20 million lines long. About 100+ currencies implies the columns that I want to see. Please let me know how I can do this.

All help is appreciated.

Cheers,
H

---------- Post updated at 11:02 AM ---------- Previous update was at 11:01 AM ----------

Please excuse my typo where I types "hell" instead of "shell"

Last edited by Ygor; 09-12-2012 at 06:46 AM.. Reason: Added code tags.
# 2  
Old 09-12-2012
If you can post a representative sample of your original input file, someone here may be able to suggest a better way to get the final output (instead of using grep and/or cut).
# 3  
Old 09-12-2012
Using the sample file...
Code:
$ cat file1
USD
10
15
20
GBP
11
12
13
FRC
14
18
24

$ awk '$1+0!=$1{x++;y=1}{a[x,y++]=$1}z<y{z=y}END{for(j=1;j<z;j++)for(i=1;i<=x;i++)printf a[i,j] (i<x?OFS:ORS)}' file1 > file2

$ cat file2
USD GBP FRC
10 11 14
15 12 18
20 13 24

$

# 4  
Old 09-12-2012
Code:
[root@node3 ~]# cat transpose 
#!/bin/bash 
# 
# transpose 
 
transpose() 
{ 
  awk ' 
      { 
              if(max_nf<NF) 
                 max_nf=NF 
              max_nr=NR 
              for(x=1; x<=NF; ++x) 
                  matrix[x, NR]=$x 
      } 
 
  END { 
              for(x=1; x<=max_nf; ++x) { 
                  for (y=1; y<=max_nr; ++y) 
                       printf("%s ", matrix[x, y]) 
                  printf("\n") 
              } 
      }'  "${1--}"
} 
 
transpose "${1--}"

Code:
[root@node3 ~]# sed -n 'h;n;H;n;H;n;H;x;s:\n: :g;p' infile | bash transpose 
USD GBP FRC 
10 11 14 
15 12 18 
20 13 24

or
Code:
xargs -n 4 < infile |  bash transpose

# 5  
Old 09-12-2012
Hi Complex.invoke,

Your solution works when the break is at 4-5 lines. The one I am testing with has more than 18000 at each point.

So I tried ...
xargs -n 18000 .....

I get an output but te listing doesnt make sense. Is there some parameter to be incremented for larger number of values (arguments)?

Thanks,
H

---------- Post updated at 01:19 AM ---------- Previous update was at 01:14 AM ----------

xargs -n 4 < infile | bash transpose
My test output .... is such that finally the number of columns would be about 100 and 18000 plus rows.

---------- Post updated at 01:35 AM ---------- Previous update was at 01:19 AM ----------

Lines in infile are say total of 18001 x 85
So my infile has (18001 x 85) lines
Run:
xargs -n 18001< infile | bash transpose

My final file should have 85 columns with 18001 records each.
# 6  
Old 09-12-2012
Code:
[root@node3 ~]# cat /etc/redhat-release 
CentOS release 6.3 (Final)
[root@node3 ~]# bash --version | grep -release
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
[root@node3 ~]# for((i=0; i!=18001*85; ++i)); do echo 0 >> file; done &
[root@node3 ~]# wc -l file 
1530085 file
[root@node3 ~]# xargs -n 18001 < file | bash transpose > result
[root@node3 ~]# wc -l result 
18001 result
[root@node3 ~]# awk '{print NF;exit}' result 
85

My solution is a general algorithm that can be used to transpose a MxN matrix to a NxM one
# 7  
Old 09-12-2012
Code:
awk '$1+0 != $1 {close(f); f=sprintf("%06d.dat",++n)} {print > f}' file
paste *.dat > output

Regards,
Alister
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Break a single URL line

I have a single line below in a file, i can have multiple also <java.net.URL>http://server:port/abcd/MaintainRequest.do?operation=121&amp;requestId=123456789&amp;subrequestId=123654789</java.net.URL> I want below to get below output http://server:port/abcd subrequestId=123654789 I am trying... (9 Replies)
Discussion started by: mirwasim
9 Replies

2. Shell Programming and Scripting

Combining columns from multiple files into one single output file

Hi, I have 3 files with one column value as shown File: a.txt ------------ Data_a1 Data_a2 File2: b.txt ------------ Data_b1 Data_b2 Data_b3 Data_b4 File3: c.txt ------------ Data_c1 Data_c2 Data_c3 Data_c4 Data_c5 (6 Replies)
Discussion started by: vfrg
6 Replies

3. Shell Programming and Scripting

Shell scripting - need to arrange the columns from multiple file into a single file

Hi friends please help me on below, i have 5 files like below file1 is x 10 y 20 z 15 file2 is x 100 z 245 file3 is y 78 z 23 file4 is x 100 (3 Replies)
Discussion started by: siva kumar
3 Replies

4. Shell Programming and Scripting

Awk match multiple columns in multiple lines in single file

Hi, Input 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr21.fa chr3.fa 7488 7389 chr1.fa chr1.fa 3546 9887 chr9.fa chr5.fa 7898 7387 chrX.fa chr3.fa Desired Output 7488 7389 chr1.fa chr1.fa 2 3546 9887 chr5.fa chr9.fa 2... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

5. Shell Programming and Scripting

Multiple columns to a single column

I have this input: 10 22 1 100 11 22 10 1 50 14 3 1 100 23 3 1 100 24 15 1 100 10 22 5 3 1 33.333 11 22 1 100 It has an inconsistent number of fields but the last field is determined by 100/(NF-2) using awk. I want to take this multiple columned input file and transform so that... (2 Replies)
Discussion started by: mdlloyd7
2 Replies

6. Shell Programming and Scripting

Break single line in 4

How can i break a single line into 5 lines # joseluiz.silvano; Ramal4846; Sala4121; SetorCorregedoria host DF04488962 { hardware ethernet 00:16:41:68:57:0B; fixed-address 10.100.111.245; } INTO # joseluiz.silvano; Ramal4846; Sala4121; SetorCorregedoria host DF04488962 {... (5 Replies)
Discussion started by: danielldf
5 Replies

7. Shell Programming and Scripting

Filtering issues with multiple columns in a single file

Hi, I am new to unix and would greatly appreciate some help. I have a file containing multiple colums containing different sets of data e.g. File 1: John Ireland 27_December_69 Mary England 13_March_55 Mike France 02_June_80 I am currently using the awk... (10 Replies)
Discussion started by: crunchie
10 Replies

8. Shell Programming and Scripting

Single command for add 2 columns and remove 2 columns in unix/performance tuning

Hi all, I have created a script which adding two columns and removing two columns for all files. Filename: Cust_information_1200_201010.txt Source Data: "1","Cust information","123","106001","street","1-203 high street" "1","Cust information","124","105001","street","1-203 high street" ... (0 Replies)
Discussion started by: onesuri
0 Replies

9. Shell Programming and Scripting

Single column to multiple columns in awk

Hi - I'm new to the awk programming language. I'm trying to print a single column of data to several columns, and I found an article on iTWorld.com (ITworld.com - Printing in columns). It looks like the mkCols2 script is very close to what I need to do, but it looks like the end of the code... (2 Replies)
Discussion started by: astroDave
2 Replies

10. UNIX for Dummies Questions & Answers

single column to multiple columns

Hello, I have a single column of data that I would like to cut/print (with awk or ...) into multiple columns at every empty row (or common character). Input: 5.99123 5.94693 7.21383 5.95202 0.907935 5.99149 6.08427 0.975774 6.077 Output: 5.99123 5.95202 6.08427 5.94693... (7 Replies)
Discussion started by: agibbs
7 Replies
Login or Register to Ask a Question