How can I stack multiple (>1000) fields into one continuous field?


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers How can I stack multiple (>1000) fields into one continuous field?
# 1  
How can I stack multiple (>1000) fields into one continuous field?

Hi,

I'm struggling with a problem at the minute. Basically, I have a file with >1000 columns / fields (with headers), each containing a variable number of values. I would like to stack each column such that all the data appears in a single column, e.g. the first value of column 2 is moved to below the last value of column 1, and then followed by all other values in the column; the first value of column 3 is moved to below this new last value in column 1, and then followed by all other values in the column etc.

As I said, there are over 1000 columns, and the max number of values in any one is around 3500. I have played around with various cut / print / awk commands but not found anything that does what I want. Having said that, I just started working in Unix yesterday, so no idea if these actually can help. Any help would be greatly appreciated!


Thanks a lot


Tom
How can I stack multiple (>1000) fields into one continuous field?-screenshot-2019-05-09-18-00-12png
How can I stack multiple (>1000) fields into one continuous field?-screenshot-2019-05-09-18-00-42png
# 2  
If your input is tab-separated:
Code:
awk -F"\t" '{ if(NF>M) M=NF; for(N=1; N<=NF; N++) if($N) D[N,++C[N]]=$N } END { for(N=1; N<=M; N++) for(X=1; X<=C[N]; X++) print D[N,X] }' inputfile > outputfile

These 4 Users Gave Thanks to Corona688 For This Post:
# 3  
Let me replay this a bit.
Pretend we have :
Code:
1 2 3 4 5
A B C E
X Y Q M

You want this be stacked all into single column:
Code:
1
2
3
4
5
A
B
C
E
X
Y
Q
M

Right?
Try this
Code:
awk '{ for (i=1 ; i<NF; i++) 
            {
               printf("%s\n", $(i) );
            } 
          } ' inputfilename  > outputfilename

Running the code
Code:
Owner@Owner-PC ~
$ mv infile inputfilename

Owner@Owner-PC ~
$ ./t.awk

Owner@Owner-PC ~
$ cat outputfilename
1
2
3
4
A
B
C
X
Y
Q

$ cat t.awk
Code:
awk '{ for (i=1 ; i<NF; i++)
            {
               printf("%s\n", $(i) );
            }
          } ' inputfilename  > outputfilename

Owner@Owner-PC ~
Code:
$ ls -l t.awk
$ ls -l t.awk
-rwxr-xr-x 1 Owner None 143 May  9 13:56 t.awk

You have to chmod +x t.awk to make it an executable script
This User Gave Thanks to jim mcnamara For This Post:
# 4  
Thanks a lot Corona688, that has done exactly what I wanted!

Jim, that's not quite the format I wanted. If I have:
Code:
1 2 3 4 5 

A B C E 

X Y Q M

What I actually want is:

Code:
1
A
X
2
B
Y
3
C
Q
4
E
M
5

Corona's script sorted this though, so all good now Smilie

Last edited by TAlcock; 05-10-2019 at 10:25 AM.. Reason: mis-formatted
This User Gave Thanks to TAlcock For This Post:
# 5  
Code:
$ awk ' { for(i=1;i<=NF;i++) print i, $i } ' file | sort | awk ' { print $2 } '
1
A
X
2
B
Y
3
C
Q
4
E
M
5

This User Gave Thanks to anbu23 For This Post:
# 6  
Hi.

Assume z3:
Code:
1       2       3       4       5
A       B       C       E
X       Y       Q       M

With invisibles shown:
Code:
1^I2^I3^I4^I5$
A^IB^IC^IE$
X^IY^IQ^IM$

Then:
Code:
transpose.pl z3 | tr '\t' '\n'

produces:
Code:
1
A
X
2
B
Y
3
C
Q
4
E
M
5

On a system like:
Code:
OS, ker|rel, machine: Linux, 3.16.0-7-amd64, x86_64
Distribution        : Debian 8.11 (jessie) 
bash GNU bash 4.3.30

And some details on transpose.pl:
Code:
transpose.pl    Swap rows and columns in the given tab-delimited table (MR). (what)
Path    : ~/bin/transpose.pl
Version : - ( local: RepRev 1.1, ~/bin/transpose.pl, 2017-01-29 )
Length  : 28 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/perl
Home    : http://www1.cuni.cz/~obo/textutils/ (doc)

Best wishes ... cheers, drl
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #35
Difficulty: Easy
The IP address 127.0.0.1 is also called 'localhost'.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Extract a column and multiple by 1000 and replace it on same file

Hi All, I need to extract a position in the file and multiple the value by 1000 and the replace it . Original 0010001200084701217637306521200000000000010010000000 ---> 000847 * 1000 0010012700086001213437404323000000000000001001000000 ---> 000860 * 1000... (2 Replies)
Discussion started by: arunkumar_mca
2 Replies

2. Shell Programming and Scripting

Inserting a field without disturbing field separator on other fields

Hi All, I have the input as below: cat input 032016002 2.891 97.109 16.605 27.172 24.017 32.207 0.233 0.021 39.810 0.077 0.026 19.644 13.882 0.131 11.646 0.102 11.449 76.265 23.735 16.991 83.009 8.840 91.160 0.020 99.980 52.102 47.898 44.004 55.996 39.963 18.625 0.121 1.126 40.189... (15 Replies)
Discussion started by: am24
15 Replies

3. Shell Programming and Scripting

Single Field to multiple fields searching

I have a fileA with one column (1000 rows), and fileB with 26 columns(13000 rows). I need to search each value of fileA with fileB and return all the 26 values from FileB to a new file- File C if matches. The search value (from FileA) may present in any of the 26 values in FileB. This value is not... (7 Replies)
Discussion started by: vamsikrishna928
7 Replies

4. Linux

How do I format a Date field of a .CSV file with multiple commas in a string field?

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 "12","B000QRIGJ4","4432","string with quotes, and with a comma, and colon: in... (3 Replies)
Discussion started by: dhruuv369
3 Replies

5. Shell Programming and Scripting

UNIX append field with comparing fields from multiple column

I have a csv dump from sql server that needs to be converted so it can be feed to another program. I already sorted on field 1 but there are multiple columns with same field 1 where it needs to be compared against and if it is same then append field 5. i.e from ANG SJ,0,B,LC22,LC22(0) BAT... (2 Replies)
Discussion started by: nike27
2 Replies

6. Shell Programming and Scripting

How to print 1st field and last 2 fields together and the rest of the fields after it using awk?

Hi experts, I need to print the first field first then last two fields should come next and then i need to print rest of the fields. Input : a1,abc,jsd,fhf,fkk,b1,b2 a2,acb,dfg,ghj,b3,c4 a3,djf,wdjg,fkg,dff,ggk,d4,d5 Expected output: a1,b1,b2,abc,jsd,fhf,fkk... (6 Replies)
Discussion started by: 100bees
6 Replies

7. Shell Programming and Scripting

compare two fields and get a third field

Hello, I'm trying to get a value based on a comparison of two fields, this is: file1 687.45 687.18 687.322 687.405 686.865 file 2 685 6.43 686 6.43 687 6.42 688 6.42 (3 Replies)
Discussion started by: Gery
3 Replies

8. UNIX for Dummies Questions & Answers

Formatting Multiple fields on 1 line to multiple rows

I'm trying extract a number of filename fields from a log file and copy them out as separate rows in a text file so i can load them into a table. I'm able to get the filenames but the all appear on one line. I tried using the cut command with the -d (delimiter) option but cant seem to make it... (1 Reply)
Discussion started by: Sinbad-66
1 Replies

9. Shell Programming and Scripting

Sorting on two fields time field and number field

Hi, I have a file that has data in it that says 00:01:48.233 1212 00:01:56.233 345 00:09:01.221 5678 00:12:23.321 93444 The file has more line than this but i just wanted to put in a snippet to ask how I would get the highest number with time stamp into another file. So from the above... (2 Replies)
Discussion started by: pat4519
2 Replies

10. Shell Programming and Scripting

How to split a field into two fields?

Hi, I have a comma delimited text file where character fields (as opposed to numeric and date fields) are always enclosed with double quotes. Records are separated by the newline character. In a shell script I would like to split a particular field into two separate fields (enclosed with double... (4 Replies)
Discussion started by: vbrown
4 Replies

Featured Tech Videos