Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Transpose a txt file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Transpose a txt file
# 1  
Old 04-29-2015
Transpose a txt file
Hello,
I have a text file with 148 rows and 2532691 columns. I need to transpose the data. The command that I am using is
Code:
 awk '
{ 
    for (i=1; i<=NF; i++)  {
        a[NR,i] = $i
    }
}
NF>p { p = NF }
END {    
    for(j=1; j<=p; j++) {
        str=a[1,j]
        for(i=2; i<=NR; i++){
            str=str" "a[i,j];
        }
        print str
    }
}' file.raw > output

does not work for me due to memory issues -> the job crashes over the server. Is there a better way to handle the data?
many thanks
# 2  
Old 04-29-2015
I dont know awk well enough, but i wouldnt think that it causes a memory issue.
This said, to me it seems that these parts might be the cause:
Code:
str=a[1,j]
    str=str" "a[i,j];

Other than that, i dont see a to be initalized at all.

hth
# 3  
Old 04-29-2015
Using intermediate files will relieve memory. Try
Code:
awk '{$1=$1; print > "FILE"NR}' FS="," OFS="\n" file.raw
paste FILE* > output

Last edited by RudiC; 04-29-2015 at 10:27 AM..
# 4  
Old 04-29-2015
Thank you, But still running into memory issues. This is my script

Code:
awk '{$1=$1; print > "FILE"NR}' FS="," OFS="\n" file.raw

 awk '
{ 
    for (i=1; i<=NF; i++)  {
        a[NR,i] = $i
    }
}
NF>p { p = NF }
END {    
    for(j=1; j<=p; j++) {
        str=a[1,j]
        for(i=2; i<=NR; i++){
            str=str" "a[i,j];
        }
        print str
    }
}' FILE* | paste FILE* > output

# 5  
Old 04-29-2015
I'm absolutely not sure why you tear apart my two commands and then complain it wouldn't work.
To prove it works, try smaller but representative sample files.
# 6  
Old 04-30-2015
well, because
Code:
awk '{$1=$1; print > "FILE"NR}' FS="," OFS="\n" file.raw

is only splitting the file into rows and not transposing it to columns. So my intenstion was to split the rows and transpose it and then join all the columns..
# 7  
Old 04-30-2015
What's your field separator? If space, drop the FS=",". If other, use that for the FS variable.
This User Gave Thanks to RudiC For This Post:
nans
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

[Python] replicating "sha256 -C checksum_file.txt file.txt"

Hello everyone, Since my python knowledge is limimted, I've challenged myself to learn as much as possible to help me with my carrere. I'm currently trying to convert a shell script to python, just to give myself a task. There is one section of the script that I'm having issues converting and... (2 Replies)
Discussion started by: da1
2 Replies

2. Shell Programming and Scripting

Desired output.txt for reading txt file using awk?

Dear all, I have a huge txt file (DATA.txt) with the following content . From this txt file, I want the following output using some shell script. Any help is greatly appreciated. Greetings, emily DATA.txt (snippet of the huge text file) 407202849... (2 Replies)
Discussion started by: emily
2 Replies

3. Windows & DOS: Issues & Discussions

2 Questions: replace text in txt file, add text to end of txt file

so... Lets assume I have a text file. The text file contains multiple "#" symbols. I want to replace all thos "#"s with a STRING using DOS/Batch I want to add a certain TEXT to the end of each line. How can I do this WITHOUT aid of sed, grep or anything linux related ? (1 Reply)
Discussion started by: pasc
1 Replies

4. Shell Programming and Scripting

To transpose columns + edit in a txt file

Hi, I have a txt file that looks like log2FoldChange Ontology_term 8.50624450251828 GO:0003700,GO:0003707,GO:0005634,GO:0006355,GO:0043401,GO:0003700,GO:0005634,GO:0006355,GO:0008270,GO:0043565 7.03936870356684 GO:0005515,GO:0008080 6.49606183738682 6.49525073909629 GO:0005515... (4 Replies)
Discussion started by: alisrpp
4 Replies

5. Shell Programming and Scripting

Need to append the date | abcddate.txt to the first line of my txt file

I want to add/append the info in the following format to my.txt file. 20130702|abcd20130702.txt FN|SN|DOB I tried the below script but it throws me some exceptions. <#!/bin/sh dt = date '+%y%m%d'members; echo $dt+|+members+$dt; /usr/bin/awk -f BEGIN { FS="|"; OFS="|"; } { print... (6 Replies)
Discussion started by: harik1982
6 Replies

6. Shell Programming and Scripting

awk append fileA.txt to growing file B.txt

This is appending a column. My question is fairly simple. I have a program generating data in a form like so: 1 20 2 22 3 23 4 12 5 43 For ever iteration I'm generating this data. I have the basic idea with cut -f 2 fileA.txt | paste -d >> FileB.txt ???? I want FileB.txt to grow, and... (4 Replies)
Discussion started by: theawknewbie
4 Replies

7. Shell Programming and Scripting

Transpose a file

input IndID ID1 ID2 ID3 a1 a/a b/b c/c a2 a/a b/b c/c a3 a/b b/b c/d a6 a/b b/b c/e a8 a/a b/c c/e a9 b/b b/d c/e output IDName IndID IDtype C_No ID1 a1 a/a 1 ID1 a2 a/a 1 ID1 a8 ... (1 Reply)
Discussion started by: stateperl
1 Replies

8. Shell Programming and Scripting

command to list .txt and .TXT file

Hi expersts, in my directory i have *.txt and *.TXT and *.TXT.log, *.txt.log I want list only .txt and .TXT files in one command... how to ?? //purple (1 Reply)
Discussion started by: thepurple
1 Replies

9. Shell Programming and Scripting

AWK CSV to TXT format, TXT file not in a correct column format

HI guys, I have created a script to read 1 column in a csv file and then place it in text file. However, when i checked out the text file, it is not in a column format... Example: CSV file contains name,age aa,11 bb,22 cc,33 After using awk to get first column TXT file... (1 Reply)
Discussion started by: mdap
1 Replies

10. UNIX for Dummies Questions & Answers

Binary txt file received when i use uuencode to send txt file as attachment

Hi, I have already read a lot of posts on sending attachments in unix...but none of them were of help for my problem...so here goes.. i wanna attach a text file and send to a mail id..used the following code : uuencode "$File1" "$File1" ;|mail -s "$Mail_sub" abc@abc.com it works... (2 Replies)
Discussion started by: ash22
2 Replies
Login or Register to Ask a Question