Shell Programming and Scripting

I have done a couple of searches on this and have found many threads but I don't think I've found one that is useful to me - probably because I have very basic comprehension of perl and beginners shell so trying to manipulate a script already posted maybe beyond my capabilities....

Anyway - I have a huge file (247 columns, over 500,000 lines). What I want to do ultimately is transpose this entire file to make the columns, rows and the rows, columns. Is there an easy way to do this in perl and/or shell? If so, how?

Cheers.

Code:

perl -F, -lane 'for ( 0 .. $#F ) { $rows[$_] .= $F[$_] }; eof && print map "$_\n", @rows' data

if the file "data" contains CSV data, like this:

Code:

1,2,3,4,5,6,7
a,b,c,d,e,f,g

The code above will output something like this:

Code:

1a
2b
3c
4d
5e
6f
7g

Whatever is the separatrix for your data can be exchanged for the comma after the "-F" switch in the code.

This should work on arbitrarily large files.

Hope That Helps

P.S. you are talking about 123,500,000 cells in a 247 by 500,000 matrix so memory could become a problem for the $rows variable, particularly if you are on 32bit. We are building up the result in the $rows array and waiting to the end to print it out. I can work on a streaming solution if you get the old "Out of Memory!" error

Yes - I am getting the out of memory error... but I dunno why. I can easily open the original file in an 32bit OS system, but when I transpose, all hell breaks loose. I tried to grep out a single line from the new data file, but I got this:

Code:

grep: line too long

What does that mean? That my dat file is in one single line?

Cheers

it could. there are many ways to find out if that is the case. Here is one way:

Code:

shell_prompt-> head -1  datafile

if that kicks back the whole file, there are no line endings

Mmmmm... this is where I get stupid. Typing that in doesn't give me anything...

use perl module called Array::Transpose:-

if file is space separated.

Code:

perl -M'Array::Transpose'  -wlane '

push @in , [ @F ] ;

END{
@out = transpose(\@in) ;
print "@out" ;
}
' infile.txt

Thanks - I'll give this a go. Though, my file is actually tab-delimited - will this matter?