Help for a Perl newcomer! Transposing data from columns to rows


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help for a Perl newcomer! Transposing data from columns to rows
# 1  
Old 11-24-2011
Help for a Perl newcomer! Transposing data from columns to rows

I have to create a Perl script which will transpose the data output from my experiment, from columns to rows, in order for me to analyse the data.

I am a complete Perl novice so any help would be greatly appreciated.

The data as it stands looks like this:
Code:
 Subject  Condition      Fp1        Fp2       FpZ
  S1          mixedneg    235.012   125.65   548.233
  S1          mixedneut   212.1     548.215    1548.2 
  S1          pureneg     1578.22   15.25      1654.2 
  S1          pureneut    4878.2     512.448    545.88 
  S2          mixedneg    5465       87465     46984

  S2          mixedneut    545       7498         5136

  S2          pureneg        587        8798        6516

  S2          pureneut      874       8730.39     879

(But the real data file is much bigger, with each subject having 8 rows, and across the top, where it reads Fp1 etc. there are 64 items like this)


I need the data to look like this to analyse:


Code:
Subject  Fp1mixedneg  Fp1mixedneut   Fp1pureneg   Fp1pureneut  Fp2mi    
  S1         235.012              125.655        548.233
  S2         212.1                  548.215        1548.2      AND so on....
  S3         1578.22               15.25          1654.2 
  S4         4878.2                 512.448        545.88 
  S5         
  S6         
  S7        
  S8

I need each subject to only have one row, and the condition column removed. The conditions need to then be specified along side each item at the top (so Fp1 repeated 4 times, with each condition next to it, Fp2 repeated 4 times with each condition next to it etc...)

In the real data set, as mentioned, there are 64 items along the top (so 64 different e.g Fp1) and there are 8 conditions for each subject (not four as my above example shows). there are 22 subjects in total in the data set too.

If any other information is needed feel free to ask.

Any sample scripts that I can use would be greatly appreciated, thanks!

---------- Post updated at 06:42 PM ---------- Previous update was at 06:41 PM ----------

Sorry the examples of my data have not aligned properly when i posted the thread but hopefully you can still understand what i mean!
Moderator's Comments:
Mod Comment Please use code tags!

Last edited by vbe; 11-28-2011 at 12:29 PM..
# 2  
Old 11-24-2011
The following is a sample script. The output seems match what you want.

Code:
#!/usr/bin/perl

use strict;
use warnings;

my $line = <STDIN>;
chomp($line);
my @header_item = split(/ /, $line);
die("Invalid header line.\n") if (@header_item < 2);

my %new_header_item = ();
my %data = ();

while ($line = <STDIN>)
{
    chomp($line);
    my @item = split(/ /, $line);
    next if (@item < 2);
    $data{$item[0]} = () if (!defined($data{$item[0]}));

    for (my $i = 2; $i < @item; $i++)
    {
        last if (!defined($header_item[$i]));
        my $name = $header_item[$i].$item[1];
        $new_header_item{$name} = '';
        $data{$item[0]}{$name} = $item[$i];
    }
}

my @h = sort keys(%new_header_item);
print $header_item[0].' '.join(' ', @h)."\n";

foreach my $subj (keys(%data))
{
    print $subj;

    foreach my $key (@h)
    {
        my $value = defined($data{$subj}{$key}) ? $data{$subj}{$key} : 0;
        print " $value";
    }

    print "\n";
}

exit(0)

Save the code in a script, then run it by:

Code:
./script.pl < data_file

Sample output:

Code:
Subject Fp1mixedneg Fp1mixedneut Fp1pureneg Fp1pureneut Fp2mixedneg Fp2mixedneut Fp2pureneg Fp2pureneut FpZmixedneg FpZmixedneut FpZpureneg FpZpureneut
S1 235.012 212.1 1578.22 4878.2 125.65 548.215 15.25 512.448 548.233 1548.2 1654.2 545.88
S2 5465 545 587 874 87465 7498 8798 8730.39 46984 5136 6516 879

# 3  
Old 11-28-2011
Thank you so much for this, it looks like it will be a great help!

Is there any way of getting the output to save in a file, so I can open it and run my analysis?

Thanks!
# 4  
Old 11-29-2011
Quote:
Originally Posted by Sarah_W
...Is there any way of getting the output to save in a file, so I can open it and run my analysis?
...
One way to do that would be to use the shell's redirection operator to redirect the output to a file -

Code:
./script.pl < data_file > output_file

The command above makes the shell redirect the input data from "data_file" to the Perl program "script.pl" and redirect the output to the file "output_file".
Once the execution is over, you could open "output_file" for further processing.

tyler_durden
This User Gave Thanks to durden_tyler For This Post:
# 5  
Old 11-29-2011
Thanks, I can run the script now and save the output however the script does not appear to be doing what I wanted and what you managed to get it to do!
Can you suggest what I may be doing wrong or any way to alter the script to get it to allign the collumns by subject number correctly?!

Thanks!!
# 6  
Old 11-29-2011
Quote:
Originally Posted by Sarah_W
Thanks, I can run the script now and save the output however the script does not appear to be doing what I wanted and what you managed to get it to do!
We can't see your computer from here.

In what way is it not working?
# 7  
Old 11-29-2011
As in this is the output i get:
Code:
subjects AF3_Li..nce AF4_Li..nce AF7_Li..nce AF8_Li..nce C1_Lin..nce C2_Lin..nce C3_Lin..nce C4_Lin..nce C5_Lin..nce C6_Lin..nce CP1_Li..nce CP2_Li..nce CP3_Li..nce CP4_Li..nce CP5_Li..nce CP6_Li..nce CPz_Li..nce Cz_Lin..nce F1_Lin..nce F2_Lin..nce F3_Lin..nce F4_Lin..nce F5_Lin..nce F6_Lin..nce F7_Lin..nce F8_Lin..nce FC1_Li..nce FC2_Li..nce FC3_Li..nce FC4_Li..nce FC5_Li..nce FC6_Li..nce FCz_Li..nce FT7_Li..nce FT8_Li..nce Fp1_Li..nce Fp2_Li..nce Fpz_Li..nce Fz_Lin..nce HEOG_L..nce M1_Lin..nce M2_Lin..nce O1_Lin..nce O2_Lin..nce Oz_Lin..nce P1_Lin..nce P2_Lin..nce P3_Lin..nce P4_Lin..nce P5_Lin..nce P6_Lin..nce P7_Lin..nce P8_Lin..nce PO3_Li..nce PO4_Li..nce PO5_Li..nce PO6_Li..nce PO7_Li..nce PO8_Li..nce POz_Li..nce Pz_Lin..nce T7_Lin..nce T8_Lin..nce TP7_Li..nce TP8_Li..nce VEOG_L..nce
   4.0749   7.7563          2.1861      S1 11.2153   PurNegForg   12.0973     12.2897       1.0983         6.4911   3.7960   10.4519  11.9439     2.0849

nothing alligned, nothing arranged by subject, the conditions are not specified by the items at the top, it just comes out a random arrangement like above..

It needs to be arranged by condition and item on the top row, with each subject only having one row.

Any help with this is greatly appreciated!

Last edited by Franklin52; 12-03-2011 at 11:37 AM.. Reason: Please use code tags for data and code samples, thank you
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Transposing rows to columns with multiple similar lines

Hi, I am trying to transpose rows to columns for thousands of records. The problem is there are records that have the same lines that need to be separated. the input file as below:- ID 1A02_HUMAN AC P01892; O19619; P06338; P10313; P30444; P30445; P30446; P30514; AC Q29680; Q29837;... (2 Replies)
Discussion started by: redse171
2 Replies

2. Shell Programming and Scripting

Transposing rows and columns (pivoting) using shell scripting

Here is the contents of an input file. A,1,2,3,4 10,aaa,bbb,ccc,ddd 11,eee,fff,ggg,hhh 12,iii,jjj,lll,mmm 13,nnn,ooo,ppp I wanted the output to be A 10 1 aaa 10 2 bbb 10 3 ccc 10 4 ddd 11 1 eee 11 2 fff 11 3 ggg 11 4 hhh ..... and so on How to do it in ksh... (9 Replies)
Discussion started by: ksatish89
9 Replies

3. Shell Programming and Scripting

transposing columns into rows

Hi, I need to transpose columns of my files into rows and save it as individual files. sample contents of the file below. 0.9120 0.7782 0.6959 0.6904 0.6322 0.8068 0.9082 0.9290 0.7272 0.9870 0.7648 0.8053 0.8300 0.9520 0.8614 0.6734 0.7910 0.6413 0.7126 0.7364 0.8491 0.8868 0.7586 0.8949... (8 Replies)
Discussion started by: ida1215
8 Replies

4. Shell Programming and Scripting

Transpose Data from Columns to rows

Hello. very new to shell scripting and would like to know if anyone could help me. I have data thats being pulled into a txt file and currently have to manually transpose the data which is taking a long time to do. here is what the data looks like. Server1 -- Date -- Other -- value... (7 Replies)
Discussion started by: Mikes88
7 Replies

5. Shell Programming and Scripting

Transposing Repeated Rows to Columns.

I have 1000s of these rows that I would like to transpose to columns. However I would like the transpose every 3 consecutive rows to columns like below, sorted by column 3 and provide a total for each occurrences. Finally I would like a grand total of column 3. 21|FE|41|0B 50\65\78 15... (2 Replies)
Discussion started by: ravzter
2 Replies

6. Shell Programming and Scripting

awk, string as record separator, transposing rows into columns

I'm working on a different stage of a project that someone helped me address elsewhere in these threads. The .docs I'm cycling through look roughly like this: 1 of 26 DOCUMENTS Copyright 2010 The Age Company Limited All Rights Reserved The Age (Melbourne, Australia) November 27, 2010... (9 Replies)
Discussion started by: spindoctor
9 Replies

7. UNIX for Dummies Questions & Answers

Suggestion to convert data in rows to data in columns

Hello everyone! I have a huge dataset looking like this: nameX nameX 0 1 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 ............... nameY nameY 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 ..... nameB nameB 0 1 2 2 2 2 2 2 2 2 1 2 2 2 1 2 2 2 ..... (can be several thousands of codes) and I need... (8 Replies)
Discussion started by: kush
8 Replies

8. Shell Programming and Scripting

Transposing rows into columns

I have a file like the one given below P1|V1|V2 P1|V1|V3 P1V1|V2 P2|V1|V4 P2|V2|V6 P2|V1|V4 I want it convert to P1|V1|V2|V2|V3 P2|V1|V4|V2|V6 2nd and 3rd column should be considered as together and so the tird row is duplicate Any ideas? (3 Replies)
Discussion started by: prasperl
3 Replies

9. Shell Programming and Scripting

Data in Rows to Columns

Hi, I am a beginner in bash&perl. I have data in form of:- A 1 B 2 C 3 D 4 E 5 I would like your help to find a simple way to change it to :- A B C D E 1 2 3 4 5 Any help would be highly appreciated. (8 Replies)
Discussion started by: umaars
8 Replies

10. Shell Programming and Scripting

Rows to columns transposing and reformating.

----File attached. Input file =========== COL_1 <IP Add 1> COL_2 <Service1> COL_3 <ABCDEFG> COL_4 <IP ADD:PORT> COL_4 <IP ADD:PORT> COL_1 <IP Add 2> COL_2 <Service2> COL_2 <Service3> COL_2 <Service4> COL_3 <AAAABBB> COL_4 <IP ADD:PORT> COL_4 <IP ADD:PORT> COL_4 <IP... (27 Replies)
Discussion started by: bluethunder
27 Replies
Login or Register to Ask a Question