Linux - Transpose rows into column


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Linux - Transpose rows into column
# 1  
Old 07-25-2017
Linux - Transpose rows into column

hello,
I have a server that collect some performance statistics of 4 servers in the following input file :

Code:
$ cat inputfile
Time,A,Server1,KPI1,data1
Time,A,Server1,KPI2,data2
Time,A,Server1,KPI3,data3
Time,A,Server1,KPI4,data4
Time,A,Server1,KPI5,data5
Time,A,Server2,KPI1,data6
Time,A,Server2,KPI2,data7
Time,A,Server2,KPI3,data8
Time,A,Server2,KPI4,data9
Time,A,Server2,KPI5,data10
Time,A,Server3,KPI1,data11
Time,A,Server3,KPI2,data12
Time,A,Server3,KPI3,data13
Time,A,Server3,KPI4,data14
Time,A,Server3,KPI5,data15
Time,A,Server4,KPI1,data16
Time,A,Server4,KPI2,data17
Time,A,Server4,KPI3,data18
Time,A,Server4,KPI4,data19
Time,A,Server4,KPI5,data20

I'ld like to convert this input file (using bash or perl script) to 4 output files (for each server) and the output will be as following

Code:
$ cat Server1.csv
Time,KPI2,KPI4,KPI5
Time,data2,data4,data5

$ cat Server2.csv
Time,KPI2,KPI4,KPI5
Time,data6,data9,data10

$ cat Server3.csv
Time,KPI2,KPI4,KPI5
Time,data12,data14,data15

$ cat Server4.csv
Time,KPI2,KPI4,KPI5
Time,data17,data19,data20

thanks in advance

Last edited by capitain25; 07-25-2017 at 02:11 PM..
# 2  
Old 07-25-2017
Welcome Captain25,

I have a few to questions pose in response first:-
  • What have you tried so far?
  • What output/errors do you get?
  • Have you searched the board yet? There are several threads asking the same thing.
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)
Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.


Kind regards,
Robin
# 3  
Old 07-25-2017
Hello
• What have you tried so far? ……I tried to create a scripts basing on some scripts that transpose rows to columns posted on this forum.
• What output/errors do you get? ……No error, but the output was not conform as what I need
• Have you searched the board yet? There are several threads asking the same thing ……… even on other forums
• What OS and version are you using? Redhat GNU/Linux
• What are your preferred tools? (C, shell, perl, awk, etc.) ……… shell or perl
• What logical process have you considered? (to help steer us to follow what you are trying to achieve) to be more clear :

I have an input file that contain the following data:

Code:
$ cat inputfile
Time,A,Server1,KPI1,data1
Time,A,Server1,KPI2,data2
Time,A,Server1,KPI3,data3
Time,A,Server1,KPI4,data4
Time,A,Server1,KPI5,data5
Time,A,Server2,KPI1,data6
Time,A,Server2,KPI2,data7
Time,A,Server2,KPI3,data8
Time,A,Server2,KPI4,data9
Time,A,Server2,KPI5,data10
Time,A,Server3,KPI1,data11
Time,A,Server3,KPI2,data12
Time,A,Server3,KPI3,data13
Time,A,Server3,KPI4,data14
Time,A,Server3,KPI5,data15
Time,A,Server4,KPI1,data16
Time,A,Server4,KPI2,data17
Time,A,Server4,KPI3,data18
Time,A,Server4,KPI4,data19
Time,A,Server4,KPI5,data20

I’ld like to convert this input file (using bash or perl script) to 4 output files (for each server) and the output will be as following

Code:
$ cat Server1.csv
Time,KPI2,KPI4,KPI5
Time,data2,data4,data5

$ cat Server2.csv
Time,KPI2,KPI4,KPI5
Time,data6,data9,data10

$ cat Server3.csv
Time,KPI2,KPI4,KPI5
Time,data12,data14,data15

$ cat Server4.csv
Time,KPI2,KPI4,KPI5
Time,data17,data19,data20

thanks in advance
# 4  
Old 07-26-2017
Here's some Perl code that appears to accomplish what you want.
It is based, however, on assumptions that part of your output is hard-coded, your input file is sorted by server name etc.

Code:
$ 
$ cat inputfile
Time,A,Server1,KPI1,data1
Time,A,Server1,KPI2,data2
Time,A,Server1,KPI3,data3
Time,A,Server1,KPI4,data4
Time,A,Server1,KPI5,data5
Time,A,Server2,KPI1,data6
Time,A,Server2,KPI2,data7
Time,A,Server2,KPI3,data8
Time,A,Server2,KPI4,data9
Time,A,Server2,KPI5,data10
Time,A,Server3,KPI1,data11
Time,A,Server3,KPI2,data12
Time,A,Server3,KPI3,data13
Time,A,Server3,KPI4,data14
Time,A,Server3,KPI5,data15
Time,A,Server4,KPI1,data16
Time,A,Server4,KPI2,data17
Time,A,Server4,KPI3,data18
Time,A,Server4,KPI4,data19
Time,A,Server4,KPI5,data20
$ 
$ perl -F, -lane 'BEGIN { @occurrences = (2, 4, 5);
                      $prefix = "Time";
                      %active = map {$_ => 1} @occurrences;
                      $header = join ",", ($prefix, map {"KPI".$_} @occurrences);
                  }
                  if ($F[2] ne $prev) {
                      $counter = 1;
                      if ($output) {print FH $output}
                      open(FH, ">", $F[2].".csv");
                      print FH $header;
                      $output = $prefix;
                  } elsif ($active{$counter}) {
                      $output .= ",".$F[4];
                  }
                  $prev = $F[2];
                  $counter++;
                  END {print FH $output}
                 ' inputfile
$ 
$ find . -name "*.csv"
./Server2.csv
./Server3.csv
./Server4.csv
./Server1.csv
$ 
$ cat Server1.csv
Time,KPI2,KPI4,KPI5
Time,data2,data4,data5
$ 
$ cat Server2.csv
Time,KPI2,KPI4,KPI5
Time,data7,data9,data10
$ 
$ cat Server3.csv
Time,KPI2,KPI4,KPI5
Time,data12,data14,data15
$ 
$ cat Server4.csv
Time,KPI2,KPI4,KPI5
Time,data17,data19,data20
$ 
$

If it doesn't work for your input file then post a sample of your real-world input data and the output desired from that real-world data.
# 5  
Old 07-26-2017
Hello,

thank you durden_tyler for your answer, your perl script work perfectly but it’s static, and I’ld like to have a dynamic script file to use it on a cronjob (every 5min) to collect the following KPIs:
dynamic script mean that the KPI and server name and time need to be collected from the input file.

Code:
Inputfile= counter_2017072516

KPI2= Used Physical Mem MB %, (4th position data on the line)
KPI4= Total CPU Usage of tcs SERVER Component %,
KPI5= Total CPU Usage of tcs Node %,

Server1= m07tcs1
Server2= m05tcs2
Server3= m04tcs3
Server4= m07tcs4

Time=2017-07-25 16:46:04 (dynamic)

Code:
$ cat counter_2017072516
2017-07-25 16:46:04,m07tcs1,SERVER@m07tcs1,Used Physical Mem MB,5
2017-07-25 16:46:04,m07tcs1,SERVER@m07tcs1,Used Physical Mem %,29
2017-07-25 16:46:04,m07tcs1,SERVER@m07tcs1,Used Heap Mem MB,9662
2017-07-25 16:46:04,m07tcs1,SERVER@m07tcs1,Total CPU Usage of tcs SERVER Component %,0.0
2017-07-25 16:46:04,m07tcs1,SERVER@m07tcs1,Total CPU Usage of tcs Node %,0.67

2017-07-25 16:46:04,m05tcs2,SERVER@m05tcs2,Used Physical Mem MB,6.4
2017-07-25 16:46:04,m05tcs2,SERVER@m05tcs2,Used Physical Mem %,28
2017-07-25 16:46:04,m05tcs2,SERVER@m05tcs2,Used Heap Mem MB,666
2017-07-25 16:46:04,m05tcs2,SERVER@m05tcs2,Total CPU Usage of tcs SERVER Component %,0.0
2017-07-25 16:46:04,m05tcs2,SERVER@m05tcs2,Total CPU Usage of tcs Node %,1.65

2017-07-25 16:46:04,m04tcs3,SERVER@m04tcs3,Used Physical Mem MB,8
2017-07-25 16:46:04,m04tcs3,SERVER@m04tcs3,Used Physical Mem %,15
2017-07-25 16:46:04,m04tcs3,SERVER@m04tcs3,Used Heap Mem MB,366
2017-07-25 16:46:04,m04tcs3,SERVER@m04tcs3,Total CPU Usage of tcs SERVER Component %,0.0
2017-07-25 16:46:04,m04tcs3,SERVER@m04tcs3,Total CPU Usage of tcs Node %,04

2017-07-25 16:46:04,m07tcs4,SERVER@m07tcs4,Used Physical Mem MB,13
2017-07-25 16:46:04,m07tcs4,SERVER@m07tcs4,Used Physical Mem %,45.5
2017-07-25 16:46:04,m07tcs4,SERVER@m07tcs4,Used Heap Mem MB,778
2017-07-25 16:46:04,m07tcs4,SERVER@m07tcs4,Total CPU Usage of tcs SERVER Component %,1.0
2017-07-25 16:46:04,m07tcs4,SERVER@m07tcs4,Total CPU Usage of tcs Node %,0.8

The output files need to be like

Code:
$ cat m07tcs1_2017072516.csv
Time, Used Physical Mem %, Total CPU Usage of tcs SERVER Component %, Total CPU Usage of tcs Node %
2017-07-25 16:46:04,29,0.0,0.67

$ cat m05tcs2_2017072516.csv
Time, Used Physical Mem %, Total CPU Usage of tcs SERVER Component %, Total CPU Usage of tcs Node %
2017-07-25 16:46:04,28,0.0,0.4

$ cat m04tcs3_2017072516.csv
Time, Used Physical Mem %, Total CPU Usage of tcs SERVER Component %, Total CPU Usage of tcs Node %
2017-07-25 16:46:04,15,1.0,04

$ cat m07tcs4_2017072516.csv
Time, Used Physical Mem %, Total CPU Usage of tcs SERVER Component %, Total CPU Usage of tcs Node %
2017-07-25 16:46:04,45.5,1.0,0.8

Thanks again for your replies

Last edited by capitain25; 07-26-2017 at 08:07 AM..
# 6  
Old 07-26-2017
So, what have you tried to achieve this? If you show us your code so far then we can suggest changes to help. That way, we're not just writing code that you don't understand and can't support and we can see what other things your description might be missing.

Giving us something that doesn't work along with the output/errors and the desired output from real input would be very useful. We're all here to learn, after all.



Thanks again,
Robin
# 7  
Old 07-26-2017
hi rbatte1

I'm a noob in scripting, and I posted this thread, looking for someone that can help me to develop the requested script.


Best regards

---------- Post updated at 02:48 PM ---------- Previous update was at 07:36 AM ----------

Hello,
Any idea ?
Br

Last edited by capitain25; 07-26-2017 at 04:47 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to transpose pieces of data in a column to multiple rows?

Hello Everyone, I am very new to the world of regular expressions. I am trying to use grep/sed for the following: Input file is something like this and there are multiple such files: abc 1 2 3 4 5 ***END*** abc 6 7 8 9 ***END*** abc 10 (2 Replies)
Discussion started by: shellnewuser
2 Replies

2. Programming

To transpose rows to column in hadoop

Hi, i am having an HDFS file which is comma seperated, i need to transpose from rows to column only the header columns text.csv cnt,name,place 1,hi,nz 2,hello,aus I need cnt, name, place while using below command in hadoop getting the error hadoop fs -fmt -1 text.csv (0 Replies)
Discussion started by: rohit_shinez
0 Replies

3. Shell Programming and Scripting

Peel syntax for transpose rows into column

Dear all, Plz let me know syntax for transposing rows into column in perl, I am having 30 csv files which are merged into a single xls sheet. but i want to transpose each row into column in excel sheet in each tab (1 CSV = 1tab in xls sheet) example is as below ... (0 Replies)
Discussion started by: sagar_1986
0 Replies

4. Shell Programming and Scripting

Transpose multiple rows (with a mix of space and enter) to a single column

How to change the uploaded weekly file data to the following format? New Well_Id,Old Well_Id,District,Thana,Date,Data,R.L,WellType,Lati.,Longi. BAG001,PT006,BARGUNA,AMTALI,1/2/1978,1.81,2.29,Piezometer,220825,901430 BAG001,PT006,BARGUNA,AMTALI,1/9/1978,1.87,2.29,Piezometer,220825,901430... (3 Replies)
Discussion started by: sara.nowreen
3 Replies

5. Shell Programming and Scripting

Transpose Column of Data to Rows

I can no longer find my commands, but I use to be able to transpose data with common fields from a single column to rows using a command line. My data is separated as follows: NAME=BOB ADDRESS=COLORADO PET=CAT NAME=SUSAN ADDRESS=TEXAS PET=BIRD NAME=TOM ADDRESS=UTAH PET=DOG I would... (7 Replies)
Discussion started by: docdave78
7 Replies

6. Shell Programming and Scripting

awk to transpose preceding row to 1st column of next rows

Gurus: How can I transpose the output below to a format in which I can plot a graph to show VSZ memory usage by PIDs? stdout: Tue Jan 22 07:29:19 CUT 2013 42336296 1841272 java wilyadm 21889232 438616 jlaunch sidadm 42532994 414336 jlaunch sidadm Tue Jan 22 07:49:20 CUT 2013... (1 Reply)
Discussion started by: ux4me
1 Replies

7. Shell Programming and Scripting

Transpose Datefield from rows to column + Print time diff

Hi Experts, Can you please help me in transposing Datefield from rows to column and calculate the time difference for each of the Jobids: Input File: 08/23/2012 12:36:09,JOB_5340 08/23/2012 12:36:14,JOB_5340 08/23/2012 12:36:22,JOB_5350 08/23/2012 12:36:26,JOB_5350 Required Output:... (6 Replies)
Discussion started by: asnandhakumar
6 Replies

8. Shell Programming and Scripting

awk transpose rows to column

Need to transpose in awk rows to column like this: input: A1,6,5,4 3,2,1, A2,8,7,9,10,11,12,13,14 A3,1,2,3,5,7,8,9 A4,9,4,8,1,5,3, output: A1,1 A1,2 A1,4 ... A2,7 A2,8 ... A3,1 A3,2 ... A4,1 A4,3 (5 Replies)
Discussion started by: sdf
5 Replies

9. Shell Programming and Scripting

transpose rows to columns

Any tips on how I can awk the input data to display the desired output per below? Thanking you in advance. input test data: 2 2010-02-16 10:00:00 111111111111 bytes 99999999999 bytes 90% 4 2010-02-16 12:00:00 333333333333 bytes 77777777777 bytes 88% 5 2010-02-16 11:00:00... (4 Replies)
Discussion started by: ux4me
4 Replies

10. Shell Programming and Scripting

Transpose Rows

Hi, Am trying to transpose a set of rows into a set of comma separated values. For eg. if the output of ps -ef | tail +2 | awk 'BEGIN{ FS=" " } { print $2 }' is 0 1 3 4 I need to transpose it to - '0','1','3','4' Am currently trying - (4 Replies)
Discussion started by: iamwha1am
4 Replies
Login or Register to Ask a Question