Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Transpose whole file and specific columns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Transpose whole file and specific columns
# 1  
Old 07-17-2012
Transpose whole file and specific columns
Hi,

I have a file like this

Code:
a b c
d e f
g h i
j k l

Case1:
I want to transpose the whole file

Output1
Code:
a d g j
b e h k
c f i l

Case2
Transpose a specific column - Say 3rd

Output2
Code:
c f i l

Case3
Transpose a range of columns - say 1 and 2

Output3
Code:
a d g j
b e h k

Case4
Transpose specific multiple columns - say 1 and 3

Output
Code:
a d g j
c f i l

Initally, I was helped in this forum for transposing a single column of this kind

input
Code:
a
b
c
d

using

Code:
tr '\n' ',' < input.txt

Output
Code:
a,b,c,d

# 2  
Old 07-17-2012
Code:
#!/bin/bash 
# 
# transpose 
 
transpose() 
{ 
  awk ' 
      { 
              if(max_nf<NF) 
                 max_nf=NF 
              max_nr=NR 
              for(x=1; x<=NF; ++x) 
                  matrix[x, NR]=$x 
      } 
 
  END { 
              for(x=1; x<=max_nf; ++x) { 
                  for (y=1; y<=max_nr; ++y) 
                       printf("%s ", matrix[x, y]) 
                  printf("\n") 
              } 
      }'  ${1} 
} 
 
transpose ${1}

for case1:
Code:
bash transpose infile

for any other cases:
Code:
awk '{print $1,$2,...,$n}' infile | bash transpose

These 2 Users Gave Thanks to complex.invoke For This Post:
aster007 jacobs.smith
# 3  
Old 07-18-2012
awk
you may try below, let's see it is transpose.sh

transpose the whole file
!>transpose.sh file

transpose the file with column range 1 to 3
!>transpose.sh file 1,3

transpose the file with column list 1, 3 and 5
!>transpose.sh file 1 3 5


Code:
file=$1
shift
function transpose()
{
    tmp_file=$1
awk '{
    for(i=1;i<=NF;i++)
      _[i]=sprintf("%s %s",_[i],$i)
}
END{
    for(i in _)
        print _[i]
}' $tmp_file
}

if [ $# -eq 0 ];then
echo "beginning to transpose the whole file..."
transpose $file
elif [ $# -eq 1 ];then
range=$1
echo "beginning to transpose range $range ..."
cmd='cut -d" "  -f'${range}' '${file}'>'${file}'.tmp'
eval $cmd
transpose ${file}.tmp
rm ${file}.tmp
elif [ $# -gt 1 ];then
while [ $# -gt 0 ];do
if [ $list ];then
list=${list}","$1
else
list=$1
fi
shift
done
echo "beginning to transpose list of columns $list ..."
cmd='cut -d" "  -f'${list}' '${file}'>'${file}'.tmp'
eval $cmd
transpose ${file}.tmp
rm ${file}.tmp
fi

This User Gave Thanks to summer_cherry For This Post:
jacobs.smith
# 4  
Old 07-18-2012
Quote:
Originally Posted by huaihaizi3
Code:
#!/bin/bash 
# 
# transpose 
 
transpose() 
{ 
  awk ' 
      { 
              if(max_nf<NF) 
                 max_nf=NF 
              max_nr=NR 
              for(x=1; x<=NF; ++x) 
                  matrix[x, NR]=$x 
      } 
 
  END { 
              for(x=1; x<=max_nf; ++x) { 
                  for (y=1; y<=max_nr; ++y) 
                       printf("%s ", matrix[x, y]) 
                  printf("\n") 
              } 
      }'  ${1} 
} 
 
transpose ${1}

for case1:
Code:
bash transpose infile

for any other cases:
Code:
awk '{print $1,$2,...,$n}' infile | bash transpose

Thanks for ur time. No output is being generated.

---------- Post updated at 02:57 PM ---------- Previous update was at 02:55 PM ----------

Quote:
Originally Posted by summer_cherry
you may try below, let's see it is transpose.sh

transpose the whole file
!>transpose.sh file

transpose the file with column range 1 to 3
!>transpose.sh file 1,3

transpose the file with column list 1, 3 and 5
!>transpose.sh file 1 3 5


Code:
file=$1
shift
function transpose()
{
    tmp_file=$1
awk '{
    for(i=1;i<=NF;i++)
      _[i]=sprintf("%s %s",_[i],$i)
}
END{
    for(i in _)
        print _[i]
}' $tmp_file
}

if [ $# -eq 0 ];then
echo "beginning to transpose the whole file..."
transpose $file
elif [ $# -eq 1 ];then
range=$1
echo "beginning to transpose range $range ..."
cmd='cut -d" "  -f'${range}' '${file}'>'${file}'.tmp'
eval $cmd
transpose ${file}.tmp
rm ${file}.tmp
elif [ $# -gt 1 ];then
while [ $# -gt 0 ];do
if [ $list ];then
list=${list}","$1
else
list=$1
fi
shift
done
echo "beginning to transpose list of columns $list ..."
cmd='cut -d" "  -f'${list}' '${file}'>'${file}'.tmp'
eval $cmd
transpose ${file}.tmp
rm ${file}.tmp
fi

Code:
$ cat test
a b c
d e f
g h i
j k l

$ ./code.sh test
beginning to transpose the whole file...
 b e h k
 c f i l
 a d g j

$ ./code.sh test 1,2
beginning to transpose range 1,2 ...
 b e h k
 a d g j

$ ./code.sh test 1 3
beginning to transpose list of columns 1,3 ...
 c f i l
 a d g j

$ ./code.sh test 2 3
beginning to transpose list of columns 2,3 ...
 c f i l
 b e h k

The only problem is the shift in order. Thanks a lot. It works great.
# 5  
Old 07-18-2012
Quote:
Originally Posted by jacobs.smith
Thanks for ur time. No output is being generated.
Code:
[root@node3 ~]# cat infile 
a b c
d e f
g h i
j k l

case1:
Code:
[root@node3 ~]# bash transpose infile 
a d g j 
b e h k 
c f i l

case2:
Code:
[root@node3 ~]# awk '{print $3}' infile | bash transpose 
c f i l

case3:
Code:
[root@node3 ~]# awk '{print $1,$2}' infile | bash transpose 
a d g j 
b e h k

case4:
Code:
 
[root@node3 ~]# awk '{print $1,$3}' infile | bash transpose 
a d g j 
c f i l

Don't be so rigid, for dealing with technology issues, we must be more flexible!
These 2 Users Gave Thanks to complex.invoke For This Post:
jacobs.smith mathbalaji
# 6  
Old 07-19-2012
Quote:
Originally Posted by huaihaizi3
Code:
transpose ${1}

That's a poor job of parameter handling. Without double-quoting, the script cannot handle filenames containing whitespace or pathname pattern-matching metacharacters (*, ?, [...]).

You can double-quote the parameter, but, if the script isn't provided an argument, $1 will expand to a zero-length string instead of no field at all. There's a difference between the two and your script depends on this difference to default to reading stdin.

Since AWK is required to support - as an explicit reference to standard input, it makes sense to use it as a default value when $1 is unset (assuring that the script can function as a filter in a pipeline). "${1--}" will do this and will properly handle whitespace and pattern matching characters in a pathname. It would also be a good idea to exit with an error if the filename argument is provided but null.

Quote:
Originally Posted by summer_cherry
Code:
file=$1
transpose $file
...
cmd='cut -d" "  -f'${range}' '${file}'>'${file}'.tmp'
eval $cmd
...
cmd='cut -d" "  -f'${list}' '${file}'>'${file}'.tmp'
eval $cmd
transpose ${file}.tmp
rm ${file}.tmp

This script suffers from the same problems as the other, but the damage is exacerbated by the incorrect use of eval.

In addition to whitespace, your eval approach is vulnerable to filenames with all kinds of shell metacharacters (to name a few): ;&*?"{(['. Aside from the possiblity of unintentionally running a command (possibly in the background), whitespace will cause field splitting to pass a partial, incorrect filename to cut and to the redirection operator.

Worst of all is that these bugs are the result of unnecessary complexity. There's nothing about ...
Code:
cmd='cut -d" "  -f'${range}' '${file}'>'${file}'.tmp'

... that requires eval; nothing at all. Just run it directly:
Code:
cut -d ' ' -f "${range}" "${file}" > "${file}.tmp"

Regards,
Alister
Last edited by alister; 07-19-2012 at 02:20 PM..
This User Gave Thanks to alister For This Post:
# 7  
Old 07-19-2012
Quote:
Originally Posted by huaihaizi3
Code:
[root@node3 ~]# cat infile 
a b c
d e f
g h i
j k l

case1:
Code:
[root@node3 ~]# bash transpose infile 
a d g j 
b e h k 
c f i l

case2:
Code:
[root@node3 ~]# awk '{print $3}' infile | bash transpose 
c f i l

case3:
Code:
[root@node3 ~]# awk '{print $1,$2}' infile | bash transpose 
a d g j 
b e h k

case4:
Code:
 
[root@node3 ~]# awk '{print $1,$3}' infile | bash transpose 
a d g j 
c f i l

Don't be so rigid, for dealing with technology issues, we must be more flexible!
Thanks. It works now. Donno what happened earlier. Sorry.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Transpose rows to certain columns

Hello, I have the following data and I want to use awk to transpose each value to a certain column , so in case the value is not available the column should be empty. Example: Box Name: BoxA Weight: 1 Length :2 Depth :3 Color: red Box Name: BoxB Weight: 3 Length :4 Color: Yellow... (5 Replies)
Discussion started by: rahman.ahmed
5 Replies

2. Shell Programming and Scripting

Transpose columns to row

Gents Using the attached file and using this code. awk '{print substr($0,4,2)}' input.txt | sort -k1n | awk '{a++}END{for(i in a) print i,a}' | sort -k1 > output i got the this output. 00 739 01 807 02 840 03 735 04 782 05 850 06 754 07 295 08 388 09 670 10 669 11 762 (8 Replies)
Discussion started by: jiam912
8 Replies

3. UNIX for Dummies Questions & Answers

Transpose matrix, and rearrange columns common with another file

This is my first post, I apologize if I have broken rules. Some assistance with the following will be very helpful. I have a couple of files, both should ultimately have common columns only, arranged in the same order. This file needs to be transposed, to bring the rows to columns ... (2 Replies)
Discussion started by: abh.kumar
2 Replies

4. Shell Programming and Scripting

To transpose columns + edit in a txt file

Hi, I have a txt file that looks like log2FoldChange Ontology_term 8.50624450251828 GO:0003700,GO:0003707,GO:0005634,GO:0006355,GO:0043401,GO:0003700,GO:0005634,GO:0006355,GO:0008270,GO:0043565 7.03936870356684 GO:0005515,GO:0008080 6.49606183738682 6.49525073909629 GO:0005515... (4 Replies)
Discussion started by: alisrpp
4 Replies

5. Shell Programming and Scripting

Transpose few columns alone in a Text file

Hi Friends, I am Stuck up with a problem on transposing Rows to Coloumns.. Though there are many threads on this my problem is little difficult.. I have a tab separated file like Below, computer selling_loc currency_type manufacturer_name salesweek-wk1 sales-wk2 ...wk-3 ..wk4 till... (7 Replies)
Discussion started by: heinz_holan
7 Replies

6. Shell Programming and Scripting

transpose rows to columns

Any tips on how I can awk the input data to display the desired output per below? Thanking you in advance. input test data: 2 2010-02-16 10:00:00 111111111111 bytes 99999999999 bytes 90% 4 2010-02-16 12:00:00 333333333333 bytes 77777777777 bytes 88% 5 2010-02-16 11:00:00... (4 Replies)
Discussion started by: ux4me
4 Replies

7. Shell Programming and Scripting

Transpose columns to Rows

I have a data A 1 B 2 C 3 D 4 E 5 i would like to change the data A B C D E 1 2 3 4 5 Pls suggest how we can do it in UNIX. Start using code tags, thanks. Also start reading your PM's you get from Mods as well read the Forum Rules. That might not do any harm. (24 Replies)
Discussion started by: aravindj80
24 Replies

8. Shell Programming and Scripting

Transpose Rows Into Columns

I'm aware there are a lot of resources dedicated to the question of transposing rows and columns, but I'm a total newbie at this and the task appears to be beyond me. I have 40 text files with content that looks like this: Dokument 1 von 146 Orange County Register (California) June 26, 2010... (2 Replies)
Discussion started by: spindoctor
2 Replies

9. Shell Programming and Scripting

Replace specific columns in one file with columns in another file

HELLO! This is my first post here! By the way, I think it is great that people do this. My question: I have two files, one is a .dilm and one is a .txt. It is my understanding that the .dilm file can be treated as a .txt file. I wrote another program where I was able to manipulate it as if it... (3 Replies)
Discussion started by: mehdib
3 Replies

10. Shell Programming and Scripting

Rows to Columns - File Transpose

Hi I have an input file and I want to transpose it but I need to take care that if any field is missing for a record it should be popoulated with space for that field - using a shell script INFILE ---------- emp=1 sal=2 loc=abc emp=2 sal=21 sal=22 loc=xyz emp=5 loc=abc OUTFILE... (10 Replies)
Discussion started by: 46019
10 Replies
Login or Register to Ask a Question