Bash formatting data into columns


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash formatting data into columns
# 1  
Old 12-16-2014
Bash formatting data into columns

Hi guys,

I'm trying to create a table of aggregated data using just bash commands. My data is in three columns, for example:

Code:
 2014-01-01  testA  64
 2014-01-01  testB  27
 2014-02-01  testA  31
 2014-02-02  testB  29
 2014-02-02  testC  12

And the result I am looking for is:

Code:
 2014-01-01  64    27    null
 2014-02-01  31    null  null
 2014-02-02  null  29    12

where columns 2, 3 and 4 map to testA, B or C.

So in effect, the first column is the y-axis, the next column is the x-axis label, and the final column is the value.

In other words, I'm trying to group on the first two columns using the third column as the value.

I'm thinking an array might be the best way to solve this problem, but I can't get my head around where to start.

Any ideas anyone?

Thanks,

Chris
# 2  
Old 12-16-2014
Try
Code:
awk     '       {LN[$1]; HD[$2]; MX[$1,$2]=$3}
         END    {               printf "%10s", ""; for (i in HD) printf "%10s", i; print "";
                 for (j in LN) {printf "%10s",j;   for (i in HD) printf "%10s", MX[j,i]; print ""}
                }
        ' file
               testA     testB     testC
2014-01-01        64        27          
2014-02-01        31                    
2014-02-02                  29        12

These 2 Users Gave Thanks to RudiC For This Post:
# 3  
Old 12-16-2014
Just some crude looking code, for a possible start (without awk)
Code:
copy file to file1
// loop thru while data in file
while ($wc -l file1 > 0)
 do
// extract data from that first column
 matchf = head -1 file1 | cut -f1
// pull out matching lines
 grep $matchf < file1 > file3
// now temp file3 has common entries for first unique column value
// put these into your column format
// write these to file4
//
// copy everything but that first column match to file2
 grep -v $matchf <file1 >file2
// copy temp file back to working file1
 cp file2 file1
 done

# 4  
Old 12-16-2014
RudiC's approach in bash using Associative Arrays:-
Code:
#!/bin/bash

declare -A LN
declare -A HD
declare -A MX

while read v1 v2 v3
do
        LN["$v1"]="$v1"
        HD["$v2"]="$v2"
        MX["${v1}${v2}"]="$v3"
done < filename

for j in "${LN[@]}"
do
        printf "%10s" "$j"
        for k in "${HD[@]}"
        do
                [ -z "${MX[${j}${k}]}" ] && printf "%10s" "null" || printf "%10s" "${MX[${j}${k}]}"
        done
        printf "\n"
done

# 5  
Old 12-17-2014
Thanks guys, it's amazing how many times this requirement has come up, and I've never been able to figure out how to approach the problem.

Appreciate the quick assistance from you all.

Cheers,

Chris
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl syntax for formatting columns and text

Dear all, Pzl let me know what is the syntax for converting the columns format to text as i have lots of values for but when i put these values in xls sheet the values are automatically converted to and one more question i have is how to call values from shell script into perl script eg. ... (3 Replies)
Discussion started by: sagar_1986
3 Replies

2. Shell Programming and Scripting

Formatting columns in file

I have a script that creates a file with three column output. The columns look like this: Policy Name Name Volume VLS-EDWARDS-ARCHIVED_FILE-WINDOWS 10 12 XLZ-BASE-CYGWIN-ARCHIVED-FILE-LINUX 2 21 ZLX-GOLD-FILES-JAPANFILERS1-LINUX 20 27 ZLX-GOLD-FILES-JAPANFILERS2-LINUX13 29... (4 Replies)
Discussion started by: newbie2010
4 Replies

3. Shell Programming and Scripting

formatting columns

Hi I have a file that is several gigs in size and basically I want to change the format of it. Obviously I cannot go through it manually. The file looks like this: 897 2 901 2 905 2 909 2 913 2 917 2 921 2 925 2 929 2 933 2 937 2 941 2 945 2 949 0 (1 Reply)
Discussion started by: kylle345
1 Replies

4. UNIX for Advanced & Expert Users

formatting the data

HI I want to make it single row if start with braces i.e. { .Any idea {1:XXX2460275191}{2:SEC00687921131112201641N}{3:{58910}}{4: :R:GENL :C::xx//xx1 :20C::yy//yy1 :2S:xxT} {1:XXX2460275190}{2:SEC00687921131112201641y}{3:{58911}}{4: :z:GENL :v::xx//xx1 :10C::yy//yy1 :4S:xxT ... (2 Replies)
Discussion started by: mohan705
2 Replies

5. Shell Programming and Scripting

Formatting output in columns

I have a file which contains data in below format: nbkv28s MgmtReporting -> TradingDesk 1 nbkv28s RMBS -> Credits 178 nbkv28s RMBS -> PassThrough 96 nbkv28s RMBS -> Prepayment 111 nbkv28s RMBS -> RMBSHome 370 nbkv28s RMBS -> TradingStrategy 98 nbkvnze RMBS -> RMBSHome 85 nbkvugn GSF... (3 Replies)
Discussion started by: mohsin.quazi
3 Replies

6. UNIX for Dummies Questions & Answers

Formatting a line of data into columns

Hi all, I'm a little stuck with a data file I've been collecting data in. The file contains one field of data running continuously down the file and I can't work out how to format the data into three columns. This is a mock up of the file: Each r# is a random number and varies in length, this... (3 Replies)
Discussion started by: nistleloy
3 Replies

7. UNIX for Dummies Questions & Answers

Formatting Data

i am writing a script that reads in a file and i just want it to print each element on a new line here is my code and the data that i want to read in #!/usr/bin/perl use strict; use CGI qw(:standard); use CGI qw(:cgi); my $data_file = "/tmp/results.txt"; my $configuration; my... (3 Replies)
Discussion started by: nmeliasp
3 Replies

8. Shell Programming and Scripting

Formatting info into columns

Hi This question has kind of been asked before but I couldn't get any of the solutions to work. I need to format fields being cut from a file into columns so it looks like a table. e.g Full name Address Hiredate Joe Smith London 11.01.01 Bill King ... (8 Replies)
Discussion started by: straight_edge
8 Replies

9. UNIX for Dummies Questions & Answers

Text formatting to 132 columns

I want to take a file that is generated in 132 columns from a unix based report generator and transfer it to a windows environment and preserve the column length. Any ideas how I can do this so that whatever windows based text editor that reads the document will know it should be in 132 columns?... (16 Replies)
Discussion started by: jmossman
16 Replies
Login or Register to Ask a Question