Converting tables of row data into columns of tables


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Converting tables of row data into columns of tables
# 1  
Old 07-13-2007
Converting tables of row data into columns of tables

I am trying to transpose tables listed in the [Input] format into [Output] format. Any help would be greatly appreciated.

Input:
Code:
test_data_1              
1  2  90%
4  3  91%
5  4  90%
6  5  90%
9  6  90%

test_data_2              
3  5  92%
5  4  92%
7  3  93%
9  2  92%
1  1  92%
...

Output:
Code:
test_data_1           test_data_2       ...
1  2  90%             3  5  92%         ...
4  3  91%             5  4  92%         ...
5  4  90%             7  3  93%         ...
6  5  90%             9  2  92%         ...
9  6  90%             1  1  92%         ...

Thanks much,

Chris Larson
Just this guy.
# 2  
Old 07-13-2007
Code:
paste file1 file2 file3 ...

# 3  
Old 07-13-2007
Reborg,

Thanks. I should have clarified that the input data resides in a single file.

Also, I'm not sure how paste is going to help place the tables into a columnar layout. What I'm looking for is a way to take a single file in the first format and end up with a file containing the date tables side by side [by side by side etc.].

In other words, the [Output] section is an example of what I would like the output file to look like.

Chris Larson
JustThisGuy
# 4  
Old 07-13-2007
Is that the actual format of the data and not a simplified version?
# 5  
Old 07-13-2007
That's the actual format. There are occasional tables with 2 elements per line, rather than 3, as the example below, but other than that, it's just as I posted above.

Code:
test_data_9
2  95%
3  94%
4  91%
5  92%
6  93%
7  90%

# 6  
Old 07-13-2007
Hi.

Here is one approach:
Code:
#!/bin/sh

# @(#) s1       Demonstrate context splitting, pasting, adjusting columns.

set -o nounset
echo
echo "GNU bash $BASH_VERSION" >&2
csplit --version | head -1 >&2
echo

FILE=${1-data1}
csplit -k -z $FILE /^test_data/ "{*}"

echo
echo " Lines per file:"
wc -l xx*

echo
paste xx* |
column -s"      " -t
# Note: there is a TAB inside -s" ".

exit 0

which, on your data file (with extra line in data set 9) produces:
Code:
% ./s1

GNU bash 2.05b.0(1)-release
csplit (coreutils) 5.2.1

76
76
54

 Lines per file:
  6 xx00
  6 xx01
  7 xx02
 19 total

test_data_1                test_data_2                test_data_9
1  2  90%                  3  5  92%                  2  95%
4  3  91%                  5  4  92%                  3  94%
5  4  90%                  7  3  93%                  4  91%
6  5  90%                  9  2  92%                  5  92%
9  6  90%                  1  1  92%                  6  93%
7  90%

See man pages for details on the options ... cheers, drl
# 7  
Old 07-16-2007
Thank y'all! drl, I largely used your example, thank you much for taking the time!

::beer:: >> drl

I've pasted a commented version of the guts of my solution, should anyone else have the same or similar questions.

It's rough edged (I need to sit down and work on handling the indenting of the header in cases other than 3 element data sets), but functional.

In the scenario I posted above, this script (let's name it 'massagetocolumn.sh') would be called as follows:

./massagetocolumn.sh data.file test_data_

Chris Larson
JustThisGuy

Code:
#!/bin/sh

# Exit if any variable is not set.
set -o nounset

# Input file:
DATA_FILE=${1}
echo "Data File: "$DATA_FILE

# Dataset Header Prefix.
HEADER_PREFIX=${2}
echo "Header Prefix: "$HEADER_PREFIX

# Make columns from space-delimited file.
# -e indicates a command, several of which can be included in one
# sed call. s indicates the string to search for, which is prefixed
# by '/'. The replacement string is prefixed by the second '/' and
# closed with a final '/'.
# Following the list of commands is the input file.
# ' > $DATA_FILE.temp" directs the output to an output file,
# in this case with '.temp' added to the filename.
# This file is removed when the script finishes. In this
# script, the 's/[[ ]]*/\t/' is finding all spaces and replacing them
# with TAB (\t). The 's/$HEADER_PREFIX\S*/&\t\t/' is finding all strings beginning
# with '$HEADER_PREFIX' and appending two tabs after each occurrence.
# Oh, and the 'g' tells sed to replace all occurrences, not just the first occurrence
# per line, which is the default behavior.
sed -e 's/[[ ]]*/\t/g' -e 's/$HEADER_PREFIX\S*/&\t\t/' $DATA_FILE > $DATA_FILE.temp

# Cut datasets from input file into separate temporary files, named as xx##.
# The '-k' option leaves the temp files in place in the case of an error.
# The '-s' option silences the default byte counts that csplit offers.
# The '-z' option deletes any output files that are empty.
# csplit cuts the data sets based on the search string, in this case:
# whatever you put as the second argument to the script.
csplit -k -s -z $DATA_FILE.temp /^$HEADER_PREFIX/ "{*}"

# Paste temporary files into output, piped through 'column' to create columns.
# NOTE: there is a TAB inside -s" ".
# The 'paste' command pastes multiple files into one, with the contents of all files side by side.
paste xx* | column -s"      " -t > $DATA_FILE.out

#Remove the temporary files.
rm xx*
rm $DATA_FILE.temp

# Exit
exit 0

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script writing for tables with binary data

Hi There, I'm trying to write a simple script that will email me when we have an application job in a certain status that needs human intervention. I've used this script for other tables and it works great. However, this one gives me the warning that there is binary data so it might not. ... (2 Replies)
Discussion started by: Colel2
2 Replies

2. Shell Programming and Scripting

UPDATE COmmand post comparing 2 columns in 2 mysql tables

my queryis : select distinct m.name, item_count, item from master m join client p on m.name=p.name where item_count = 1 and item > 1; But how should I update them? i used update statetment : Update from client Set item =1 where m.name=p.name and item_count=1 AND item>1 Is this wrong? (1 Reply)
Discussion started by: siya@
1 Replies

3. Shell Programming and Scripting

Append data by looking up 2 tables for multiple files

I want to lookup values from two different tables based on common columns and append. The trick is the column to be looked up is not fixed and varies , so it has to be detected from the header. How can I achieve this at once, for multiple data files, but lookup tables fixed. The two lookup... (5 Replies)
Discussion started by: ritakadm
5 Replies

4. Shell Programming and Scripting

Getting input data from 2 tables

Hello All, I have one table contains: Table1: 5 10 30 40 60 80 ... Table 2: 10 20 60 80 (7 Replies)
Discussion started by: krsnadasa
7 Replies

5. Shell Programming and Scripting

Extracting data from tables......

HOw to extracts data from tables in database. Merges them into one output file. This output file is loaded into another tables in database. (1 Reply)
Discussion started by: nari.bommi
1 Replies

6. Shell Programming and Scripting

Help converting row data to columns

I've been trying to figure this out for a while but I'm completely stumped. I have files with data in rows and I need to convert the data to columns. Each record contains four rows with a "field name: value" pair. I would like to convert it to four columns with the field names as column headers... (5 Replies)
Discussion started by: happy_ee
5 Replies

7. Shell Programming and Scripting

Shell script for comparing data of tables

Hi, I have two databases with same tables on different servers.I need to check the data content in each table and if something is missing, should print that. I have a tool which takes the snapshot the table structure,index so on and compares with the other server tables snapshot. Now i need... (1 Reply)
Discussion started by: nessj
1 Replies

8. UNIX Desktop Questions & Answers

Extracting data from tables and storing in a file

Hi I am trying to write a script to extract information from a database and save that info to a csv file. I am using sql, an oracle database I have no idea how to even begin this. Can somebody please help. (8 Replies)
Discussion started by: ladyAnne
8 Replies

9. Shell Programming and Scripting

Reading data from multiple tables from Oracle DB

Hi , I want to read the data from 9 tables in oracle DB into 9 different files in the same connection instance (session). I am able to get data from one table to one file with below code : X=`sqlplus -s user/pwd@DB <<eof select col1 from table1; EXIT; eof` echo $X>myfile Can anyone... (2 Replies)
Discussion started by: net
2 Replies

10. UNIX for Dummies Questions & Answers

extract data from html tables

hi i need to use unix to extract data from several rows of a table coded in html. I know that rows within a table have the tags <tr> </tr> and so i thought that my first step should be to to delete all of the other html code which is not contained within these tags. i could then use this method... (8 Replies)
Discussion started by: Streetrcr
8 Replies
Login or Register to Ask a Question