Rearranging Data Set


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Rearranging Data Set
# 1  
Old 12-02-2016
Rearranging Data Set

Hello everybody,

I've got the following problem:

The data set I have is an ASCII file containing a header over 4 lines and the actual data comprised of dezimal numbers in a 1000x1000 grid (1000 lines and 1000 columns).

Since I want to plot the data in GMT I need to convert it into the following format:

[x coordinate] [y coordinate] [value]

This means every single value will be in a single line preceeded by its coordinates. For simplicity lets say the upper left corner has coordinats 0,0 and the grid spacing is 500 in both directions.

I tried the following using "awk"

Code:
awk 'BEGIN {
xUL=0
yUL=0
for (i=1; i<=1000; i++)
      print "xUL+($i-1)*500", yUL+($i-1)*500, $i
}'

input.txt > output.txt

Since I'm posting my problem here it is obvious the the code above is not working. Can anybody tell why it is not working or a better solution?

Cheers
Evil

PS: I'm scripting in bash.
# 2  
Old 12-02-2016
You are running a loop from 1 to 1000 over columns which don't exist and lines haven't been read. BEGIN takes place before any data is read.

Further, awk does not process expressions inside strings.

That you have a loop from 1 to 1000 doesn't tell awk to read 1000 lines anyway. awk has its own built in loop, which it will use if you don't specifically force it not to (by using BEGIN or END).

I think you want something like this:

Code:
awk '{ for(N=1; N<=NF; N++) printf("%d %d %s\n", N, NR, $N); }' inputfile > outputfile

NF is the number of columns for the current line.
This User Gave Thanks to Corona688 For This Post:
# 3  
Old 12-02-2016
Welcome to the forum.

It is always beneficial to post sample input and output data, describe the underlying structure, show the (intended!) logics or algorithms that connect the two, and - in case of failures - what and where fails, and error messages.

In your case - which are the coordinates, and which are the values to plot? Is it [row No.] [col No.] field value? When looping across the fields, does it make sense to use $(i-1) and $i, and then only increment by 1 (which would mean you're using $i twice)?
# 4  
Old 12-02-2016
Thanks Corona688!

This is exactly what I needed! I was able to adjust your suggested Code to my needs and everything is working as intended. You guys are really fast.

Cheers
Evil
This User Gave Thanks to Evilknievel For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. Programming

C++ help in large data set

Hi All, We are trying to replace a 3rdparty where we don't know how they handled the reader part here. The query below is getting 197 * 2038017 row in the table. In the below code we are trying to run the query and execute in the DB part and fetch and read the record. That is where it is... (1 Reply)
Discussion started by: arunkumar_mca
1 Replies

2. Shell Programming and Scripting

Help with sum range of data set together

Input File: 2000 3 1998 2 1997 2 1994 1 1991 1 1989 1 1987 2 1986 2 1985 1 1984 1 . . 10 277256 9 278274 8 282507 7 284837 6 287066 5 292967 (4 Replies)
Discussion started by: perl_beginner
4 Replies

3. Shell Programming and Scripting

Data rearranging from rows to column

Hello Everyone, I have a input file looks like -0.005-0.004-0.003-0.002-0.00100.0010.0020.0030.0040.005My desired output should look like -0.005 -0.004 -0.003 -0.002 -0.001 0 0.001 0.002 0.003 0.004 0.005I had some success in getting the desired output. But i face a problem when i... (15 Replies)
Discussion started by: dinesh.n
15 Replies

4. Shell Programming and Scripting

Help with reformat data set

Input file 4CL1 O24145 CoA1 4CL1 P31684 CoA1 4CL1 Q54P77 CoA_1 73 O36421 Unknown 4CL3 Q9S777 coumarate 4CL3 Q54P79 coumarate 4CL3 QP7932 coumarate Desired output result 4CL1 O24145#P31684 CoA1 4CL1 Q54P77 CoA_1 73 O36421 Unknown 4CL3 Q9S777#Q54P79#QP7932 coumarate I... (5 Replies)
Discussion started by: perl_beginner
5 Replies

5. Shell Programming and Scripting

Help with sum of data set

Input file 2 1159,310, 4 142,199,218,91, 3 91,273,349, Desired output result 2 1469 4 650 3 713 I have long list of input file as shown above. It has a "," delimited to separate between each record in column 2.... (1 Reply)
Discussion started by: perl_beginner
1 Replies

6. Shell Programming and Scripting

Unable to set a data to array

Hi All, Iam trying to set the value to the array... Still its not happening Following is the code: #!/usr/bin/ksh filenames="x"; filenames="y"; echo $filenames; echo $filenames; O/P: x x Iam expecting (2 Replies)
Discussion started by: kiranlalka
2 Replies

7. Shell Programming and Scripting

rearranging the data in file (from columnwise to rowwise)

Hi I have one file which is having data like 10201 10202 10205 10206 10207 10208 10209 10210 10211 10213 10215 10801 10802 11406 11415 11422 11426 11513 11514 11515 11516 11517 11518 11519 11520 11521 11522 11523 11524 11525 11530 11604 11608 11611 11717 11718 11719 11722 11725... (3 Replies)
Discussion started by: reldb
3 Replies
Login or Register to Ask a Question