Ok, so I was able to run this last night on a test file:

Code:

a,a,1
a,b,0.8
a,c,0.5
b,b,1
b,c,0.2
c,c,1

It returned:

Code:

,a,b,c
,0.5
,0.2
c,1

so the first line looks good, meaning 't' is correct but the subsequent line of code isn't quite right.

Running without the final x[i]:

Code:

awk -F, 'BEGIN {OFS = ","}
{
   t = t ? (l != $1 ? t OFS $1 : t) : OFS $1
   x[$1] = x[$1] ? x[$1] OFS $NF : $NF
   l = $1
} END {
   print t
   for (i in x) print i
}' file

returns:

Code:

,a,b,c
a
b
c

which is what we want for the first row and the first column.

Shamrock, do you have any thoughts on what we could do to the 'x' variable to create the correlation matrix?

Thanks very much for your help.

---------- Post updated at 10:11 AM ---------- Previous update was at 09:42 AM ----------

If we process our data in one batch, the matrix will have 5,122 rows and columns. If there are inherent line limitations that prevent this, we'll come up with ways to chunk the data into smaller batches. Presumably, there will be a handful of empty matrix elements where the original correlation analyses failed to converge.

There are no multiple entries with indexes of the same order - that is

Code:

x, y, value

will not repeat. However,

Code:

x, y, value
and
y, x, value

are both present if we do not reduce our data. shamrock and birei already provided means to remove these redundancies. Furthermore, shamrock's awk program in this thread that we are currently working on is more problematic if

Code:

x, y, value
y, x, value

are both present in the input data, even if the values are identical.

Thanks bakunin, this does indeed help, as we will pursue something along these lines if we cannot get shamrock's awk program to make the correlation matrix.