Thanks a lot, Tyler! It works out perfectly with small matrix.
Yes, the memory is an issue for me with the 25000x25000 matrix. I will try to use other pc with more RAM. Thank you very much again!
---------- Post updated at 09:08 PM ---------- Previous update was at 08:58 PM ----------
The matrix is the correlation coefficiencies of the expression level of ~25000 genes of the genome. Some genes express similarly that can be indicated by high correlation coefficiency, but most of them not. The ID is used to track the gene name of the genome, and to find the pattern of expression.
Say, if I want see which genes are expressing similarly with 244901_AT, GREP it would give the single row of the correlation coefficiencies. Theoretically, the half matrix contains all the information of the square one, but it is hard for me to retrieve any specific gene(s) of my interest. By the way, I am a geneticist and just started trying programming.
I can't imagine that it would be an easy thing to just grep some gene ID and look at a line with 25000 values. But it seems like that's what you want to see. ???
How high of a correlation coefficient is enough to be considered as expressing similarly? And would it be fair to say that, given a specific gene name, you just want to know which other genes, if any, express similarly? Or, given a gene name and a specified coefficient, you want to know which other genes have a coefficent greater than or equal to the specified value?
Yes, the memory is an issue for me with the 25000x25000 matrix. I will try to use other pc with more RAM.
A little bit of calculation is in order. Your actual text file is probably a huge one.
There's an ID of 11 characters, followed by 25000 tokens, each of 6 characters in line 1, 24999 in line 2, and so on. So we are looking at at a file of size 1.88 GB approximately.
The perl scripts are going to use approximately this + some more memory for temporary operations. And once done, your final file would be about twice that size ~ 3.75 GB.
That's a huge size for a text file and it's going to be a challenge to process such a file.
If you are trying this out on your home desktop/laptop, then I'd imagine you need one of those computers with 3 - 4 GB RAM (Gamer's Laptops ? - the ones that are optimized for computer games). Otherwise you may want to try it out in a server at your work place.
Finally, you may want to consider not storing the other half of your matrix. A simple script can generate the entire line of the square matrix, given a line number.
I have two files. Row id in File1 matches the column id in file2 (starting from column7 )except the last 2 characters. File1 has 50 rows and File 2 has 56 columns. If the id matches I want to multiply the value in column3 of File1 to the entire column in File2. and in the final output print only... (11 Replies)
A big question for the UNIX INTELLIGENCE
I have a CSV file as follows:
0,userF,userH,userG... (15 Replies)
I want to print this charts diagonal data into straight lines.
This is a matrix 24X24 Horizontal and vertical.
- I want to print all the diagonal cutting characters into straight line:
E F S S A H A L L A T M C N O T S O B O D U Q H
I W I B N L O C N I L N L A N S I N... (9 Replies)
I am quite new in this but I need some help to keep going with my analysis.
I am struggling with a short script to read a square matrix and convert it in two collumns.
A B C D
A 0.00 0.06 0.51 0.03
B 0.06 0.00 0.72 0.48
C 0.51 0.72 0.00 ... (7 Replies)
I have a table in the format:
1 0 -1 1 0
2 0 1 -1 0 0 0
3 0 1 1 0 0 0 0 0 0
I am trying to input this to a program, however it is complaining about the fact that it is not in matrix format. How do I add 0's to end of the rows to make them even?
Thanks in advance! (2 Replies)
I am very new at awk but think that that might be the best strategy for this. I have a matrix very similar to a correlation matrix and in practical terms I need to convert it into a list containing the values from the matrix (one value per line) with the first field of the line (row... (5 Replies)
Is there a way to convert full data matrix to linearised left data matrix?
e.g full data matrix
Bh1 Bh2 Bh3 Bh4 Bh5 Bh6 Bh7
Bh1 0 0.241058 0.236129 0.244397 0.237479 0.240767 0.245245
Bh2 0.241058 0 0.240594 0.241931 0.241975 ... (8 Replies)
I would like to merge multiple files with the same row and column size into a matrix format
In a folder I have multiple files in the following format
a 4 (7 Replies)