Extract values from a matrix given the rows and columns
Hi All,
I have a huge (and its really huge!) matrix about 400GB in size (2 million rows by 1.5 million columns) . I am trying to optimize its space by creating a sparse representation of it.
Miniature version of the matrix looks like this (matrix.mtx):
So, this is what I have done until now.
1. I got the important rows and columns from another means not by processing this great matrix, those rows and columns which I really care about, and have those rows and columns stored in another text file called row_column.tmp
My row_column.tmp looks like this:
So, this means first row and first column is really important to me and I would like to extract the value from the huge matrix and make my output file look like this:
output.mtx
The above output shows that by reading in the rows and columns from the row_column.tmp, I go to the main matrix file matrix.mtx and extract the value from that particular row and column and put the value against that row and column in my output.mtx file.
Things which I need to care about is that I should not load the entire matrix in memory else things will get really messy. I am using Linux with BASH.
Yes, you can do it in awk and it's not difficult. But I very seriously doubt that awk and shell is appropriate tools for processing 400GB files. Just try to time some very simple awk script, ie:
I believe It may take days or weeks. I suggest to use any compiled language, use available parallel tools and convert your file to noSQL database (with numbers, not strings) before processing. Or maybe use some specialized tools/languages like Matlab (or Octave).
If its days or weeks, then probably I cannot afford that. I'll write my C program then and run it in parallel. MATLAB in the first place gave up "Not Enough Memory" This is a computational challenge, I believe and C can handle this very well. I'll paste my C code here when I am done with that.
Well, I've never dealt with data files of such sizes.
Before starting to code try to find all possible information about processing this kind of data. The best of course if you could find someone who really worked with huge matrices stored in text files.
This is my first time too that's why facing many computational bottlenecks. But its fun at the end of the day
One possible solution is to split the matrix file and do parallel processing on those split files.
Another which I am currently doing is to do away with the matrix file itself and change my source program which actually created the matrix file to create the sparse file. This matrix is really really huge..blew off all my disk space too
---------- Post updated at 04:47 PM ---------- Previous update was at 11:07 AM ----------
I just did some tweaks to my C program and made it more efficient. Instead of creating that huge matrix file, I read in the files with the rows and columns information
I then read wrote my program in such a way, that I get the output as I have given above:
and BINGO!!! It worked pretty much well and occupied just 600MB of disk space and the program took just few minutes to execute whereas my last program that generated the BIG matrix file ran for the entire night. I am not posting my C code here as it won;t make sense and people cannot understand what the entire purpose of the program is.
Hello
I have a file like this:
> cat examplefile
ghi|NN603762|eee
mno|NN607265|ttt
pqr|NN613879|yyy
stu|NN615002|uuu
jkl|NN607265|rrr
vwx|NN615002|iii
yzA|NN618555|ooo
def|NN190486|www
BCD|NN628717|ppp
abc|NN190486|qqq
EFG|NN628717|aaa
HIJ|NN628717|sss
>
I can sort the file by... (5 Replies)
Hallo,
I would need to extract only rows which has different value in the second and third column.
Thank you very much for any advices
Input:
A 0 0
B 0 1
C 1 1
D 1 3
Output
B 0 1
D 1 3 (4 Replies)
Hi all, I'm pretty much a newbie to UNIX. I would appreciate any help with UNIX coding on comparing two large csv files (greater than 10 GB in size), and output a file with matching columns.
I want to compare file1 and file2 by 'id' and 'chain' columns, then extract exact matching rows'... (5 Replies)
Hi I have a matrix with n rows and m columns like below example. i want to extract all the pairs with values <200.
Input
A B C D
A 100 206 51 300
B 206 100 72 48
C 351 22 100 198
D 13 989 150 100
Output format
A,A:200
A,C:51
B,B:100... (2 Replies)
Hi all,
I have a large matrix of 720 x 25. I want to get sum of range of rows and columns. Like, I need sum of all columns and row number 2 to 21, then leaving 22nd row, again sum of all columns and row number 23 to 42 again leaving 43rd row and then sum of 44th to 63. Means I want to add all... (4 Replies)
Hello,
I want to extract several columns and rows from a huge tab delimited file
for example: I want to print from from column 3 to 68 till row number 30.
I have tried using cut command but it was extracting whole 3rd and 68th column.
Please suggest a solution.
Ryan (8 Replies)
Hi
My pipe delimited .txt file contains rows with 10 columns.
Can anyone advise how I output to file only those rows with the letters ‘ci'
as the first 2 characters in the 3rd column ?
Many thanks (4 Replies)
Hello guys,
Please help me to solve this problem. I have tried some awk commands but couldn't succeed.
I have a tab delimited file where each record is separated by ------ and 4th column of each record is same.
<INPUT FILE>
------
peon 53931587 53931821 ... (12 Replies)
Hi guys I want to print the values by using this script but its giving the no of rows and columns as input instead of values
Would you plz help me on this
FILE- chr1.txt
1981 1
1971 1
1961 1
1941 1
perl script
#!/usr/bin/perl -w
$infile1 = 'chr1.txt';
$outfile3 = 'out3.txt';
... (3 Replies)