Sponsored Content
Top Forums Shell Programming and Scripting Extract values from a matrix given the rows and columns Post 302553123 by shoaibjameel123 on Tuesday 6th of September 2011 09:46:56 PM
Old 09-06-2011
Extract values from a matrix given the rows and columns

Hi All,

I have a huge (and its really huge!) matrix about 400GB in size (2 million rows by 1.5 million columns) . I am trying to optimize its space by creating a sparse representation of it.

Miniature version of the matrix looks like this (matrix.mtx):

Code:
3.4543 65.7876 54.564
2.12344 0.776565 4.563
1 4 7

So, this is what I have done until now.

1. I got the important rows and columns from another means not by processing this great matrix, those rows and columns which I really care about, and have those rows and columns stored in another text file called row_column.tmp

My row_column.tmp looks like this:
Code:
2 3
1 1
1 3
2 2
3 1

So, this means first row and first column is really important to me and I would like to extract the value from the huge matrix and make my output file look like this:

output.mtx

Code:
2 3 4.563
1 1 3.4543
1 3 54.564
2 2 0.776565
3 1 1

The above output shows that by reading in the rows and columns from the row_column.tmp, I go to the main matrix file matrix.mtx and extract the value from that particular row and column and put the value against that row and column in my output.mtx file.

Things which I need to care about is that I should not load the entire matrix in memory else things will get really messy. I am using Linux with BASH.

This is what I have done but not working:

Code:
awk -F' ' '
  NR == FNR { a[$1]=NR }
  NR != FNR {
    sub("row_column.tmp", "", FILENAME)
    print a[$2], FILENAME,  $1
  }
' matrix.mtx

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

perl script to print to values in rows and columns

Hi guys I want to print the values by using this script but its giving the no of rows and columns as input instead of values Would you plz help me on this FILE- chr1.txt 1981 1 1971 1 1961 1 1941 1 perl script #!/usr/bin/perl -w $infile1 = 'chr1.txt'; $outfile3 = 'out3.txt'; ... (3 Replies)
Discussion started by: nogu0001
3 Replies

2. Shell Programming and Scripting

Extract difference of two columns from different rows

Hello guys, Please help me to solve this problem. I have tried some awk commands but couldn't succeed. I have a tab delimited file where each record is separated by ------ and 4th column of each record is same. <INPUT FILE> ------ peon 53931587 53931821 ... (12 Replies)
Discussion started by: sam_2921
12 Replies

3. Shell Programming and Scripting

Selecting rows based on values in columns

Hi My pipe delimited .txt file contains rows with 10 columns. Can anyone advise how I output to file only those rows with the letters ‘ci' as the first 2 characters in the 3rd column ? Many thanks (4 Replies)
Discussion started by: malts18
4 Replies

4. Shell Programming and Scripting

Extract several columns with few rows

Hello, I want to extract several columns and rows from a huge tab delimited file for example: I want to print from from column 3 to 68 till row number 30. I have tried using cut command but it was extracting whole 3rd and 68th column. Please suggest a solution. Ryan (8 Replies)
Discussion started by: ryan9011
8 Replies

5. Shell Programming and Scripting

Sum of range of rows and columns in matrix

Hi all, I have a large matrix of 720 x 25. I want to get sum of range of rows and columns. Like, I need sum of all columns and row number 2 to 21, then leaving 22nd row, again sum of all columns and row number 23 to 42 again leaving 43rd row and then sum of 44th to 63. Means I want to add all... (4 Replies)
Discussion started by: CAch
4 Replies

6. Shell Programming and Scripting

Extracting rows and columns in a matrix based on condition

Hi I have a matrix with n rows and m columns like below example. i want to extract all the pairs with values <200. Input A B C D A 100 206 51 300 B 206 100 72 48 C 351 22 100 198 D 13 989 150 100 Output format A,A:200 A,C:51 B,B:100... (2 Replies)
Discussion started by: anurupa777
2 Replies

7. Shell Programming and Scripting

Compare 2 csv files by columns, then extract certain columns of matcing rows

Hi all, I'm pretty much a newbie to UNIX. I would appreciate any help with UNIX coding on comparing two large csv files (greater than 10 GB in size), and output a file with matching columns. I want to compare file1 and file2 by 'id' and 'chain' columns, then extract exact matching rows'... (5 Replies)
Discussion started by: bkane3
5 Replies

8. Shell Programming and Scripting

Extract rows with different values at 2 columns

Hallo, I would need to extract only rows which has different value in the second and third column. Thank you very much for any advices Input: A 0 0 B 0 1 C 1 1 D 1 3 Output B 0 1 D 1 3 (4 Replies)
Discussion started by: kamcamonty
4 Replies

9. UNIX for Beginners Questions & Answers

How to select rows that have opposite values (A vs B, or B vs A) on first two columns?

I have a dateset like this: Gly1 Gly2 2 1 0 Gly3 Gly4 3 4 5 Gly3 Gly5 1 3 2 Gly2 Gly1 3 6 2 Gly4 Gly3 2 2 1 Gly6 Gly4 4 2 1what I expected is: Gly1 Gly2 2 1 0 Gly2 Gly1 3 6 2 Gly3 Gly4 3 4 5 Gly4 Gly3 2 2 1 A vs B, or B vs A are the same... (7 Replies)
Discussion started by: nengcheng
7 Replies

10. Shell Programming and Scripting

Extract and exclude rows based on duplicate values

Hello I have a file like this: > cat examplefile ghi|NN603762|eee mno|NN607265|ttt pqr|NN613879|yyy stu|NN615002|uuu jkl|NN607265|rrr vwx|NN615002|iii yzA|NN618555|ooo def|NN190486|www BCD|NN628717|ppp abc|NN190486|qqq EFG|NN628717|aaa HIJ|NN628717|sss > I can sort the file by... (5 Replies)
Discussion started by: CHoggarth
5 Replies
MPSMatrixVectorMultiplication(3)			 MetalPerformanceShaders.framework			  MPSMatrixVectorMultiplication(3)

NAME
MPSMatrixVectorMultiplication SYNOPSIS
#import <MPSMatrixMultiplication.h> Inherits MPSMatrixBinaryKernel. Instance Methods (nonnull instancetype) - initWithDevice:transpose:rows:columns:alpha:beta: (nonnull instancetype) - initWithDevice:rows:columns: (nonnull instancetype) - initWithDevice: (void) - encodeToCommandBuffer:inputMatrix:inputVector:resultVector: Additional Inherited Members Detailed Description This depends on Metal.framework. A matrix-vector multiplication kernel. A MPSMatrixVectorMultiplication object computes: y = alpha * op(A) * x + beta * y A is a matrix represented by a MPSMatrix object. alpha and beta are scalar values (of the same data type as values of y) which are applied as shown above. A may have an optional transposition operation applied. A MPSMatrixVectorMultiplication object is initialized with the transpose operator to apply to A, sizes for the operation to perform, and the scalar values alpha and beta. Method Documentation - (void) encodeToCommandBuffer: (nonnull id< MTLCommandBuffer >) commandBuffer(MPSMatrix *__nonnull) inputMatrix(MPSVector *__nonnull) inputVector(MPSVector *__nonnull) resultVector Encode a MPSMatrixVectorMultiplication object to a command buffer. Parameters: commandBuffer A valid MTLCommandBuffer to receive the encoded kernel. inputMatrix A valid MPSMatrix object which specifies the input matrix A. inputVector A valid MPSVector object which specifies the input vector x. resultVector A valid MPSVector object which specifies the addend vector which will also be overwritten by the result. The left input matrix must be large enough to hold an array of size (rows x columns) elements beginning at primarySourceMatrixOrigin. The input vector must be large enough to hold an array of size (columns) elements beginning at secondarySourceMatrixOrigin.x secondarySourceMatrixOrigin.y and secondarySourceMatrixOrigin.z must be zero. The result vector must be large enough to hold an array of size (rows) elements beginning at resultMatrixOrigin.x. resultMatrixOrigin.y and resultMatrixOrigin.z must be zero. - (nonnull instancetype) initWithDevice: (nonnull id< MTLDevice >) device Use the above initialization method instead. Reimplemented from MPSKernel. - (nonnull instancetype) initWithDevice: (nonnull id< MTLDevice >) device(NSUInteger) rows(NSUInteger) columns Convenience initialization for a matrix-vector multiplication with no transposition, unit scaling of the product, and no accumulation of the result. The scaling factors alpha and beta are taken to be 1.0 and 0.0 respectively. Parameters: device The device on which the kernel will execute. rows The number of rows in the input matrix A, and the number of elements in the vector y. columns The number of columns in the input matrix A, and the number of elements in the input vector x. Returns: A valid MPSMatrixVectorMultiplication object or nil, if failure. - (nonnull instancetype) initWithDevice: (nonnull id< MTLDevice >) device(BOOL) transpose(NSUInteger) rows(NSUInteger) columns(double) alpha(double) beta Initialize an MPSMatrixVectorMultiplication object on a device for a given size and desired transpose and scale values. Parameters: device The device on which the kernel will execute. transpose A boolean value which indicates if the input matrix should be used in transposed form. if 'YES' then op(A) == A**T, otherwise op(A) == A. rows The number of rows in the input matrix op(A), and the number of elements in the vector y. columns The number of columns in the input matrix op(A), and the number of elements in the input vector x. alpha The scale factor to apply to the product. Specified in double precision. Will be converted to the appropriate precision in the implementation subject to rounding and/or clamping as necessary. beta The scale factor to apply to the initial values of y. Specified in double precision. Will be converted to the appropriate precision in the implementation subject to rounding and/or clamping as necessary. Returns: A valid MPSMatrixVectorMultiplication object or nil, if failure. Author Generated automatically by Doxygen for MetalPerformanceShaders.framework from the source code. Version MetalPerformanceShaders-100 Thu Feb 8 2018 MPSMatrixVectorMultiplication(3)
All times are GMT -4. The time now is 11:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy