Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Matrix to 3 col sorted
Top Forums Shell Programming and Scripting Matrix to 3 col sorted Post 303003840 by senhia83 on Thursday 21st of September 2017 11:32:58 AM
Old 09-21-2017
Matrix to 3 col sorted
Hello experts, I have matrices sorted by position, there are 400k rows, 3000 columns.

Code:
ID      CHR     POS     M1      M2      M3      M4      M5
ID1     1       1       4.6     2.6     2.1     3.5     4.2
ID2     1       100     3.6     2.9     3.2     2.6     2.5
ID3     1       1000    4.1     2.9     2.4     2.8     2.5
ID4     1       2000    4.2     2.4     2.8     3.7     2.8
ID5     1       3000    4.6     2.9     3.7     3.5     2.7
ID6     1       4000    5.2     3       3.2     4.2     3.7
ID7     1       5000    5.8     2.2     2.8     3.1     2.7

I`m looking for a sorted long-format output by position, grouped by markers (Ms)

Code:
M1 ID1__1__1 4.6
M1 ID2__1__100 3.6
M1 ID3__1__1000 4.1
M1 ID4__1__2000 4.2
M1 ID5__1__3000 4.6
M1 ID6__1__4000 5.2
M1 ID7__1__5000 5.8
M2 ID1__1__1 2.6
M2 ID2__1__100 2.9
M2 ID3__1__1000 2.9
M2 ID4__1__2000 2.4
M2 ID5__1__3000 2.9
M2 ID6__1__4000 3
M2 ID7__1__5000 2.2
M3 ID1__1__1 2.1
M3 ID2__1__100 3.2
M3 ID3__1__1000 2.4
M3 ID4__1__2000 2.8
M3 ID5__1__3000 3.7
M3 ID6__1__4000 3.2
M3 ID7__1__5000 2.8
M4 ID1__1__1 3.5
M4 ID2__1__100 2.6
M4 ID3__1__1000 2.8
M4 ID4__1__2000 3.7
M4 ID5__1__3000 3.5
M4 ID6__1__4000 4.2
M4 ID7__1__5000 3.1

I have two working methods

1. This step requires a final sorting which for 400000x3000 datapoints will be very row or run out of memory.


Code:
awk -F"\t" 'NR==1{split($0,lines);}NR>1{for(i=4;i<length(lines);i++) print lines[i],$1"__"$2"__"$3,$i}' tmp | sort -k1,1


2. This step is very very slow as it requires to take the entire matrix into memory before making splits.

Code:
awk -F"\t" '{a[NR]=$0}END{split(a[1],lines); for(i=4;i<=length(lines);i++) { for(j=2;j<=length(a);j++) { split(a[j],haps); print lines[i],haps[1]"__"haps[2]"__"haps[3],haps[i]}}}'  tmp


Is there a more efficient way to get the output I need? Thanks in advance.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Help On col command

Hello Can Any1 tell me the difference between the col command and the col command with the -f option. I tried running both of them but i can't see any difference. Please guide me. (1 Reply)
Discussion started by: rahulrathod
1 Replies

2. Ubuntu

Match col 1 of File 1 with col 1 File 2 and create a 3rd file

Hello, I have a 1.6 GB file that I would like to modify by matching some ids in col1 with the ids in col 1 of file2.txt and save the results into a 3rd file. For example: File 1 has 1411 rows, I ignore how many columns it has (thousands) File 2 has 311 rows, 1 column Would like to... (7 Replies)
Discussion started by: sogi
7 Replies

3. Shell Programming and Scripting

diagonal matrix to square matrix

Hello, all! I am struggling with a short script to read a diagonal matrix for later retrieval. 1.000 0.234 0.435 0.123 0.012 0.102 0.325 0.412 0.087 0.098 1.000 0.111 0.412 0.115 0.058 0.091 0.190 0.045 0.058 1.000 0.205 0.542 0.335 0.054 0.117 0.203 0.125 1.000 0.587 0.159 0.357... (11 Replies)
Discussion started by: yifangt
11 Replies

4. Shell Programming and Scripting

i can't cut the third col

SW_dist_intr false Enable SW distribution of interrupts True autorestart true Automatically REBOOT OS after a crash True boottype disk N/A False capacity_inc 1.00 ... (7 Replies)
Discussion started by: maxim42
7 Replies

5. Shell Programming and Scripting

how to add new col in a file

Hi, Experts, I have a requirement as following: my source file: a a a b b c c c c I need add one more colume as following: 1 a 2 a 3 a 1 b 2 b 1 c 2 c (4 Replies)
Discussion started by: ken002
4 Replies

6. Ubuntu

How to convert full data matrix to linearised left data matrix?

Hi all, Is there a way to convert full data matrix to linearised left data matrix? e.g full data matrix Bh1 Bh2 Bh3 Bh4 Bh5 Bh6 Bh7 Bh1 0 0.241058 0.236129 0.244397 0.237479 0.240767 0.245245 Bh2 0.241058 0 0.240594 0.241931 0.241975 ... (8 Replies)
Discussion started by: evoll
8 Replies

7. UNIX for Advanced & Expert Users

Print line based on highest value of col (B) and repetion of values in col (A)

Hello everyone, I am writing a script to process data from the ATP world tour. I have a file which contains: t=540 y=2011 r=1 p=N409 t=540 y=2011 r=2 p=N409 t=540 y=2011 r=3 p=N409 t=540 y=2011 r=4 p=N409 t=520 y=2011 r=1 p=N409 t=520 y=2011 r=2 p=N409 t=520 y=2011 r=3 p=N409 The... (4 Replies)
Discussion started by: imahmoud
4 Replies

8. Shell Programming and Scripting

awk? adjacency matrix to adjacency list / correlation matrix to list

Hi everyone I am very new at awk but think that that might be the best strategy for this. I have a matrix very similar to a correlation matrix and in practical terms I need to convert it into a list containing the values from the matrix (one value per line) with the first field of the line (row... (5 Replies)
Discussion started by: stonemonkey
5 Replies

9. Shell Programming and Scripting

Printing from col x to end of line, except last col

Hello, I have some tab delimited data and I need to move the last col. I could hard code it, awk '{ print $1,$NF,$2,$3,$4,etc }' infile > outfile but it would be nice to know the syntax to print a range cols. I know in cut you can do, cut -f 1,4-8,11- to print fields 1,... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

10. Shell Programming and Scripting

Modifying col values based on another col

Hi, Please help with this. I have several excel files (with and .xlsx format) with 10-15 columns each. They all have the same type of data but the columns are not ordered in the same way. Here is a 3 column example. What I want to do add the alphabet from column 2 to column 3, provided... (9 Replies)
Discussion started by: newbie83
9 Replies

LEARN ABOUT LINUX

irsend

IRSEND(1)							   User Commands							 IRSEND(1)

NAME

       irsend - basic LIRC program to send infra-red commands

SYNOPSIS

       irsend [options] DIRECTIVE REMOTE CODE [CODE...]

DESCRIPTION

       Asks the lircd daemon to send one or more CIR (Consumer Infra-Red) commands. This is intended for remote control of electronic devices such
       as TV boxes, HiFi sets, etc.

       DIRECTIVE can be:
	  SEND_ONCE	    - send CODE [CODE ...] once
	  SEND_START	    - start repeating CODE
	  SEND_STOP	    - stop repeating CODE
	  LIST		    - list configured remote items
	  SET_TRANSMITTERS  - set transmitters NUM [NUM ...]
	  SIMULATE	    - simulate IR event

       REMOTE is the name of a remote, as described in the lircd configuration file.

       CODE is the name of a remote control key of REMOTE, as it appears in the lircd configuration file.

       NUM is the transmitter number of the hardware device.

       For the LIST DIRECTIVE, REMOTE and/or CODE can be empty:

	  LIST	 ""    ""   - list all configured remote names
	  LIST REMOTE  ""   - list all codes of REMOTE
	  LIST REMOTE CODE  - list only CODE of REMOTE

       The SIMULATE command only works if it has been explicitly enabled in lircd.

       -h --help
	      display usage summary

       -v --version
	      display version

       -d --device
	      use given lircd socket [/var/run/lirc/lircd]

       -a --address=host[:port]
	      connect to lircd at this address

       -# --count=n
	      send command n times

EXAMPLES

	  irsend LIST DenonTuner ""
	  irsend SEND_ONCE  DenonTuner PROG-SCAN
	  irsend SEND_ONCE  OnkyoAmpli VOL-UP VOL-UP VOL-UP VOL-UP
	  irsend SEND_START OnkyoAmpli VOL-DOWN ; sleep 3
	  irsend SEND_STOP  OnkyoAmpli VOL-DOWN
	  irsend SET_TRANSMITTERS 1
	  irsend SET_TRANSMITTERS 1 3 4
	  irsend SIMULATE "0000000000000476 00 OK TECHNISAT_ST3004S"

FILES

       /etc/lirc/lircd.conf
	      Default lircd configuration file.  It should contain all the remotes, their infra-red codes and the corresponding timing	and  wave-
	      form details.

DIAGNOSTICS

       If lircd is not running (or /var/run/lirc/lircd lacks write permissions) irsend aborts with the following diagnostics:
       "irsend: could not connect to socket"
       "irsend: Connection refused" (or "Permission denied").

SEE ALSO

       The documentation for lirc is maintained as html pages. They are located under html/ in the documentation directory.

       lircd(8), mode2(1), smode2(1), xmode2(1), irrecord(1), irw(1), http://www.lirc.org.

irsend 0.9.0-pre1						   October 2010 							 IRSEND(1)
All times are GMT -4. The time now is 01:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy