read line by line and calculate the co-presence of variables


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting read line by line and calculate the co-presence of variables
# 1  
Old 10-12-2012
Question read line by line and calculate the co-presence of variables

Hey guyz,

I have a table which shows the presence or absence of my variables (A,B,C,...) in my observations (1,2,3,...)

* A B C ...
1 1 0 1
2 1 1 0
3 1 0 0
...

I want to calculate the co-presence of my variables. to have a table shows the pairwise presence of the variables (have value 1). something like this:

* A B C ...
A 3 1 1
B 1 1 0
C 1 0 1
...

You see in this table there are 3 observations which variable A is present and for A&B there is one but for example for B&C there is no co-presence and so far.

I would be so grateful if you can help me with this.
# 2  
Old 10-12-2012
Well, the SQL flavored solution is to decompose the table into a simpler table, columns into rows, with a union in a derived table, for instance:
Code:
Var Obs
A 1
A 2
A 3
B 2
C 1

Now you can join that table to itself, where a.obs=b.obs and (to avoid self and duplicate corelations, if you care) a.var < b.var

In SQL, when someone puts the cells of like entities that should be many rows into columns, a rotated array, you need unions to view the few rows many columns as one column many rows. Conversely, if you need to turn cells from many rows into columns of one row, you need to use joins. It is a common RDBMS design error to put arrays horizontally in rows when they belong vertically in one column many rows.

Now if we start talking lines and columns, 'union' is concatenate (cat, >> or just multiple passes in a subshell), 'join' is, surprise, 'sort' and 'join'; 'where' is in the join or grep. Deciding 'A' < 'B' is more a perl chr()/ord() or C question, but you were accepting self and duplicate matches above.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Read line and save fields as variables

Hej guys, I am trying to read a csv file line by line, save it's fields as variables per line so I can use them as parameters and execute stuff. I am new to shell scripting and was just strictly following a tutorial. Somehow my version seems to ignore the loop. Any help? TY! :) #!/bin/bash... (12 Replies)
Discussion started by: Splinter479
12 Replies

2. Shell Programming and Scripting

How to read file line by line and compare subset of 1st line with 2nd?

Hi all, I have a log file say Test.log that gets updated continuously and it has data in pipe separated format. A sample log file would look like: <date1>|<data1>|<url1>|<result1> <date2>|<data2>|<url2>|<result2> <date3>|<data3>|<url3>|<result3> <date4>|<data4>|<url4>|<result4> What I... (3 Replies)
Discussion started by: pat_pramod
3 Replies

3. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

4. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

5. Shell Programming and Scripting

How to calculate mean in AWK? line by line several files, thousands of lines

I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this 050 0.023 0.504336 050 0.024 0.529521 050 0.025 0.538908 050 0.026 0.537035 I want to find the mean line by line of the third column from the files named like this: Stat-f-1.dat .... Stat-f-7.dat Stat-s-1.dat... (8 Replies)
Discussion started by: AriasFco
8 Replies

6. Shell Programming and Scripting

Read 1-line file and separate into multiple variables

I have one line files with 17 records separated by a semi-colon. I need to create a variable from each record, which I can do via a separate awk for each one, but I know there has to be a better way. Along with pulling out the variable, I need to convert some url coding like a + to a space, etc.... (4 Replies)
Discussion started by: numele
4 Replies

7. Shell Programming and Scripting

bash: read file line by line (lines have '\0') - not full line has read???

I am using the while-loop to read a file. The file has lines with null-terminated strings (words, actually.) What I have by that reading - just a first word up to '\0'! I need to have whole string up to 'new line' - (LF, 10#10, 16#A) What I am doing wrong? #make file 'grb' with... (6 Replies)
Discussion started by: alex_5161
6 Replies

8. Shell Programming and Scripting

Input 2 files, calculate diffs line by line

Hi I am new to Unix and need help with the following (to you all I'm sure) simple task. I am trying to output the differences between previous snaphots of various filesystem sizes to the present sizes. I have three files (e.g.) : file 1 file 2 file 3 10 100 /var... (4 Replies)
Discussion started by: bigbuk
4 Replies

9. Shell Programming and Scripting

Read variables from line to fixed length

I would like to make a script to read three variables (no fixed length or position) from a line and write them into a file, with fixed length and right-justified in each column. The fixed text (text1-text4) prior to the thee variables and the variables themselves are originally separated by spaces... (3 Replies)
Discussion started by: SharkM
3 Replies

10. Shell Programming and Scripting

How to read a line and put it into 3 variables

Hi All, I'll get a file whose 2nd line contains 3 fields: filename(variable length), file size char(10), and record count int(10). How do I cut it and put it into 3 variables? eg: abcd.csv01234567891111111111 now I want: $one = abcd.csv, $two = 0123456789, $three = 1111111111. I also... (8 Replies)
Discussion started by: Mandab
8 Replies
Login or Register to Ask a Question