UNIX for Dummies Questions & Answers

Hi,

I have a very wide dataset with pairs of columns starting from column 7 onwards (see example below).

Code:

0 123456 -1 0 0 -9 0 0 1 2 2 2 1 1 1 1 2 2...
0 123457 -1 0 0 -9 1 2 1 1 2 2 0 0 0 0 2 2...
0 123458 -1 0 0 -9 0 0 1 2 2 2 1 1 2 2 1 2...
0 123459 -1 0 0 -9 1 2 0 0 2 2 1 1 1 2 1 1...

For each pair of columns 7&8, 9&10, 11&12... I would like to produce an indicator of whether there is more than 1 possibility for each pair in the column. A pair 0 0 represents a missing value and should be ignored (ie: not counted as its own class).

So for the first pair of columns the possibilities are 0 0 and 1 2. However because 0 0 represents a missing value, the only remaining combination is 1 2. Therefore there is only 1 possibility for the pair of columns.

For the second pair of columns the possibilities are 0 0, 1 1 and 1 2. Therefore there are 2 possibilities for the pair of columns.

The output for the above example would then be: 0 1 0 0 1 1 where the zero means there is only 1 possibility for the pair and a 1 means there is more than one possibility for the pair in the column.

I hope that makes sense - it is a bit difficult to explain.

I would then also like to apply the filtering 0 1 0 0 1 1 to a second file which looks as follows (there is one row for every pair of columns in the first file):

Code:

 
22 gen1 12345
22 gen2 34678
22 gen3 47659
22 gen4 57647
22 gen5 68543
22 gen6 75624
.
.
.

The final output would then be a file with rows from column 2 of this file where the corresponding column of the filter is a 1.
eg: gen2
gen5
gen6
.
.

Any help would be appreciated. I have a code to do this in R and one in J but they are a bit too resource expensive and I thought that something using awk might work a lot better.

Many thanks in advance,
Kathryn

Moderator's Comments:

Video tutorial on how to use code tags in The UNIX and Linux Forums.

See if this will be slower or faster than what you already have:

Code:

#!/usr/bin/ksh
rm -f File*
typeset -i mPair
cut -d' ' -f7- b | while read mLine; do
  mPair=0
  echo ${mLine} | xargs -n2 | while read m1 m2; do
    mPair=${mPair}+1
    mFName='File'${mPair}
    if [[ "${m1}" != "0" && "${m2}" != "0" ]]; then
      echo ${m1}${m2} >> ${mFName}
    fi
  done
done
mFTemp='Temp_File'
for mFName in File*; do
  sort -o ${mFName} -u ${mFName}
  mCnt=$(wc -l < ${mFName})
  if [[ "${mCnt}" = "0" || "${mCnt}" = "1" ]]; then
    echo '0' >> ${mFTemp}
  else
    echo '1' >> ${mFTemp}
  fi
done
paste -d' ' ${mFTemp} The_Search_File | while read m1 m2 m3 m4; do
  if [[ "${m1}" = "1" ]]; then
    echo ${m3}
  fi
done
rm -f File*
rm -f ${mFTemp}

Thanks very much for that. I've since found out that the problem is slightly different from what I thought so I'm going to re-post. I will try your code out though as it looks like it would be helpful. Thanks!