Filter table of different length


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Filter table of different length
# 1  
Old 04-26-2018
Filter table of different length

Dear Forum,

I have to filter (e.g. PF=0.8) a text file according to some measured and recorded values for different fields (sensor). The files can be large and the recorded data points (ID) could differ in length. I have worked out a solution but it is very messy and not flexible. Does anybody have a better, cleaner and more robust idea to solve this problem?

Input File:

Code:
ID_AB1\tsensor01(1.0),sensor02(0.6),sensor03(0.5),sensor04(0.45)
ID_AB2\tsensor01(1.0),sensor02(0.95),sensor03(0.90),sensor04(0.80)
ID_AC3\tsensor01(1.0),sensor02(1.0),sensor03(1.0),sensor04(1.0)
ID_AD1\tsensor01(0.9),sensor02(0.6)
ID_BA2

Output File needed:

Code:
ID_AB1;sensor01;low;low;low
ID_AB2;sensor01;sensor02;sensor03;low
ID_AC3;sensor01;sensor02;sensor03;sensor04
ID_AD1;sensor01;low;na;na
ID_BA2;na;na;na;na

I used to following commands but it does not works for records of shorter length.

Code:
sed 's/(/,/g' out.singnal |\
 sed 's/)//g' |\
 awk -F"\t|," -v PF=0.8 '{
   printf "%s;", $1
  if($3>=PF && $5>=PF && $7>=PF && $9>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,$8
  else if($3>=PF && $5>=PF && $7>=PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"low"
  else if($3>=PF && $5>=PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","low"
  else if($3>=PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","low"
  else if($3<PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", "low","low","low","low"
 }'

I add a "length control" to it but to cover all cases it would be messy Smilie.

Code:
sed 's/(/,/g' out.singnal |\
 sed 's/)//g' |\
 awk -F"\t|," -v PF=0.8 '{
  printf "%s;", $1
  if(NF==9 && $3>=PF && $5>=PF && $7>=PF && $9>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,$8
  else if(NF==9 && $3>=PF && $5>=PF && $7>=PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"low"
  else if(NF==9 && $3>=PF && $5>=PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","low"
  else if(NF==9 && $3>=PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","low"
  else if(NF==9 && $3<PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", "low","low","low","low"
  else if(NF==7 && $3>=PF && $5>=PF && $7>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"na"
  else if(NF==7 && $3>=PF && $5>=PF && $7<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","na"
  else if(NF==7 && $3>=PF && $5<PF && $7<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","na"
  else if(NF==7 && $3<PF && $5<PF && $7<PF)
          printf " %s; %s; %s; %s\n","low","low","low","na"
 }'

# 2  
Old 04-26-2018
What output do you want?
# 3  
Old 04-27-2018
Hi Corona688,

the output should be a text file

Code:
ID_AB1;sensor01;low;low;low
ID_AB2;sensor01;sensor02;sensor03;low
ID_AC3;sensor01;sensor02;sensor03;sensor04
ID_AD1;sensor01;low;na;na
ID_BA2;na;na;na;na

Thank for considering my problem
# 4  
Old 04-27-2018
Code:
awk '
{lines[NR]=$0; if (NF > columns) columns=NF}
END {
   for (j=1; j<=NR; j++) {
      split(lines[j], sensors, FS);
      $0=sensors[1];
      for (i=2; i<=columns; i++) {
         split(sensors[i], fields, "[()]");
         $0=$0 OFS ((fields[2] <= PF) ? (length(sensors[i]) ? "low" : "na") : fields[1]);
      }
      print $0;
   }
}
' FS="[,\t]" OFS=";" PF=0.80 input_file


Last edited by rdrtx1; 04-27-2018 at 03:35 PM..
These 2 Users Gave Thanks to rdrtx1 For This Post:
# 5  
Old 04-30-2018
Dear rdrtx1,

thank you for your help.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filter all the lines with minimum specified length of words of a text file

Hi Can someone tell me which script will work best (in terms of speed and simplicity to write and run) for a large text file to filter all the lines with a minimum specified length of words ? A sample script with be definitely of great help !!! Thanks in advance. :) (4 Replies)
Discussion started by: my_Perl
4 Replies

2. Shell Programming and Scripting

Filter file by length, looking only at lines that don't begin with ">"

I have a file that stores data in pairs of lines, following this format: line 1: header (preceded by ">") line 2: sequence Example.txt: >seq1 name GATTGATGTTTGAGTTTTGGTTTTT >seq2 name TTTTCTTC I want to filter out the sequences and corresponding headers for all sequences that are less... (2 Replies)
Discussion started by: pathunkathunk
2 Replies

3. Shell Programming and Scripting

Filter (by max length) only lines not matching regex

I have a large file of many pairs of sequences and their headers, which always begin with '>' I'm looking for help on how to retain only sequences (and their headers) below a certain length. So if min length was 10, output would be I can filter by length, but I'm not sure how to exclude... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

4. Shell Programming and Scripting

Filter rows from table

Hi , I need to filter input file according to following All rows with the following conditions should be removed 1) If in a row, the number of 'N's starting col 2 exceeds 2 (3 or more) OR 2) If a row is duplicated with the same value, starting col 2, A value 'N' is considered missing... (1 Reply)
Discussion started by: newbie83
1 Replies

5. Shell Programming and Scripting

Flat file-make field length equal to header length

Hello Everyone, I am stuck with one issue while working on abstract flat file which i have to use as input and load data to table. Input Data- ------ ------------------------ ---- ----------------- WFI001 Xxxxxx Control Work Item A Number of Records ------ ------------------------... (5 Replies)
Discussion started by: sonali.s.more
5 Replies

6. Shell Programming and Scripting

How to filter a table by two columns

Dear Forum, I would like to know how could I found every result from one column at a table based at two table. Exemplo: Table: Red 4 Red 5 Red 10 Black 33 Black 44 Black 5 Green 2 Green 55 Green 78 I would like to have every color with the lower result... (12 Replies)
Discussion started by: lColli
12 Replies

7. UNIX for Dummies Questions & Answers

What the command to find out the record length of a fixed length file?

I want to find out the record length of a fixed length file? I forgot the command. Any body know? (9 Replies)
Discussion started by: tranq01
9 Replies

8. UNIX for Dummies Questions & Answers

Sed working on lines of small length and not large length

Hi , I have a peculiar case, where my sed command is working on a file which contains lines of small length. sed "s/XYZ:1/XYZ:3/g" abc.txt > xyz.txt when abc.txt contains lines of small length(currently around 80 chars) , this sed command is working fine. when abc.txt contains lines of... (3 Replies)
Discussion started by: thanuman
3 Replies

9. Shell Programming and Scripting

creating a fixed length output from a variable length input

Is there a command that sets a variable length? I have a input of a variable length field but my output for that field needs to be set to 32 char. Is there such a command? I am on a sun box running ksh Thanks (2 Replies)
Discussion started by: r1500
2 Replies
Login or Register to Ask a Question