Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Filter table of different length
Top Forums UNIX for Advanced & Expert Users Filter table of different length Post 303016402 by GDC on Thursday 26th of April 2018 06:26:28 AM
Old 04-26-2018
Filter table of different length
Dear Forum,

I have to filter (e.g. PF=0.8) a text file according to some measured and recorded values for different fields (sensor). The files can be large and the recorded data points (ID) could differ in length. I have worked out a solution but it is very messy and not flexible. Does anybody have a better, cleaner and more robust idea to solve this problem?

Input File:

Code:
ID_AB1\tsensor01(1.0),sensor02(0.6),sensor03(0.5),sensor04(0.45)
ID_AB2\tsensor01(1.0),sensor02(0.95),sensor03(0.90),sensor04(0.80)
ID_AC3\tsensor01(1.0),sensor02(1.0),sensor03(1.0),sensor04(1.0)
ID_AD1\tsensor01(0.9),sensor02(0.6)
ID_BA2

Output File needed:

Code:
ID_AB1;sensor01;low;low;low
ID_AB2;sensor01;sensor02;sensor03;low
ID_AC3;sensor01;sensor02;sensor03;sensor04
ID_AD1;sensor01;low;na;na
ID_BA2;na;na;na;na

I used to following commands but it does not works for records of shorter length.

Code:
sed 's/(/,/g' out.singnal |\
 sed 's/)//g' |\
 awk -F"\t|," -v PF=0.8 '{
   printf "%s;", $1
  if($3>=PF && $5>=PF && $7>=PF && $9>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,$8
  else if($3>=PF && $5>=PF && $7>=PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"low"
  else if($3>=PF && $5>=PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","low"
  else if($3>=PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","low"
  else if($3<PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", "low","low","low","low"
 }'

I add a "length control" to it but to cover all cases it would be messy Smilie.

Code:
sed 's/(/,/g' out.singnal |\
 sed 's/)//g' |\
 awk -F"\t|," -v PF=0.8 '{
  printf "%s;", $1
  if(NF==9 && $3>=PF && $5>=PF && $7>=PF && $9>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,$8
  else if(NF==9 && $3>=PF && $5>=PF && $7>=PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"low"
  else if(NF==9 && $3>=PF && $5>=PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","low"
  else if(NF==9 && $3>=PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","low"
  else if(NF==9 && $3<PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", "low","low","low","low"
  else if(NF==7 && $3>=PF && $5>=PF && $7>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"na"
  else if(NF==7 && $3>=PF && $5>=PF && $7<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","na"
  else if(NF==7 && $3>=PF && $5<PF && $7<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","na"
  else if(NF==7 && $3<PF && $5<PF && $7<PF)
          printf " %s; %s; %s; %s\n","low","low","low","na"
 }'

 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

creating a fixed length output from a variable length input

Is there a command that sets a variable length? I have a input of a variable length field but my output for that field needs to be set to 32 char. Is there such a command? I am on a sun box running ksh Thanks (2 Replies)
Discussion started by: r1500
2 Replies

2. UNIX for Dummies Questions & Answers

Sed working on lines of small length and not large length

Hi , I have a peculiar case, where my sed command is working on a file which contains lines of small length. sed "s/XYZ:1/XYZ:3/g" abc.txt > xyz.txt when abc.txt contains lines of small length(currently around 80 chars) , this sed command is working fine. when abc.txt contains lines of... (3 Replies)
Discussion started by: thanuman
3 Replies

3. UNIX for Dummies Questions & Answers

What the command to find out the record length of a fixed length file?

I want to find out the record length of a fixed length file? I forgot the command. Any body know? (9 Replies)
Discussion started by: tranq01
9 Replies

4. Shell Programming and Scripting

How to filter a table by two columns

Dear Forum, I would like to know how could I found every result from one column at a table based at two table. Exemplo: Table: Red 4 Red 5 Red 10 Black 33 Black 44 Black 5 Green 2 Green 55 Green 78 I would like to have every color with the lower result... (12 Replies)
Discussion started by: lColli
12 Replies

5. Shell Programming and Scripting

Flat file-make field length equal to header length

Hello Everyone, I am stuck with one issue while working on abstract flat file which i have to use as input and load data to table. Input Data- ------ ------------------------ ---- ----------------- WFI001 Xxxxxx Control Work Item A Number of Records ------ ------------------------... (5 Replies)
Discussion started by: sonali.s.more
5 Replies

6. Shell Programming and Scripting

Filter rows from table

Hi , I need to filter input file according to following All rows with the following conditions should be removed 1) If in a row, the number of 'N's starting col 2 exceeds 2 (3 or more) OR 2) If a row is duplicated with the same value, starting col 2, A value 'N' is considered missing... (1 Reply)
Discussion started by: newbie83
1 Replies

7. Shell Programming and Scripting

Filter (by max length) only lines not matching regex

I have a large file of many pairs of sequences and their headers, which always begin with '>' I'm looking for help on how to retain only sequences (and their headers) below a certain length. So if min length was 10, output would be I can filter by length, but I'm not sure how to exclude... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

8. Shell Programming and Scripting

Filter file by length, looking only at lines that don't begin with ">"

I have a file that stores data in pairs of lines, following this format: line 1: header (preceded by ">") line 2: sequence Example.txt: >seq1 name GATTGATGTTTGAGTTTTGGTTTTT >seq2 name TTTTCTTC I want to filter out the sequences and corresponding headers for all sequences that are less... (2 Replies)
Discussion started by: pathunkathunk
2 Replies

9. Shell Programming and Scripting

Filter all the lines with minimum specified length of words of a text file

Hi Can someone tell me which script will work best (in terms of speed and simplicity to write and run) for a large text file to filter all the lines with a minimum specified length of words ? A sample script with be definitely of great help !!! Thanks in advance. :) (4 Replies)
Discussion started by: my_Perl
4 Replies
All times are GMT -4. The time now is 09:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy