awk loop for to filter lines by max value Post: 302918413

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers awk loop for to filter lines by max value Post 302918413 by lsantome on Tuesday 23rd of September 2014 07:10:43 AM

09-23-2014

Registered User

awk loop for to filter lines by max value

Hi all,

I'm struggling to filter my data frame. I need to print only those lines whose max value (the number of columns may vary) is above a cut-off value.

My data looks like this:

Code:

chr22    17565753    17565754    5    4    5    5    6    2    5    5    6    2
chr22    17565754    17565755    5    6    5    8    6    2    5    5    6    3
chr22    17565755    17565756    6    6    5    9    6    2    5    5    6    4
chr22    17565756    17565757    7    6    6    9    6    3    6    5    6    4
chr22    17565757    17565758    10    6    6    10    6    3    6    5    6    4
chr22    17565758    17565759    11    7    6    10    7    3    7    5    6    7
chr22    17565759    17565760    11    7    7    10    8    3    7    8    6    9
chr22    17565760    17565761    11    8    8    11    8    3    7    8    8    10
chr22    17565761    17565762    11    8    8    12    8    3    7    8    8    10
chr22    17565762    17565763    12    9    8    12    8    3    7    9    8    10
chr22    17565763    17565764    12    9    9    12    8    3    7    10    8    10
chr22    17565764    17565765    14    10    9    12    8    4    7    10    8    10
chr22    17565765    17565766    14    10    10    12    9    4    7    10    9    10
chr22    17565766    17565767    15    10    11    13    9    4    7    10    9    11
chr22    17565767    17565768    15    10    11    14    9    4    9    10    9    11
chr22    17565768    17565769    15    12    11    14    10    4    10    10    9    13

The fisrt three fields are constant (they show complementary information). The next fields (variable number of columns) contains the values to be used for filtering.
Let's say I want to print only those lines whose max value (starting from column 4 to ... $NF) is => 10.

In this case, the expected output should be:

Code:

chr22    17565757    17565758    10    6    6    10    6    3    6    5    6    4
chr22    17565758    17565759    11    7    6    10    7    3    7    5    6    7
chr22    17565759    17565760    11    7    7    10    8    3    7    8    6    9
chr22    17565760    17565761    11    8    8    11    8    3    7    8    8    10
chr22    17565761    17565762    11    8    8    12    8    3    7    8    8    10
chr22    17565762    17565763    12    9    8    12    8    3    7    9    8    10
chr22    17565763    17565764    12    9    9    12    8    3    7    10    8    10
chr22    17565764    17565765    14    10    9    12    8    4    7    10    8    10
chr22    17565765    17565766    14    10    10    12    9    4    7    10    9    10
chr22    17565766    17565767    15    10    11    13    9    4    7    10    9    11
chr22    17565767    17565768    15    10    11    14    9    4    9    10    9    11
chr22    17565768    17565769    15    12    11    14    10    4    10    10    9    13

I can get this result with a basic awk command like awk '($4 =>10) || ($5=>10)...' but I don't want to modify my script every time I run it on a different table format.
I'm not experienced in awk loops so I'd thank any suggestion.

Best,

Luis

Last edited by rbatte1; 09-23-2014 at 08:23 AM.. Reason: Add CODE & ICODE tags, emboldened commands and corrected spelling

lsantome

View Public Profile for lsantome

Find all posts by lsantome

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

loop in awk - column max for each column

Hello all, this should really be easy for you... I need AWK to print column maxima for each column of such input: Input: 1 2 3 1 2 1 1 3 2 1 1 2 Output should be: 2 2 3 3 This does the sum, but i need max instead: { for(i=1; i<=NF; i++) sum +=$i } END {for(i=1; i in sum;...

2. UNIX for Dummies Questions & Answers

loop? print max column in each line for 800 files and merge

Hello, I have 800 or so files with 3 columns each and >10000 lines each. For each file and each line I would like to print the maximum column number for each line. Then I would like to 'paste' each of these files together (column-wise) so that the file with expression in label '_1' is the...

3. Shell Programming and Scripting

AWK script - extracting min and max values from selected lines

Hi guys! I'm new to scripting and I need to write a script in awk. Here is example of file on which I'm working ATOM 4688 HG1 PRO A 322 18.080 59.680 137.020 1.00 0.00 ATOM 4689 HG2 PRO A 322 18.850 61.220 137.010 1.00 0.00 ATOM 4690 CD ...

4. Shell Programming and Scripting

awk to loop lines and print substring

The input data: mbeanName: com.sap.default:name=tc~bl~deploy_controller,j2eeType=SAP_J2EEServicePerNode,SAP_J2EEClusterNode=3620850,SAP_J2EECluster=XXX Get attribute Properties: {undepl_parallelism_strategy=normal, deployment_forbidden=off, locking_retries=50, suppress_ds_warnings=on,...

5. Homework & Coursework Questions

Min/Max/counter/while loop from file

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: The program is supposed to read in text from a given file ( different samples provided in the homework but not...

6. Shell Programming and Scripting

Filter (by max length) only lines not matching regex

I have a large file of many pairs of sequences and their headers, which always begin with '>' I'm looking for help on how to retain only sequences (and their headers) below a certain length. So if min length was 10, output would be I can filter by length, but I'm not sure how to exclude...

7. Shell Programming and Scripting

awk to filter multiple lines

Hi. I need to filter lines based upon matches in multiple tab-separated columns. For all matching occurrences in column 1, check the corresponding column 4. IF all column 4 entries are identical, discard all lines. If even one entry in column 4 is different, then keep all lines. How can I...

8. Shell Programming and Scripting

awk to filter out lines containing unique values in a specified column

Hi, I have multiple files that each contain four columns of strings: File1: Code: 123 abc gfh 273 456 ddff jfh 837 789 ghi u4u 395 File2: Code: 123 abc dd fu 456 def 457 nd 891 384 djh 783 I want to compare the strings in Column 1 of File 1 with each other file and Print in...

9. Shell Programming and Scripting

Filter Row Based On Max Column Value After Group BY

Hello Team, Need your expertise on following: Here is the set of data: C1|4|C1SP1|A1|C1BP1|T1 C1|4|C1SP2|A1|C1BP2|T2 C2|3|C2SP1|A2|C2BP1|T2 C3|3|C3SP1|A3|C3BP1|T2 C2|2|C2SP2|A2|C2BP2|T1 I need to filter above date base on following two steps: 1. Group them by column 1 and 4 2....

10. Shell Programming and Scripting

Run a shell script in a loop with max number of threads

hi guys. i have a question for you i have a one file and inside this file there are 1000 lines and each line is a linux command running this commands takes long time so i want to create one bash script and run this lines in a loop with max number of threads for example i want to run...