Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

awk loop for to filter lines by max value

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers awk loop for to filter lines by max value
# 1  
Old 09-23-2014
awk loop for to filter lines by max value
Hi all,

I'm struggling to filter my data frame. I need to print only those lines whose max value (the number of columns may vary) is above a cut-off value.

My data looks like this:

Code:
chr22    17565753    17565754    5    4    5    5    6    2    5    5    6    2
chr22    17565754    17565755    5    6    5    8    6    2    5    5    6    3
chr22    17565755    17565756    6    6    5    9    6    2    5    5    6    4
chr22    17565756    17565757    7    6    6    9    6    3    6    5    6    4
chr22    17565757    17565758    10    6    6    10    6    3    6    5    6    4
chr22    17565758    17565759    11    7    6    10    7    3    7    5    6    7
chr22    17565759    17565760    11    7    7    10    8    3    7    8    6    9
chr22    17565760    17565761    11    8    8    11    8    3    7    8    8    10
chr22    17565761    17565762    11    8    8    12    8    3    7    8    8    10
chr22    17565762    17565763    12    9    8    12    8    3    7    9    8    10
chr22    17565763    17565764    12    9    9    12    8    3    7    10    8    10
chr22    17565764    17565765    14    10    9    12    8    4    7    10    8    10
chr22    17565765    17565766    14    10    10    12    9    4    7    10    9    10
chr22    17565766    17565767    15    10    11    13    9    4    7    10    9    11
chr22    17565767    17565768    15    10    11    14    9    4    9    10    9    11
chr22    17565768    17565769    15    12    11    14    10    4    10    10    9    13

The fisrt three fields are constant (they show complementary information). The next fields (variable number of columns) contains the values to be used for filtering.
Let's say I want to print only those lines whose max value (starting from column 4 to ... $NF) is => 10.

In this case, the expected output should be:
Code:
chr22    17565757    17565758    10    6    6    10    6    3    6    5    6    4
chr22    17565758    17565759    11    7    6    10    7    3    7    5    6    7
chr22    17565759    17565760    11    7    7    10    8    3    7    8    6    9
chr22    17565760    17565761    11    8    8    11    8    3    7    8    8    10
chr22    17565761    17565762    11    8    8    12    8    3    7    8    8    10
chr22    17565762    17565763    12    9    8    12    8    3    7    9    8    10
chr22    17565763    17565764    12    9    9    12    8    3    7    10    8    10
chr22    17565764    17565765    14    10    9    12    8    4    7    10    8    10
chr22    17565765    17565766    14    10    10    12    9    4    7    10    9    10
chr22    17565766    17565767    15    10    11    13    9    4    7    10    9    11
chr22    17565767    17565768    15    10    11    14    9    4    9    10    9    11
chr22    17565768    17565769    15    12    11    14    10    4    10    10    9    13

I can get this result with a basic awk command like awk '($4 =>10) || ($5=>10)...' but I don't want to modify my script every time I run it on a different table format.
I'm not experienced in awk loops so I'd thank any suggestion.

Best,

Luis
Last edited by rbatte1; 09-23-2014 at 08:23 AM.. Reason: Add CODE & ICODE tags, emboldened commands and corrected spelling
# 2  
Old 09-23-2014
There is no => operator in awk. I assume you meant >=. Try:
Code:
awk '{for(i = 4; i <= NF; i++) if($i >= 10) {print; next}}' file

# 3  
Old 09-23-2014
Thanks!
Quote:
Originally Posted by Don Cragun
There is no => operator in awk. I assume you meant >=. Try:
Code:
awk '{for(i = 4; i <= NF; i++) if($i >= 10) {print; next}}' file

Wow! It looks easy but I could expend the rest of my life on doing it.

Thank you so much Don Cragun!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Run a shell script in a loop with max number of threads

hi guys. i have a question for you i have a one file and inside this file there are 1000 lines and each line is a linux command running this commands takes long time so i want to create one bash script and run this lines in a loop with max number of threads for example i want to run... (2 Replies)
Discussion started by: avtaritet
2 Replies

2. Shell Programming and Scripting

Filter Row Based On Max Column Value After Group BY

Hello Team, Need your expertise on following: Here is the set of data: C1|4|C1SP1|A1|C1BP1|T1 C1|4|C1SP2|A1|C1BP2|T2 C2|3|C2SP1|A2|C2BP1|T2 C3|3|C3SP1|A3|C3BP1|T2 C2|2|C2SP2|A2|C2BP2|T1 I need to filter above date base on following two steps: 1. Group them by column 1 and 4 2.... (12 Replies)
Discussion started by: angshuman
12 Replies

3. Shell Programming and Scripting

awk to filter out lines containing unique values in a specified column

Hi, I have multiple files that each contain four columns of strings: File1: Code: 123 abc gfh 273 456 ddff jfh 837 789 ghi u4u 395 File2: Code: 123 abc dd fu 456 def 457 nd 891 384 djh 783 I want to compare the strings in Column 1 of File 1 with each other file and Print in... (3 Replies)
Discussion started by: owwow14
3 Replies

4. Shell Programming and Scripting

awk to filter multiple lines

Hi. I need to filter lines based upon matches in multiple tab-separated columns. For all matching occurrences in column 1, check the corresponding column 4. IF all column 4 entries are identical, discard all lines. If even one entry in column 4 is different, then keep all lines. How can I... (5 Replies)
Discussion started by: owwow14
5 Replies

5. Shell Programming and Scripting

Filter (by max length) only lines not matching regex

I have a large file of many pairs of sequences and their headers, which always begin with '>' I'm looking for help on how to retain only sequences (and their headers) below a certain length. So if min length was 10, output would be I can filter by length, but I'm not sure how to exclude... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

6. Homework & Coursework Questions

Min/Max/counter/while loop from file

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: The program is supposed to read in text from a given file ( different samples provided in the homework but not... (1 Reply)
Discussion started by: c++newb
1 Replies

7. Shell Programming and Scripting

awk to loop lines and print substring

The input data: mbeanName: com.sap.default:name=tc~bl~deploy_controller,j2eeType=SAP_J2EEServicePerNode,SAP_J2EEClusterNode=3620850,SAP_J2EECluster=XXX Get attribute Properties: {undepl_parallelism_strategy=normal, deployment_forbidden=off, locking_retries=50, suppress_ds_warnings=on,... (5 Replies)
Discussion started by: ux4me
5 Replies

8. Shell Programming and Scripting

AWK script - extracting min and max values from selected lines

Hi guys! I'm new to scripting and I need to write a script in awk. Here is example of file on which I'm working ATOM 4688 HG1 PRO A 322 18.080 59.680 137.020 1.00 0.00 ATOM 4689 HG2 PRO A 322 18.850 61.220 137.010 1.00 0.00 ATOM 4690 CD ... (18 Replies)
Discussion started by: grincz
18 Replies

9. UNIX for Dummies Questions & Answers

loop? print max column in each line for 800 files and merge

Hello, I have 800 or so files with 3 columns each and >10000 lines each. For each file and each line I would like to print the maximum column number for each line. Then I would like to 'paste' each of these files together (column-wise) so that the file with expression in label '_1' is the... (6 Replies)
Discussion started by: peanuts48
6 Replies

10. Shell Programming and Scripting

loop in awk - column max for each column

Hello all, this should really be easy for you... I need AWK to print column maxima for each column of such input: Input: 1 2 3 1 2 1 1 3 2 1 1 2 Output should be: 2 2 3 3 This does the sum, but i need max instead: { for(i=1; i<=NF; i++) sum +=$i } END {for(i=1; i in sum;... (3 Replies)
Discussion started by: irrevocabile
3 Replies
Login or Register to Ask a Question