Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Cat Values from Several files if it meets criteria for column values

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Cat Values from Several files if it meets criteria for column values
# 1  
Old 08-31-2011
Cat Values from Several files if it meets criteria for column values
I have results from some statistical analyses.
The format of the results are as given below:
  1. I want to select lines that have a p-value (last column) less than 0.05 from all the results files (*.results) and cat to a new results file.
  2. It would be very nice if a new column is added that tells me the name of the original results file for each line added.
awk preferred, but can also do python.

Code:
e.g: Stat1.result
Bin ID ZIP P
3 Individual-1813375 11972 1.99E-05
3 Individual-4681817 58156 0.0001712
2 Individual-13020362 23877 0.0006332
1 Individual-17226184 20192 0.003108
4 Individual-17037125 84105 0.008756
3 Individual-4680035 15428 0.01189
4 Individual-759458 46333 0.0283
1 Individual-2762682 29182 0.03233
4 Individual-11099561 23056 0.03826
3 Individual-7551560 13650 0.0576
2 Individual-1036543 65098 0.0579
2 Individual-235385 13339 0.05882
2 Individual-1430261 12487 0.08075
3 Individual-4677602 71462 0.09687
1 Individual-9398631 11902 0.1085
1 Individual-3767635 16008 0.1127
3 Individual-11733459 64659 0.1555
2 Individual-1867856 32616 0.1628
1 Individual-11581364 16013 0.1708

Result File should be like this:
Code:
File Bin ID ZIP P
Stat1.result 3 Individual-1813375 11972 1.99E-05
Stat1.result 3 Individual-4681817 58156 0.0001712
Stat1.result 2 Individual-13020362 23877 0.0006332
Stat1.result 1 Individual-17226184 20192 0.003108
Stat1.result 4 Individual-17037125 84105 0.008756
Stat1.result 3 Individual-4680035 15428 0.01189
Stat1.result 4 Individual-759458 46333 0.0283
Stat1.result 1 Individual-2762682 29182 0.03233
Stat2.result 4 Individual-2985340 29244 0.016565
Stat3.result 3 Individual-10177001 19173 0.0466
Stat6.result 2 Individual-1036543 65098 0.0479

Thank you for your help.
Last edited by genehunter; 08-31-2011 at 09:00 PM.. Reason: P-value is last column
# 2  
Old 08-31-2011
Try this:

Code:
awk ' /ZIP/ {next;} $NF < .05 { printf( "%s %s\n", $0, FILENAME ); }' *.results >new.file

Adds the filename as the right most column to each output line. Does discards the header lines.
Last edited by agama; 08-31-2011 at 08:36 PM.. Reason: oops -- added redirection of output
This User Gave Thanks to agama For This Post:
genehunter
# 3  
Old 09-01-2011
Worked exactly as described.
A big thank you and repped.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Only print specific xml values that meet two criteria in python

I have a large XML file that I want to parse, and only print one specific value if two values are met. This is the code so far: #!/usr/local/bin/python import xml.etree.ElementTree as ET tree = ET.parse('onedb-dhcp.xml') root = tree.getroot() # This successfully gets all... (1 Reply)
Discussion started by: brianjb
1 Replies

2. Shell Programming and Scripting

Compare two files column values using awk

Judi # cat File1 judi /export/home 76 judi /usr 83 judi # judi # cat File2 judi /export/home 79 judi /usr 82 judi # if COLUMN3 of File2 is greater that COLUMN3 of File1, then print File2's lines juid /export/home 79 Code tags please (2 Replies)
Discussion started by: judi
2 Replies

3. Shell Programming and Scripting

Convert Column Values to a Range of Values

I have a list of columns with values that I need to transform into a row containing the range of each column. For example: "Column A" 1 2 3 4 10 12 14 15 16 17 18 "Column B" 1 4 5 6 (4 Replies)
Discussion started by: newbio
4 Replies

4. Shell Programming and Scripting

Query the table and return values to shell script and search result values from another files.

Hi, I need a shell script, which would search the result values from another files. 1)execute " select column1 from table_name" query on the table. 2)Based on the result, need to be grep from .wft files. could please explain about this.Below is the way i am using. #!/bin/sh... (4 Replies)
Discussion started by: Rami Reddy
4 Replies

5. Shell Programming and Scripting

Converting odd values to even values(or vice-versa) located in a column

Hello All, I have a below data in a .csv file where all rows where col1 is A, col2 is odd numbers, similarly even numbers for all rows where col1 is B. Note that my data has some other columns(not shown here) too (around 100) after col2. Tool,Data A,1 A,3 A,5 .... so on B,2 B,4 .... ... (4 Replies)
Discussion started by: ks_reddy
4 Replies

6. Shell Programming and Scripting

Compare values in two files. For matching rows print corresponding values from File 1 in File2.

- I have two files (File 1 and File 2) and the contents of the files are mentioned below. - I am trying to compare the values of Column1 of File1 with Column1 of File2. If a match is found, print the corresponding value from Column2 of File1 in Column5 of File2. - I tried to modify and use... (10 Replies)
Discussion started by: Santoshbn
10 Replies

7. UNIX for Dummies Questions & Answers

shift values in one column as header for values in another column

Hi Gurus, I have a tab separated text file with two columns. I would like to make the first column values as headings for the second column values. Ex. >value1 subjects >value2 priorities >value3 requirements ...etc and I want to have a file >value1 subjects >value2 priorities... (4 Replies)
Discussion started by: Unilearn
4 Replies

8. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies

9. Shell Programming and Scripting

Awk: Summing values with group criteria

Hi Guys, I have a text file with ";" like separator F1;F2;F3;F4;F5 444;100041;IT;GLOB;1800000000 444;100041;TM;GLOB;1000000000 444;10300264;IT;GLOB;2000000000 444;10300264;IT;GLOB;2500000000 I have to sum the cullums F5 for same F2 and F3 collums The result must be: ... (7 Replies)
Discussion started by: gianluca2
7 Replies

10. Shell Programming and Scripting

How to pick values from column based on key values by usin AWK

Dear Guyz:) I have 2 different input files like this. I would like to pick the values or letters from the inputfile2 based on inputfile1 keys (A,F,N,X,Z). I have done similar task by using awk but in that case the inputfiles are similar like in inputfile2 (all keys in 1st column and values in... (16 Replies)
Discussion started by: repinementer
16 Replies
Login or Register to Ask a Question