Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Filtering based on column values

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Filtering based on column values
# 1  
Old 05-27-2017
Code Filtering based on column values
Hi there,

I am trying to filter a big file with several columns using values on a column with values like (AC=5;AN=10;SF=341,377,517,643,662;VRT=1). I wont to filter the data based on SF= values that are (bigger than 400)

Code:
Chr	Start	End	REF	ALT	GENE	EFFECT	Frequency	Existing	INFO	Number	Zygous
chr18	81447	81447	T	C	NON17	nonsynonymous SNV	0.003056	rs11	AC=5;AN=10;SF=341,377,517,643,662;VRT=1	5	10
chr19	12886	12886	G	C	NON17	nonsynonymous SNV	0.003056	rs14	AC=5;AN=10;SF=348,398,400,741,778;VRT=1	5	10
chr8	84550	84550	G	A	TSPO	nonsynonymous SNV	0.003056	rs18	AC=5;AN=10;SF=606,704,723,736,763;VRT=1	5	10
chr8	82784	82784	C	T	TSPO	nonsynonymous SNV	0.003056	rs19	AC=5;AN=10;SF=372,478,684,708,799;VRT=1	5	10

For example


Code:
Chr	Start	End	REF	ALT	GENE	EFFECT	Frequency	Existing	INFO	Number	Zygous
chr8	84550	84550	G	A	TSPO	nonsynonymous SNV	0.003056	rs18	AC=5;AN=10;SF=606,704,723,736,763;VRT=1	5	10

I would appreciate a help as I am doing that manually
Last edited by daashti; 05-27-2017 at 09:29 AM..
# 2  
Old 05-27-2017
I guess you're talking neither of the 8th nor the INFO column, but of col 10, headed "Number Zygous", in a <TAB> separated file. If the file structure is as given, i.e. col 10 is the only one with = , ; , and , in it, try
Code:
awk 'FNR == 1 || $6 > 400' FS="[,;=]" file
Chr	Start	End	REF	ALT	GENE	EFFECT	Frequency	Existing	INFO	Number	Zygous
chr8	84550	84550	G	A	TSPO	nonsynonymous SNV	0.003056	rs18	AC=5;AN=10;SF=606,704,723,736,763;VRT=1	5	10

Last edited by RudiC; 05-27-2017 at 09:32 AM..
# 3  
Old 05-27-2017
Yes its the 10th column, corrected the typo, however, the header was INFO. Will give it a try

---------- Post updated at 08:00 AM ---------- Previous update was at 07:33 AM ----------

It does filtering but i am focusing on SF= values. I have rows like these
AC=997;AN=1038;SF=0,1;VRT=1

I should be the SF=vales that are bigger than 400. is there a way to make it specific
# 4  
Old 05-27-2017
As long as SF is the third in that column, it will work. It does with your new example. Where and how does it fail?
# 5  
Old 05-27-2017
I don't think its filtering the right way. This is some of the output. it did filter rows but not sure what it did filter. e.g AC=65;AN=92;SF=0,1;VRT=1
it should be above 400
Code:
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT
chr1	65872	.	T	G	2480.51	.	AC=65;AN=92;SF=0,1;VRT=1	GT:GQ:DP:AD:PL
chr1	65893	.	G	A	433.77	.	AC=3;AN=4;SF=0,1;VRT=1	GT:GQ:PL:AD:DP
chr1	65898	.	T	C	2055.11	.	AC=7;AN=10;SF=0,1;VRT=1	GT:DP:AD:PL:GQ
chr1	69511	rs75062661	A	G	822333	.	AC=894;AN=934;SF=0,1;VRT=1	GT:GQ:DP:AD:PL
chr1	69552	rs55874132	G	C	73.21	.	AC=20;AN=22;SF=0,1;VRT=1	GT:PL:AD:DP:GQ
chr1	721450	rs2977675	G	A	18243.4	.	AC=95;AN=188;SF=0,1;VRT=1	GT:PL:AD:DP:GQ

# 6  
Old 05-27-2017
That looks a bit different from what you posted as a sample in post#1. But, none of those new lines will come through:
Code:
awk 'FNR == 1 || $6 > 400' FS="[,;=]" file
#CHROM    POS    ID    REF    ALT    QUAL    FILTER    INFO    FORMAT

# 7  
Old 05-27-2017
yes it is different, does that change the filtering command ?

---------- Post updated at 08:55 AM ---------- Previous update was at 08:42 AM ----------

I tried to change the column number, none of them give the right output. I am guessing it doesn't have an affect on SF= values
Code:
awk 'FNR == 1 || $6 > 400' FS="[,;=]" file
awk 'FNR == 1 || $7 > 400' FS="[,;=]" file
awk 'FNR == 1 || $8 > 400' FS="[,;=]" file
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Filtering records of a csv file based on a value of a column

Hi, I tried filtering the records in a csv file using "awk" command listed below. awk -F"~" '$4 ~ /Active/{print }' inputfile > outputfile The output always has all the entries. The same command worked for different users from one of the forum links. content of file I was... (3 Replies)
Discussion started by: sunilmudikonda
3 Replies

2. Shell Programming and Scripting

Concatenate values in the first column based on the second column.

I have a file (myfile.txt) with contents like this: 1.txt apple is 3.txt apple is 5.txt apple is 2.txt apple is a 7.txt apple is a 8.txt apple is a fruit 4.txt orange not a fruit 6.txt zero isThe above file is already sorted using this command: sort -k2 myfile.txtMy objective is to get... (3 Replies)
Discussion started by: shoaibjameel123
3 Replies

3. UNIX for Dummies Questions & Answers

Repositioning based on column values

Dear all ... I have a file which I want to change the structure based on the values in some columns and I would be grateful if you can help... one of my files looks like ... they all have ten rows 1,0,0 10,0,0 2,0,0 3,0,0 4,1,1 4,1,1 4,1,1 5,0,0 6,0,0 7,0,0 8,0.5,2 9,0.33,3 9,0.33,3... (1 Reply)
Discussion started by: A-V
1 Replies

4. Shell Programming and Scripting

Sum column values based in common identifier in 1st column.

Hi, I have a table to be imported for R as matrix or data.frame but I first need to edit it because I've got several lines with the same identifier (1st column), so I want to sum the each column (2nd -nth) of each identifier (1st column) The input is for example, after sorted: K00001 1 1 4 3... (8 Replies)
Discussion started by: sargotrons
8 Replies

5. Shell Programming and Scripting

Adding values of a column based on another column

Hello, I have a data such as this: ENSGALG00000000189 329 G A 4 2 0 ENSGALG00000000189 518 T C 5 1 0 ENSGALG00000000189 1104 G A 5 1 0 ENSGALG00000000187 3687 G T 5 1 0 ENSGALG00000000187 4533 A T 4 2 0 ENSGALG00000000233 5811 T C 4 2 0 ENSGALG00000000233 5998 C A 5 1 0 I want to... (3 Replies)
Discussion started by: Homa
3 Replies

6. Shell Programming and Scripting

Filtering lines for column elements based on corresponding counts in another column

Hi, I have a file like this ACC 2 2 21 aaa AC 443 3 22 aaa GCT 76 1 33 xxx TCG 34 2 33 aaa ACGT 33 1 22 ggg TTC 99 3 44 wee CCA 33 2 33 ggg AAC 1 3 55 ddd TTG 10 1 22 ddd TTGC 98 3 22 ddd GCT 23 1 21 sds GTC 23 4 32 sds ACGT 32 2 33 vvv CGT 11 2 33 eee CCC 87 2 44... (1 Reply)
Discussion started by: polsum
1 Replies

7. Shell Programming and Scripting

Perl: filtering lines based on duplicate values in a column

Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times. 13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG 13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG 13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT 13 18 1 + chromosome 1,... (5 Replies)
Discussion started by: polsum
5 Replies

8. Shell Programming and Scripting

How to averaging column based on first column values

Hello I have file that consist of 2 columns of millions of entries timestamp and throughput I want to find the average (throughput ) for each equal timestamp before change it to proper format e.g : i want to average 2 coloumnd fot all 1308154800 values in column 1 and then print... (4 Replies)
Discussion started by: aadel
4 Replies

9. Shell Programming and Scripting

How to pick values from column based on key values by usin AWK

Dear Guyz:) I have 2 different input files like this. I would like to pick the values or letters from the inputfile2 based on inputfile1 keys (A,F,N,X,Z). I have done similar task by using awk but in that case the inputfiles are similar like in inputfile2 (all keys in 1st column and values in... (16 Replies)
Discussion started by: repinementer
16 Replies

10. UNIX for Dummies Questions & Answers

Filtering records of a file based on a value of a column

Hi all, I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g. COL1 COL2 COL3 ............... COL47 1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies
Login or Register to Ask a Question