Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Xargs, awk, match, if greater - as a one-liner

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Xargs, awk, match, if greater - as a one-liner
# 8  
Old 08-10-2016
This is what I get from your above sample file
Code:
awk 'match ($0, /PVALUE=[0-9.]*/) && substr($0, RSTART+7, RLENGTH-7) > 0.05' *.txt
chrX	110000	NRHITS=10;PVALUE=0.6
chrX	120000	NRHITS=18;PVALUE=0.2

# 9  
Old 08-10-2016
You are right. The command works but how can i use it with xargs. I have multiple files to process and i want the separate output files for each.
# 10  
Old 08-10-2016
xargs won't help you here. It will put the file names found onto the command line as parameters to awk, like the shell does in above proposal. In either case, awk will work on that input stream writing ALL results to stdout. If you want the output by input file name, you need to redirect within awk.
This User Gave Thanks to RudiC For This Post:
ts89490
# 11  
Old 08-10-2016
The 1st post in this thread explicitly requested that every field in your input files be searched for PVALUE=number. But, the sample data provided never shows more than once such string on an input line and, on lines that do have something matching that pattern, it always appears in the last field on the line. But, we have no indication of whether or not the sample data provided in post #5 in this thread is representative of the actual data that needs to be processed. From the code samples posted, it appears that the submitter wants one output file produced for each input file that contains matched lines. The submitter seems to also want to have 15 copies of awk running in parallel (which only makes sense if those 15 awk commands won't be thrashing CPU and/or disk accesses.

Assuming that parallel processing won't really help much here (and might actually slow down processing), avoiding xargs completely, and assuming that an input line may contain more than one of the patterns above; I would try something more like:
Code:
awk -F';' '
FNR == 1 {
	if(of != "")
		close(of)
	of = FILENAME ".fail"
}
{	for(i = 1; i <= NF; i++)
		if($i ~ /^PVALUE=/ && (substr($i, 8) + 0) > .05) {
			print > of
			next
		}
}' *.txt

Note that this doesn't create an output file for every input file; it only creates an output file if one or more lines in the corresponding input file meets your criteria.

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
ts89490
# 12  
Old 08-10-2016
thanks Don
Parallel processing is not an issue here as I am doing it on a cluster.

The string to match appears invariably in third column but the order of variables NRHITS and PVALUE in third column might vary.

While the code does not write separate output files for each input, I am wondering if a combination of xargs and sed can help. If so, how ?
Thanks

Quote:
Originally Posted by Don Cragun
The 1st post in this thread explicitly requested that every field in your input files be searched for PVALUE=number. But, the sample data provided never shows more than once such string on an input line and, on lines that do have something matching that pattern, it always appears in the last field on the line. But, we have no indication of whether or not the sample data provided in post #5 in this thread is representative of the actual data that needs to be processed. From the code samples posted, it appears that the submitter wants one output file produced for each input file that contains matched lines. The submitter seems to also want to have 15 copies of awk running in parallel (which only makes sense if those 15 awk commands won't be thrashing CPU and/or disk accesses.

Assuming that parallel processing won't really help much here (and might actually slow down processing), avoiding xargs completely, and assuming that an input line may contain more than one of the patterns above; I would try something more like:
Code:
awk -F';' '
FNR == 1 {
	if(of != "")
		close(of)
	of = FILENAME ".fail"
}
{	for(i = 1; i <= NF; i++)
		if($i ~ /^PVALUE=/ && (substr($i, 8) + 0) > .05) {
			print > of
			next
		}
}' *.txt

Note that this doesn't create an output file for every input file; it only creates an output file if one or more lines in the corresponding input file meets your criteria.

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
---------- Post updated at 01:35 PM ---------- Previous update was at 01:12 PM ----------

I received a suggestion of using perl -ne. I used the following command.

Code:
ls *.txt | xargs -I {} sh -c "perl -ne 'if ($_ =~ m/PVALUE=(\d+)/) {if ($1 >= 0.05){print $_}}' {} >{}.fail"

But i get an error:

Code:
syntax error at -e line 1, near "( =~"
syntax error at -e line 1, near "}}"
Execution of -e aborted due to compilation errors.

Any suggestions to correct the above command ?
Last edited by ts89490; 08-10-2016 at 03:24 PM..
# 13  
Old 08-10-2016
Quote:
Originally Posted by ts89490
thanks Don
Parallel processing is not an issue here as I am doing it on a cluster.

The string to match appears invariably in third column but the order of variables NRHITS and PVALUE in third column might vary.

While the code does not write separate output files, I am wondering if a combination of xargs and sed can help. If so, how ?
Thanks
OK. I completely misunderstood your example. I thought your field separator was <semicolon>, but now I'm guessing that <tab> is your field separator, and <semicolon> is a subfield separator in your third field.

And you are wrong. The code I suggested produces a separate output file for each input file that contains lines that meet your criteria.

Using your updated description (but assuming that no <semicolon> characters appear anywhere in the 1st two fields in your input files AND assuming that a single <tab> character separates the first three fields), my code adjusted for your new description of the problem is:
Code:
awk -F'[\t;]' '
FNR == 1 {
	if(of != "")
		close(of)
	of = FILENAME ".fail"
}
{	for(i = 3; i <= NF; i++)
		if($i ~ /^PVALUE=/ && (substr($i, 8) + 0) > .05) {
			print > of
			next
		}
}' *.txt

And, with the following input files:
file1.txt:
Code:
#row1	
#row2	
#row3	
#row4	
#row5	
CHR	POS	INFO
chrX	10000	NRHITS=35;PVALUE=0.04
chrX	109000	NRHITS=6;PVALUE=
chrX	110000	NRHITS=10;PVALUE=0.6
chrX	120000	NRHITS=18;PVALUE=0.2
chrX	130000	NRHITS=39;PVALUE=0.035

file2.txt:
Code:
#row1	
#row2	
#row3	
#row4	
#row5	
CHR	POS	INFO
chrX	10000	PVALUE=0.04;NRHITS=35
chrX	109000	PVALUE=;NRHITS=6
chrX	110000	PVALUE=0.6;NRHITS=10
chrX	120000	PVALUE=0.2;NRHITS=18
chrX	130000	PVALUE=0.035;NRHITS=39

file3.txt:
Code:
#row1	
#row2	
#row3	
#row4	
#row5	
CHR	POS	INFO
chrX	10000	EXTRA=1;NRHITS=35;PVALUE=0.04
chrX	109000	NRHITS=6;PVALUE=;EXTRA=2
chrX	110000	NRHITS=10;EXTRA=3;PVALUE=0.6
chrX	120000	EXTRA=4;NRHITS=18;PVALUE=0.2
chrX	130000	NRHITS=39;PVALUE=0.035;EXTRA=5

file4.txt:
Code:
#row1	
#row2	
#row3	
#row4	
#row5	
CHR	POS	INFO
chrX	10000	NRHITS=35;PVALUE=0.04
chrX	109000	NRHITS=6;PVALUE=
chrX	110000	NRHITS=10;PVALUE=0.006
chrX	120000	NRHITS=18;PVALUE=0.02
chrX	130000	NRHITS=39;PVALUE=0.035

It produces the output files:
file1.txt.fail:
Code:
chrX	110000	NRHITS=10;PVALUE=0.6
chrX	120000	NRHITS=18;PVALUE=0.2

file2.txt.fail:
Code:
chrX	110000	PVALUE=0.6;NRHITS=10
chrX	120000	PVALUE=0.2;NRHITS=18

file3.txt.fail:
Code:
chrX	110000	NRHITS=10;EXTRA=3;PVALUE=0.6
chrX	120000	EXTRA=4;NRHITS=18;PVALUE=0.2

Note that there is no file4.txt.fail file because no line in file4.txt meets your criteria.
This User Gave Thanks to Don Cragun For This Post:
ts89490
# 14  
Old 08-10-2016
WHY do you insist on xargs? You have received some proposals working entirely without it, although they may be somewhat off target as the target is not THAT clear. perl, sed, awk - they all will do what (we think) you need on an input stream of the desired file names.
This User Gave Thanks to RudiC For This Post:
ts89490
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed one Liner inverse range match-help required

cat test.txt a c d e g (2 Replies)
Discussion started by: TomG
2 Replies

2. Shell Programming and Scripting

awk one liner

The below code is a simple modified sample from a file with millions of lines containing hundreds of extra columns xxx="yyy" ... <app addr="1.2.3.4" rem="1000" type="aaa" srv="server1" usr="user1"/> <app usr="user2" srv="server2" rem="1001" type="aab" addr="1.2.3.5"/>What's the most efficient awk... (2 Replies)
Discussion started by: cabrao
2 Replies

3. Shell Programming and Scripting

HELP with AWK one-liner. Need to employ an If condition inside AWK to check for array variable ?

Hello experts, I'm stuck with this script for three days now. Here's what i need. I need to split a large delimited (,) file into 2 files based on the value present in the last field. Samp: Something.csv bca,adc,asdf,123,12C bca,adc,asdf,123,13C def,adc,asdf,123,12A I need this split... (6 Replies)
Discussion started by: shell_boy23
6 Replies

4. Shell Programming and Scripting

Need one liner to search pattern and print everything expect 6 lines from where pattern match made

i need to search for a pattern from a big file and print everything expect the next 6 lines from where the pattern match was made. (8 Replies)
Discussion started by: chidori
8 Replies

5. Shell Programming and Scripting

Search & Replace regex Perl one liner to AWK one liner

Thanks for giving your time and effort to answer questions and helping newbies like me understand awk. I have a huge file, millions of lines, so perl takes quite a bit of time, I'd like to convert these perl one liners to awk. Basically I'd like all lines with ISA sandwiched between... (9 Replies)
Discussion started by: verge
9 Replies

6. Shell Programming and Scripting

Awk one-liner?

Hello, I have two files... File #1 1 3 2 5 File #2 3 5 3 1 3 7 9 1 5 2 5 8 3 3 1 I need to extract all lines from File #2 where the first two columns match each line of File #1. So in the example, the output would be: 1 3 7 2 5 8 Is there a quick one-liner that would... (4 Replies)
Discussion started by: palex
4 Replies

7. UNIX for Dummies Questions & Answers

need an awk one liner

example input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 (3 Replies)
Discussion started by: kenneth.mcbride
3 Replies

8. UNIX for Dummies Questions & Answers

awk one liner

I need a one liner to" find /pattern/ print from x lines before "pattern" to y lines after "pattern" (3 Replies)
Discussion started by: kenneth.mcbride
3 Replies

9. Shell Programming and Scripting

awk one liner

input a 100 200 300 b 400 10 output a 100 a 200 a 300 b 400 b 10 Thanx (6 Replies)
Discussion started by: repinementer
6 Replies

10. Shell Programming and Scripting

AWK greater than?

Sorry for such a basic question, but I have spent hours trying to work this out! I need an awk command (or similar) that will look at a text file and output to the screen if the 4th column of each line has a value greater than or equal to x. data.txt This is the 1 line This is the 2 line This... (6 Replies)
Discussion started by: dlam
6 Replies
Login or Register to Ask a Question