Xargs to call python executable to process multiple bam files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Xargs to call python executable to process multiple bam files
# 1  
Old 12-20-2019
Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when
| xargs sh is added I get the error. I tried adding | xargs python instead, but get a different error where the standard options are not reconized. It seems like the executable is not be called
, but I do not know enough python to troubleshoot. Thank you Smilie.

Code:
xxx_00.bam
yyy_01.bam
zzz_02.bam

Code:
echo "${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py $(for f in *.bam; do echo -n "--bam ${f} "; done;) --referenceFasta ${fasta} --callRegions ${bed} --exome --runDir ${dir} ${dir}/runWorkflow.py -m local -j 20" | xargs sh

echo only
set -x
Code:
echo "${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py $(for f in *.bam; do echo -n "--bam ${f} "; done;) --referenceFasta ${fasta} --callRegions ${bed} --exome --runDir ${dir} ${dir}/runWorkflow.py -m local -j 20" | xargs sh
+ xargs sh
++ for f in '*.bam'
++ echo -n '--bam xxx_00.bam '
++ for f in '*.bam'
++ echo -n '--bam yyy_01.bam '
++ for f in '*.bam'
++ echo -n '--bam zzz_02.bam '
+ echo 'path/to/configureStrelkaGermlineWorkflow.py --bam xxx_00.bam --bam yyy_01.bam --bam zzz_02.bam  --referenceFasta hg19.fasta --callRegions file.bed --exome --runDir path/to/data path/to/data/runWorkflow.py -m local -j 20'

error
Code:
/path/to/bin/configureStrelkaGermlineWorkflow.py: line 23: $'\nThis script configures the strelka germline small variant calling workflow\n': command not found
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
/path/to/bin/configureStrelkaGermlineWorkflow.py: line 27: syntax error near unexpected token `('
/path/to/bin/configureStrelkaGermlineWorkflow.py: line 27: `if sys.version_info >= (3,0):'


Last edited by cmccabe; 12-20-2019 at 05:44 PM..
# 2  
Old 12-20-2019
(3,0) A parenthesis is a reserved symbol in bash, not a string.
"(3,0)" Try making it a string...
This User Gave Thanks to jim mcnamara For This Post:
# 3  
Old 12-21-2019
xargs is used to turn a stream on stdin into command line arguments / parameters. sh expects its commands on stdin. Try without xargs, like
Code:
echo "..." | sh


Or, use your .py as the xargs' command but make sure it will be interpreted by python (by defining the correct "shebang").
This User Gave Thanks to RudiC For This Post:
# 4  
Old 12-23-2019
I changed the shebang line in the python executable to #!/usr/bin/env python to match the other .py on my system and get the below error.

Code:
    python --version on centos 7
    Python 2.7.5

Code:
    echo "${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py $(for f in *dups.bam; do echo -n "--bam ${f} "; done;) --referenceFasta ${fasta} --callRegions ${bed} --exome --runDir ${dir} ${dir}/runWorkflow.py -m local -j 20" | python
    
    
    File "<stdin>", line 1
    ${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py --bam xxx_00.bam --bam yyy_01.bam --bam zzz_02.bam  --referenceFasta ${fasta} --callRegions ${bed} --exome --runDir ${dir} ${dir}/runWorkflow.py -m local -j 20
                                                                                                                                                                                                ^
                                                                                                                                                                          SyntaxError: invalid syntax
         .... | xargs python  
                                                                                                                                                                          
            error: no such option: -m  this is a standard option
            
         .... | sh
                                                                                                                                                                          
            error: no such option: -m    this is a standard option
            
         .... | xargs sh
                                                                                                                                                                          
           ${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py: line 23: $'\nThis script configures the strelka germline small variant calling workflow\n': command not found
           import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
           ${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py: line 27: syntax error near unexpected token `('
           ${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py: line 27: `if sys.version_info >= (3,0):'

I also tried
Code:
    echo "${path_to_strelka}/bin/configureStrelkaGermlineWorkflow.py $"(for f in *dups.bam; do echo -n "--bam ${f} "; done;)" --referenceFasta ${fasta} --callRegions ${bed} --exome --runDir ${dir} ${dir}/runWorkflow.py -m local -j 20" | xargs sh

    -bash: syntax error near unexpected token `('

Thank you Smilie.

Last edited by cmccabe; 12-23-2019 at 09:58 AM.. Reason: added details
# 5  
Old 12-28-2019
Code:
ls *dups.bam \
| xargs -I % echo --bam % \
| xargs python "${path_to_strelka}"/bin/configureStrelkaGermlineWorkflow.py \
--referenceFasta "${fasta}" --callRegions "${bed}" --exome \
--runDir "${dir}" "${dir}"/runWorkflow.py -m local -j 20

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script with python slicing on multiple data files

I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a... (0 Replies)
Discussion started by: jdubbz
0 Replies

2. UNIX for Advanced & Expert Users

How to process multiple files parallely

Hi, I have a requirement to process multiple files in a directory parallely.Consider the below scenario: In a directory there are three files file1,file2 and file3.When I use for loop each file will be executed in sequence but I want to process parallely. Any Help would be appreciated.... (1 Reply)
Discussion started by: liyakathali
1 Replies

3. Shell Programming and Scripting

Make multiple awk files into an executable

Hello everyone, The following are my input files. The following are my sequence of steps. Can someone please let me know about how to make these bunch of steps into a single script so that I start the script with 1.txt and 2.txt, after execution gives me the final... (11 Replies)
Discussion started by: jacobs.smith
11 Replies

4. Programming

Call a C programming executable inside a C program

hi guys i have only basic knowledge of C so guys plz help me ..... is C language support call the C executable inside the C ?? example contect mainA.c have a many function define in the struct,when i compile mainA and make a executable the name is ( A ),can i use executable C inside the C... (5 Replies)
Discussion started by: isnoname
5 Replies

5. Programming

how to call c executable inside c program??

hi guys i have only basic knowledge of c so guys plz help me ..... i want 2 call c executable which requires file name as argument and i need to modify file contents before calling that executable now my question is how can i call this c executable inside another c program with arguments ?? i... (9 Replies)
Discussion started by: zedex
9 Replies

6. Shell Programming and Scripting

Unable to call executable from script

Now I am using the HP-UX11.11 version. The scripts are runninh in KSH shell. While I wan to call one executable of any Pro*C file, I have got the following error, however the executable is running fine directly. testpri Started at 10.05.200923:40 /usr/lib/dld.sl: Bad magic number for... (0 Replies)
Discussion started by: priyankak
0 Replies

7. Shell Programming and Scripting

Need to call an Executable (.exe) using shell

Hi all , I need to call an executable (.exe) using shell script. Actual need is i need to call that shell script and do an export of some tables is there any way . the executable is datamover Please let me know if there are any option !! (2 Replies)
Discussion started by: raghav1982
2 Replies

8. UNIX for Dummies Questions & Answers

How do I Process Multiple Files

Hi, First post here. In Mac OSX terminal I need to run a program against multiple files in a directory and append the output to tab separated variable file. I currently type the following to process just one file MBP:/Users/dc1743/Desktop/SFT root# ./myprogram myfile1.plist >>... (1 Reply)
Discussion started by: dc1743
1 Replies

9. Programming

function call in executable

Hi, I want to write a program in C or in Perl which will tell me that a function is called in which executables. I tried to use the unix command like 'nm', 'strings' and so on to find out whether a function is called in that executable or not but could not able to find a clue. The whole... (1 Reply)
Discussion started by: rocky_74
1 Replies
Login or Register to Ask a Question