Run sed and awk in multiple files in adirectory


 
Thread Tools Search this Thread
Top Forums Programming Run sed and awk in multiple files in adirectory
# 8  
Old 06-05-2018
Saludos


I edited the script as suggested but I still have empty output, here is the report :
Code:
$ ls -sh
total 317M
148M ERR598955_orfm.fna.1   16M ERR598955_orfm.fna.1.test  149M ERR598955_orfm.fna.2  5.3M ERR598955_orfm.fna.2.test  4.0K sov.sh

Code:
[dderilus@boqueron test]$ cat sov.sh 
#!/bin/bash
fmask="ERR598955_orfm.fna"
for f in $fmask.*
do
  [ -f "$f" ] || continue # ensure this is an existing file
  ext=${f#$fmask} # strip off the leading fmask
  ext=${ext#.} # strip off a leading dot
  pattern=$(awk '{x=$1} END {print x}' "$f") # pattern is the 1st word in the last line
  newfile=first_out.$ext
  sed -n '1,/'"${pattern}"'/d; /^>/,$p' "$f" > "$newfile"
done

Code:
[dderilus@boqueron test]$ /bin/bash -x sov.sh 
+ fmask=ERR598955_orfm.fna
+ for f in '$fmask.*'
+ '[' -f ERR598955_orfm.fna.1 ']'
+ ext=.1
+ ext=1
++ awk '{x=$1} END {print x}' ERR598955_orfm.fna.1
+ pattern=GGSSFMGCPSSVMSPASGYSKPAIILNSVVHPIKDDPPHKRSVNTVFQNYALFPHMTVSQNIG
+ newfile=first_out.1
+ sed -n '1,/GGSSFMGCPSSVMSPASGYSKPAIILNSVVHPIKDDPPHKRSVNTVFQNYALFPHMTVSQNIG/d; /^>/,$p' ERR598955_orfm.fna.1
+ for f in '$fmask.*'
+ '[' -f ERR598955_orfm.fna.1.test ']'
+ ext=.1.test
+ ext=1.test
++ awk '{x=$1} END {print x}' ERR598955_orfm.fna.1.test
+ pattern=ERR598955.61408_2_2_1
+ newfile=first_out.1.test
+ sed -n '1,/ERR598955.61408_2_2_1/d; /^>/,$p' ERR598955_orfm.fna.1.test
+ for f in '$fmask.*'
+ '[' -f ERR598955_orfm.fna.2 ']'
+ ext=.2
+ ext=2
++ awk '{x=$1} END {print x}' ERR598955_orfm.fna.2
+ pattern=LSEKKSSQNPLLFSICLIFFWTTFLILPEKAFWRV
+ newfile=first_out.2
+ sed -n '1,/LSEKKSSQNPLLFSICLIFFWTTFLILPEKAFWRV/d; /^>/,$p' ERR598955_orfm.fna.2
+ for f in '$fmask.*'
+ '[' -f ERR598955_orfm.fna.2.test ']'
+ ext=.2.test
+ ext=2.test
++ awk '{x=$1} END {print x}' ERR598955_orfm.fna.2.test
+ pattern=ERR598955.712540_97_1_3
+ newfile=first_out.2.test
+ sed -n '1,/ERR598955.712540_97_1_3/d; /^>/,$p' ERR598955_orfm.fna.2.test
[dderilus@boqueron test]$ ls -sh
total 317M
148M ERR598955_orfm.fna.1   16M ERR598955_orfm.fna.1.test  149M ERR598955_orfm.fna.2  5.3M ERR598955_orfm.fna.2.test     0 first_out.1     0 first_out.1.test     0 first_out.2     0 first_out.2.test  4.0K sov.sh


Regards

Last edited by rbatte1; 06-05-2018 at 12:25 PM.. Reason: Added CODE tags
# 9  
Old 06-06-2018
Apparently
1.
Code:
sed -n '1,/ERR598955.712540_97_1_3/d; /^>/,$p' ERR598955_orfm.fna.2.test

has an empty output, while your initial
2.
Code:
cat ERR598955_orfm.fna.1 | sed -e '1,/ERR598955.7664144_89_2_3/ d' | sed -ne '/^>/,$ p'

works?
Does
3.
Code:
cat ERR598955_orfm.fna.2.test | sed -e '1,/ERR598955.712540_97_1_3/ d' | sed -ne '/^>/,$ p'

work?
I do not see a functional difference between 1. and 3.
Could you post the ERR598955_orfm.fna.2.test? (if too long, only the relevant part, AND WRAP IT IN CODE TAGS for readability, or attach the file in the Advanced Editor).
# 10  
Old 06-07-2018
Dear moderator


Thank you for yor assistance, I finally get the code and and since I have various files on my directory. With a for loop It can work on all my directory. Here is the code :



<<
Code:
for filename in ERR*_orfm.fna.*.test ; do
    last_seq=$(awk '{print $1}' $filename | tail -n1)
    seqname=$(basename $filename .test)
    outname="$seqname".first_out
    echo "cat $seqname | sed -e "1,/$last_seq/ d" | sed -ne '/^>/,$ p' > $outname"
done

>>




Regards




Moderator's Comments:
Mod Comment SERIOUSLY: Please use CODE tags as required by forum rules! You received an infraction for repeatedly ignoring polite requests to do so.

Last edited by RudiC; 06-07-2018 at 11:49 AM.. Reason: Added CODE tags.
# 11  
Old 06-07-2018
The main goal is that you understand
  • how you can modify the manual commands
  • insert variables where the values are variable, and
  • construct the variables from a loop variable.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to use a loop for multiple files in a folder to run awk command?

Dear folks I have two data set which there names are "final.map" and "1.geno" and look like this structures: final.map: gi|358485511|ref|NC_006088.3| 2044 gi|358485511|ref|NC_006088.3| 2048 gi|358485511|ref|NC_006088.3| 2187 gi|358485511|ref|NC_006088.3| 17654 ... (2 Replies)
Discussion started by: sajmar
2 Replies

2. Shell Programming and Scripting

Run one script on multiple files and print out multiple files.

How can I Run one script on multiple files and print out multiple files. FOR EXAMPLE i want to run script.pl on 100 files named 1.txt ....100.txt under same directory and print out corresponding file 1.gff ....100.gff.THANKS (4 Replies)
Discussion started by: grace_shen
4 Replies

3. UNIX for Dummies Questions & Answers

Run one script on multiple files and print out multiple files.

How can I run the following command on multiple files and print out the corresponding multiple files. perl script.pl genome.gff 1.txt > 1.gff However, there are multiples files of 1.txt, from 1----100.txt Thank you so much. No duplicate posting! Continue here. (0 Replies)
Discussion started by: grace_shen
0 Replies

4. Shell Programming and Scripting

Bash Scipting (New); Run multiple greps > multiple files

Hi everyone, I'm new to the forums, as you can probably tell... I'm also pretty new to scripting and writing any type of code. I needed to know exactly how I can grep for multiple strings, in files located in one directory, but I need each string to output to a separate file. So I'd... (19 Replies)
Discussion started by: LDHB2012
19 Replies

5. UNIX for Dummies Questions & Answers

renaming multiple files using sed or awk one liner

hi, I have a directory "test" under which there are 3 files a.txt,b.txt and c.txt. I need to rename those files to a.pl,b.pl and c.pl respectively. is it possible to achieve this in a sed or awk one liner? i have searched but many of them are scripts. I need to do this in a one liner. I... (2 Replies)
Discussion started by: pandeesh
2 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

8. Shell Programming and Scripting

Split line to multiple files Awk/Sed/Shell Script help

Hi, I need help to split lines from a file into multiple files. my input look like this: 13 23 45 45 6 7 33 44 55 66 7 13 34 5 6 7 87 45 7 8 8 9 13 44 55 66 77 8 44 66 88 99 6 I want to split every 3 lines from this file to be written to individual files. (3 Replies)
Discussion started by: saint2006
3 Replies

9. Shell Programming and Scripting

How to run multiple awk files

I'm trying some thing like this. But not working It worked for bash files Now I want some thing like that along with multiple input files by redirecting their outputs as inputs of next command like below Could you guyz p0lz help me on this #!/usr/bin/awk -f BEGIN { } script1a.awk... (2 Replies)
Discussion started by: repinementer
2 Replies

10. UNIX for Dummies Questions & Answers

when I try to run rm on multiple files I have problem to delete files with space

Hello when I try to run rm on multiple files I have problem to delete files with space. I have this command : find . -name "*.cmd" | xargs \rm -f it doing the work fine but when it comes across files with spaces like : "my foo file.cmd" it refuse to delete it why? (1 Reply)
Discussion started by: umen
1 Replies
Login or Register to Ask a Question