Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Run script on multiple files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Run script on multiple files
# 8  
Old 01-13-2014
If you want script to be executed , the path to thme must be set in your $PATH...
# 9  
Old 01-13-2014
Actually I saw that on another post and did put it in the path. Then I tried some other tests; I am not sure what is wrong but I am getting closer. Here I tried to run the script

cat test3a
Code:
'!x[$0]++'


Code:
 for f in F1 F2; do gawk -f /tmp/test3a $f;done

I get the following:
Code:
gawk: /tmp/test3a:1: '!x[$0]++'
gawk: /tmp/test3a:1: ^ Invalid char ''' in expression

If I run it like this:

Code:
for f in F1 F2; do  /tmp/test3a $f;done


Then I get command not found, probably because there is no gawk -f. But I cannot run it without gawk -f. If I do, I get "invalid character" What's missing? Probably it is simple but I have not run awk much so I am not understanding
# 10  
Old 01-13-2014
Remove single quotes from your gawk program file: test3a
Code:
!x[$0]++

This User Gave Thanks to Yoda For This Post:
newbie2010
# 11  
Old 01-13-2014
Yoda, you solved all the problems of getting these scripts to run. Many thanks! The only issue I have left is the one that looks like this:

Code:
{count[$2]++; keyword[$2] = $1}
END {
for (k in count)
{if (count[k] == 2) keyword[k] = "rearranging"
if (count[k] == 3) keyword[k] = "deleting"
if (count[k] > 3)  print k ": Volumes has more than 3 repeats" > /dev/stderr
else print keyword[k] " " k
}
}

does not deal with each file at a time. As a result, it combines lists from different files. Example:

Code:
cat F1
--- FILE-HISTORY-INPUT-TASK    
deleting   /vol/common
deleting   /vol/eng_tsk_0001
rearranging  /vol/eng_tsk_0001
rearranging  /vol/common

Code:
 cat F2
--- FILE-HISTORY-WORK1-TASK     
deleting   /vol/working_vol10
deleting   /vol/working_vol11
rearranging  /vol/working_vol10
rearranging  /vol/working_vol11
lab-nbu{660}$

Code:
lab-nbu{662}$ for f in F1 F2; do gawk -f script_0113 $f;done
rearranging /vol/common
rearranging /vol/eng_tsk_0001
--- FILE-HISTORY-INPUT-TASK

rearranging /vol/working_vol11
--- FILE-HISTORY-WORK1-TASK     
rearranging /vol/working_vol10

Why won't it process each file and move to the next one? Do I need a next command for this? I've tried every way and I can't get this script to process one file at a time.
Last edited by Franklin52; 01-13-2014 at 04:11 PM.. Reason: Code tags
# 12  
Old 01-13-2014
Put the redirection expression in quotes, and give it a awk shebang:
Code:
#!/path/to/gawk -f
{count[$2]++; keyword[$2] = $1}
END {
for (k in count)
{if (count[k] == 2) keyword[k] = "rearranging"
if (count[k] == 3) keyword[k] = "deleting"
if (count[k] > 3)  print k ": Volumes has more than 3 repeats" > "/dev/stderr"
else print keyword[k] " " k
}
}

This one you can call as well with gawk -f script
Alternatively, give it a shell shebang and pass the shell arguments with "$@":
Code:
#!/bin/sh
awk '{count[$2]++; keyword[$2] = $1}
END {
for (k in count)
{if (count[k] == 2) keyword[k] = "rearranging"
if (count[k] == 3) keyword[k] = "deleting"
if (count[k] > 3)  print k ": Volumes has more than 3 repeats" > "/dev/stderr"
else print keyword[k] " " k
}
}' "$@"

This one you can call as well with sh script
# 13  
Old 01-13-2014
To me it looks like it is doing what you want it to do. It is collecting every file's lines, sets the keyword according to No. of occurrences of $2 (always "rearranging" as there's always two), and then, in the END action, outputs what it collected so far, working on each file separately. When you consider that the putput order of (k in count) is undefined in awk, that's the output that you post.
# 14  
Old 01-13-2014
Rudi, you are right that I want to control the output order. I've tried the suggestions but I do get

Code:
lab-nbu{811}$ for f in F1 F2; do gawk -f script_0113 $f;done
rearranging /vol/common
rearranging /vol/eng_tsk_0001
--- FILE-HISTORY-INPUT-TASK

On the 1st file, the header is gone The header was --- FILE-HISTORY-INPUT-TASK

I also notice that on other files the lines /vol/filename are jumbled around. How to control the order so that one group of files stays under the proper header? Don't know how to control order
Last edited by Franklin52; 01-13-2014 at 04:10 PM.. Reason: Please use code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Run script on multiple files

Hi Guys, I've been having a look around to try and understand how i can do the below however havent come across anything that will work. Basically I have a parser script that I need to run across all files in a certain directory, I can do this one my by one on comand line however I... (1 Reply)
Discussion started by: mutley2202
1 Replies

2. Shell Programming and Scripting

Run one script on multiple files and print out multiple files.

How can I Run one script on multiple files and print out multiple files. FOR EXAMPLE i want to run script.pl on 100 files named 1.txt ....100.txt under same directory and print out corresponding file 1.gff ....100.gff.THANKS (4 Replies)
Discussion started by: grace_shen
4 Replies

3. UNIX for Dummies Questions & Answers

Run one script on multiple files and print out multiple files.

How can I run the following command on multiple files and print out the corresponding multiple files. perl script.pl genome.gff 1.txt > 1.gff However, there are multiples files of 1.txt, from 1----100.txt Thank you so much. No duplicate posting! Continue here. (0 Replies)
Discussion started by: grace_shen
0 Replies

4. Shell Programming and Scripting

How to run perl script on multiple files of two directories?

Hi I have 100 files under file A labled 1.txt 2.txt.....100.txt(made up name) I have 1 files under file B labled name.txt How can i run the same perl script on 100 files and file name.txt I want to run perl script.pl A/1.txt B/name.txt perl script.pl A/2.txt B/name.txt ....... perl... (3 Replies)
Discussion started by: grace_shen
3 Replies

5. Shell Programming and Scripting

Bash Scipting (New); Run multiple greps > multiple files

Hi everyone, I'm new to the forums, as you can probably tell... I'm also pretty new to scripting and writing any type of code. I needed to know exactly how I can grep for multiple strings, in files located in one directory, but I need each string to output to a separate file. So I'd... (19 Replies)
Discussion started by: LDHB2012
19 Replies

6. Shell Programming and Scripting

Run perl script on files in multiple directories

Hi, I want to run a Perl script on multiple files, with same name ("Data.txt") but in different directories (eg : 2010_06_09_A/Data.txt, 2010_06_09_B/Data.txt). I know how to run this perl script on files in the same directory like: for $i in *.txt do perl myscript.pl $i > $i.new... (8 Replies)
Discussion started by: ad23
8 Replies

7. Shell Programming and Scripting

How to run multiple awk files

I'm trying some thing like this. But not working It worked for bash files Now I want some thing like that along with multiple input files by redirecting their outputs as inputs of next command like below Could you guyz p0lz help me on this #!/usr/bin/awk -f BEGIN { } script1a.awk... (2 Replies)
Discussion started by: repinementer
2 Replies

8. UNIX for Dummies Questions & Answers

Foreach loop to run a perl script on multiple files

Hi, I have thousands of files in a directory that have the following 2 formats: 289620178.aln 289620179.aln 289620180.aln 289620183.aln 289620184.aln 289620185.aln 289620186.aln 289620187.aln 289620188.aln 289620189.aln 289620190.aln 289620192.aln.... and: alnCDS_1.fasta (1 Reply)
Discussion started by: greptastic
1 Replies

9. Shell Programming and Scripting

1 script or multiple scripts?? - check files, run jobs

Question for anyone that might be able to help: My objective is to eheck if a file (a source file) exists in a directory. If it does then, I'd like to call an application (Informatica ETL file...not necessary to know) to run a program which extracts data and loads it into multiple targets. ... (6 Replies)
Discussion started by: jnanasakti
6 Replies

10. UNIX for Dummies Questions & Answers

when I try to run rm on multiple files I have problem to delete files with space

Hello when I try to run rm on multiple files I have problem to delete files with space. I have this command : find . -name "*.cmd" | xargs \rm -f it doing the work fine but when it comes across files with spaces like : "my foo file.cmd" it refuse to delete it why? (1 Reply)
Discussion started by: umen
1 Replies
Login or Register to Ask a Question