perl script on multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting perl script on multiple files
# 1  
Old 05-20-2009
perl script on multiple files

I have a script that runs on one file (at a time).
like this:
$> perl myscript.pl filename > output

How can I run it on >6000 files and have the output sent out into slightly modified file name
$> perl myscript 6000files> output6000files.new extension

Thanks in anticipation
# 2  
Old 05-20-2009
In the Bash shell, assuming your current directory contains all your input files that match the pattern "*.txt"

Code:
for i in *.txt
do
  perl myscript.pl $i > $i.new
done

Alternatively, you can open, process and close the files inside your perl program itself.

tyler_durden
# 3  
Old 05-20-2009
perl one-liner with xargs for multiple input files

First list the 6 thousand files in one column, then pipe the output to xargs, telling xargs to execute perl for each file, {} is where xargs with the -i option puts the input file.
the -i.org is for perl to save an original file, and the -p, not really sure if you will need the -p with the perl script

Code:
ls -1 6000files* | xargs -i perl -i.output -p myscript.pl {}

# 4  
Old 05-20-2009
Terrific!

Both options did the job for me. Thanks!
NOW, if you want to educate me ( and perhaps a few others) on how to add this in the perl script, so I do not have to run a bash or command line script it would be greatly appreciated.

Cheers

Sm
# 5  
Old 05-21-2009
Quote:
Originally Posted by aritakum
... how to add this in the perl script, so I do not have to run a bash or command line script...
Suppose you have these text files in your directory that you want to process inside a perl program:

Code:
$
$ cat a.txt
This is the first line in file a.txt
$
$ cat b.txt
This is the first line in file b.txt
This is the second line in file b.txt
$
$ cat c.txt
This is the first line in file c.txt
$
$

Here's one sample program:

Code:
$
$ cat process_files.pl
#!perl -w
# Script to process all *.txt files in current directory and put the output
# in the corresponding *.txt.new files in the current directory again.
my $indir  = ".";  # the input directory name
my $outdir = ".";  # the output directory name
my $i = 0;         # loop index
my $infile;        # the input file name
my $outfile;       # the output file name
while (defined($infile = glob($indir."\\*.txt"))) {             # loop through the txt files in directory $indir
  printf("(%d)\tNow processing file => %s\t",++$i,$infile);
  $outfile = $outdir."\\".substr($infile,rindex($infile,"\\")+1).".new";  # name the output file
  open (IFILE, "<$infile") or die "Can't open $infile: $!";     # open input file for reading
  open (OFILE, ">$outfile") or die "Can't create file: $!";     # create output file for writing
  while (<IFILE>) {                                             # loop through contents of the input file
    # now do all your processing here
    # print to the output file
    print OFILE "We've processed this line -> ".$_;
  }
  close(IFILE) or die "Can't close $infile: $!";                # close input file
  close(OFILE) or die "Can't close $outfile: $!";               # close output file
  printf("Output file => %s\n",$outfile);
}
$
$

I've used the Windows file separator since this is run on Windows + Cygwin. You may want to check the documentation for the functions like glob, print etc. You can set the input and output directories in this code; it was unnecessary to declare or use them here, since we are doing all processing in the current directory.

The execution is shown below:

Code:
$
$
$ perl process_files.pl
(1)     Now processing file => .\a.txt  Output file => .\a.txt.new
(2)     Now processing file => .\b.txt  Output file => .\b.txt.new
(3)     Now processing file => .\c.txt  Output file => .\c.txt.new
$
$ cat a.txt.new
We've processed this line -> This is the first line in file a.txt
$
$ cat b.txt.new
We've processed this line -> This is the first line in file b.txt
We've processed this line -> This is the second line in file b.txt
$
$ cat c.txt.new
We've processed this line -> This is the first line in file c.txt
$
$

Hope that helps,
tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

2. Shell Programming and Scripting

Tabbed multiple csv files into one single excel file with using shell script not perl

Hi Experts, I am querying backup status results for multiple databases and getting each and every database result in one csv file. so i need to combine all csv files in one excel file with separate tabs. I am not familiar with perl script so i am using shell script. Could anyone please... (4 Replies)
Discussion started by: ramakrk2
4 Replies

3. UNIX for Dummies Questions & Answers

Run one script on multiple files and print out multiple files.

How can I run the following command on multiple files and print out the corresponding multiple files. perl script.pl genome.gff 1.txt > 1.gff However, there are multiples files of 1.txt, from 1----100.txt Thank you so much. No duplicate posting! Continue here. (0 Replies)
Discussion started by: grace_shen
0 Replies

4. Shell Programming and Scripting

How to run perl script on multiple files of two directories?

Hi I have 100 files under file A labled 1.txt 2.txt.....100.txt(made up name) I have 1 files under file B labled name.txt How can i run the same perl script on 100 files and file name.txt I want to run perl script.pl A/1.txt B/name.txt perl script.pl A/2.txt B/name.txt ....... perl... (3 Replies)
Discussion started by: grace_shen
3 Replies

5. Shell Programming and Scripting

How can I do one liner import multiple custom .pm files in my perl script?

I am new for Perl I want to ask one question. I have around 50 custom packages which i am using in my Perl script. I want to import all .pm packages in my Perl script in an easy way. Right now i have to import each package individually. So Is there any way to do so?? Right Now i am doing like: ... (1 Reply)
Discussion started by: Navrattan Bansa
1 Replies

6. Programming

help need in the perl script that create one xml file form multiple files.

Hi every one, Please excuse me if any grammatical mistakes is there. I have multiple xml files in one directory, I need to create multiple XML files into one XML file.example files like this</p> file1:bvr.xml ... (0 Replies)
Discussion started by: veerubiji
0 Replies

7. Shell Programming and Scripting

Writing a Perl Script that processes multiple files

I want to write a Perl script that manipulates multiple files. In the directory, I have files 250.*chr$.ped where * is from 1 to 1000 and $ is from 1-22 for a total of 22 x 10,000 = 22,000 files. I want to write a script that only manipulates files 250.1chr*.ped where * is from 1 to 22.... (10 Replies)
Discussion started by: evelibertine
10 Replies

8. Shell Programming and Scripting

Run perl script on files in multiple directories

Hi, I want to run a Perl script on multiple files, with same name ("Data.txt") but in different directories (eg : 2010_06_09_A/Data.txt, 2010_06_09_B/Data.txt). I know how to run this perl script on files in the same directory like: for $i in *.txt do perl myscript.pl $i > $i.new... (8 Replies)
Discussion started by: ad23
8 Replies

9. UNIX for Dummies Questions & Answers

Foreach loop to run a perl script on multiple files

Hi, I have thousands of files in a directory that have the following 2 formats: 289620178.aln 289620179.aln 289620180.aln 289620183.aln 289620184.aln 289620185.aln 289620186.aln 289620187.aln 289620188.aln 289620189.aln 289620190.aln 289620192.aln.... and: alnCDS_1.fasta (1 Reply)
Discussion started by: greptastic
1 Replies

10. Shell Programming and Scripting

Combining Multiple files in one in a perl script

All, I want to combine multiple files in one file. Something like what we do on the commad line as follows -> cat file1 file2 file3 > Main_File. Can something like this be done in a perl script very efficiently? Thanks, Rahul. (1 Reply)
Discussion started by: rahulrathod
1 Replies
Login or Register to Ask a Question