perl script on multiple files

aritakum · #1 (**permalink**) 05-20-2009

I have a script that runs on one file (at a time).
like this:
$> perl myscript.pl filename > output

How can I run it on >6000 files and have the output sent out into slightly modified file name
$> perl myscript 6000files> output6000files.new extension

Thanks in anticipation

durden_tyler · #2 (**permalink**) 05-20-2009

In the Bash shell, assuming your current directory contains all your input files that match the pattern "*.txt"

Code:

for i in *.txt
do
  perl myscript.pl $i > $i.new
done

Alternatively, you can open, process and close the files inside your perl program itself.

tyler_durden

doutdes · #3 (**permalink**) 05-20-2009

First list the 6 thousand files in one column, then pipe the output to xargs, telling xargs to execute perl for each file, {} is where xargs with the -i option puts the input file.
the -i.org is for perl to save an original file, and the -p, not really sure if you will need the -p with the perl script

Code:

ls -1 6000files* | xargs -i perl -i.output -p myscript.pl {}

aritakum · #4 (**permalink**) 05-20-2009

Both options did the job for me. Thanks!
NOW, if you want to educate me ( and perhaps a few others) on how to add this in the perl script, so I do not have to run a bash or command line script it would be greatly appreciated.

Cheers

Sm

durden_tyler · #5 (**permalink**) 05-21-2009

Quote:

Originally Posted by aritakum

... how to add this in the perl script, so I do not have to run a bash or command line script...

Suppose you have these text files in your directory that you want to process inside a perl program:

Code:

$
$ cat a.txt
This is the first line in file a.txt
$
$ cat b.txt
This is the first line in file b.txt
This is the second line in file b.txt
$
$ cat c.txt
This is the first line in file c.txt
$
$

Here's one sample program:

Code:

$
$ cat process_files.pl
#!perl -w
# Script to process all *.txt files in current directory and put the output
# in the corresponding *.txt.new files in the current directory again.
my $indir  = ".";  # the input directory name
my $outdir = ".";  # the output directory name
my $i = 0;         # loop index
my $infile;        # the input file name
my $outfile;       # the output file name
while (defined($infile = glob($indir."\\*.txt"))) {             # loop through the txt files in directory $indir
  printf("(%d)\tNow processing file => %s\t",++$i,$infile);
  $outfile = $outdir."\\".substr($infile,rindex($infile,"\\")+1).".new";  # name the output file
  open (IFILE, "<$infile") or die "Can't open $infile: $!";     # open input file for reading
  open (OFILE, ">$outfile") or die "Can't create file: $!";     # create output file for writing
  while (<IFILE>) {                                             # loop through contents of the input file
    # now do all your processing here
    # print to the output file
    print OFILE "We've processed this line -> ".$_;
  }
  close(IFILE) or die "Can't close $infile: $!";                # close input file
  close(OFILE) or die "Can't close $outfile: $!";               # close output file
  printf("Output file => %s\n",$outfile);
}
$
$

I've used the Windows file separator since this is run on Windows + Cygwin. You may want to check the documentation for the functions like glob, print etc. You can set the input and output directories in this code; it was unnecessary to declare or use them here, since we are doing all processing in the current directory.

The execution is shown below:

Code:

$
$
$ perl process_files.pl
(1)     Now processing file => .\a.txt  Output file => .\a.txt.new
(2)     Now processing file => .\b.txt  Output file => .\b.txt.new
(3)     Now processing file => .\c.txt  Output file => .\c.txt.new
$
$ cat a.txt.new
We've processed this line -> This is the first line in file a.txt
$
$ cat b.txt.new
We've processed this line -> This is the first line in file b.txt
We've processed this line -> This is the second line in file b.txt
$
$ cat c.txt.new
We've processed this line -> This is the first line in file c.txt
$
$

Hope that helps,
tyler_durden

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Multiple Files deletion in perl	pulkit	Shell Programming and Scripting	1	02-12-2008 05:55 AM
Perl program to read from multiple files	jyotipg	Shell Programming and Scripting	1	07-19-2006 09:26 PM
Combining Multiple files in one in a perl script	rahulrathod	Shell Programming and Scripting	1	12-18-2005 01:51 AM
Perl - Emailing MULTIPLE files as attachment	Tonka52	Shell Programming and Scripting	3	02-11-2004 01:19 PM
Adding 3 Lines to Multiple c and h files with perl	Lazzar	Shell Programming and Scripting	2	10-28-2003 01:30 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by