perl script for file processing


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting perl script for file processing
# 1  
Old 03-23-2008
Java perl script for file processing

Aim:

To scan a file and ignore all characters that has an ASCII value from 0 to 31 and 127 to 255 and accept only those characters having an ASCII between 32 and 126.

Script:

#!/usr/local/bin/perl
$filename = "$ARGV[0]";
if (-e $filename)
{
open(OUT, "${filename}") || die "can't open $filename\n";
while (<OUT>){
$found= "";
$stat=0;
chomp $_;
my @charArray = split(//, $_);
my $ref = \@charArray;
foreach (@charArray) {
$val = ord($$ref[$stat]);
if(($val>31)&&($val<127)){
$found = "$found$$ref[$stat]";
}
$stat++;
}
$found = "$found\n";
print $found;
}
close(OUT);
}

Problem:
The code mentioned above runs for 20-25 mins for a 500 MB file. This is very slow.


Can someone let me know if this can be done in a more efficient way so as to reduce the file processing duration?
# 2  
Old 03-23-2008
try this,

Code:
#! /opt/third-party/bin/perl

open(FILE, "<", $ARGV[0]) || die ("unable to open <$!>\n");

while( read(FILE, $data, 1) == 1 ) {
  $ordVal = ord($data);
  print "$ordVal"  if( $ordVal >= 32 && $ordVal <= 126 );
}

close(FILE);

exit(0);

# 3  
Old 03-24-2008
Hi Madhan,

Corrected code:

#!/usr/local/bin/perl
open(FILE, "<", $ARGV[0]) || die ("unable to open <$!>\n");
while( read(FILE, $data, 1) == 1 ) {
if((ord($data)>=32)&&(ord($data)<=126)){
print "$data";
}
if(ord($data)==10){
print "\n";}
}
close(FILE);

Its great it takes just 10 mins now. Is there anything else that can be done to reduce the duration further?
# 4  
Old 03-24-2008
Minor change but this will make a difference

change the following

Quote:
if((ord($data)>=32)&&(ord($data)<=126)){
print "$data";
}
if(ord($data)==10){
print "\n";}
}
to

Code:
print "$data" if((ord($data)>=32)&&(ord($data)<=126));

print "\n" if(ord($data)==10);

# 5  
Old 03-24-2008
Hi Madhan,

Thanks..It still takes 10 mins. I have one question here, if we are reading the entire file and moving one character by character won't it consume valuable memory? For eg in C we can take certain bytes (as first batch) from the file and process it and then follow with the next batch of the file.
Can anything be done here?

Please correct me if I am wrong.
# 6  
Old 03-24-2008
Quote:
It still takes 10 mins.
Are you testing with the same input file ?
And probably with the same load each time ?
# 7  
Old 03-24-2008
Quote:
For eg in C we can take certain bytes (as first batch) from the file and process it and then follow with the next batch of the file
Yes.

Probably you could try something like
Code:
while( read(FILE, $data, 1000) == 1000 ) {

and then split $data and use to process,
I think it will definitely reduce I/O here.

Let me know how it goes ! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

2. Shell Programming and Scripting

Perl script required for processing the data

I have following result.log file (always has 2 lines) which I need to process, cat result.log name.cmd.method,"result","abc","xyz"; name="hello,mine.12345,"&"tree"&" xyz "&" tree "&" xyz", data="way,"&" 1"&"rate-me"&"1"&"rate-me",str="",ret=""; now I need to extract the strings/data as... (4 Replies)
Discussion started by: perlDiva
4 Replies

3. Shell Programming and Scripting

perl problem in processing excel file

Dear all, I got a perl script to write some data into an excel file using Spreadsheet::ParseExcel::SaveParser. After that I find all formulas in the excel file are gone. Does any body encounter this problem or have any work around? (2 Replies)
Discussion started by: eldonlck
2 Replies

4. Programming

help me with perl script xml processing

Hi everyone, I have Xml files in a folder, I need to extract some attribute values form xml files and store in a hash. My xml file look like this. <?xml version="1.0" encoding="UTF-8"?> <Servicelist xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"... (0 Replies)
Discussion started by: pavani reddy
0 Replies

5. Shell Programming and Scripting

perl script processing error

open(IN,"input_file") or die "Can't open Input file.\n"; while (<IN>) { chomp; $line = $_; if($line != '') { print "\nprocessing $line\n"; $size = 0; $hrid = $line; @project_id_array = null; $size = @project_id_array;... (3 Replies)
Discussion started by: vishwakar
3 Replies

6. Shell Programming and Scripting

Perl file processing

I have an input array like : "SVR1" GRP="EVT_BOX06B" SRID=100 MIN=2 "SVR1" GRP="EVT_BOX06B" SRID=200 MIN=1 "SVR2" GRP="ADM_BOX06B" SRID=100 MIN=1 "SVR1" GRP="EVT_BOX88B" SRID=100 MIN=2 "SVR1" GRP="EVT_BOX88B" SRID=200 MIN=1... (4 Replies)
Discussion started by: deo_kaustubh
4 Replies

7. Shell Programming and Scripting

Processing a file in perl

Qspace ABC Queue doCol: true Queue order: fifo Queue setCol: red Queue order: fifo Qspace XYZ Queue getCol: true Queue order: fifo I need to append every line in this file with Qspace & Queue, so that final o/p shall look like this, Qspace: ABC Queue: doCol Qspace: ABC Queue: doCol... (2 Replies)
Discussion started by: deo_kaustubh
2 Replies

8. Shell Programming and Scripting

Simple Script needed for Processing CSV file perl

I am new to perl, and need a script to pull a CSV file, strip out 2 leading columns, and 2 ending columns, and resave the file in a new location. The file is basic and has less than 10,000 records. Secondly, can I schedule perl scripts to run daily? Can someone provide the basic script to... (1 Reply)
Discussion started by: cobbjob
1 Replies

9. Shell Programming and Scripting

awk, perl Script for processing a single line text file

I need a script to process a huge single line text file: The sample of the text is: "forward_inline_item": "Inline", "options_region_Australia": "Australia", "server_event_err_msg": "There was an error attempting to save", "Token": "Yes", "family": "Family","pwd_login_tab": "Enter Your... (1 Reply)
Discussion started by: hmsadiq
1 Replies

10. Shell Programming and Scripting

File processing on perl

Hey everyone ... I wanted to process the contents of a file, as in modify its contents. whats the best way to do it on perl? In more detail I hav to go through the contents of the file, match patterns n then modify the contents of the same file depending on the matching results. Any help is... (2 Replies)
Discussion started by: garric
2 Replies
Login or Register to Ask a Question