Recursive file processing from a path and printing output in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Recursive file processing from a path and printing output in a file
# 1  
Old 12-10-2012
Recursive file processing from a path and printing output in a file

Hi All,

The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the
std xml format in the outputfile.

now, the problem is, script read the directories/sub directories and file contents at one level but it's not searching recursively.

Explain you with example :
path : C:\test has dirA and this dirA has sub dirB and sub dirC and these 2 sub directories has 2 files each. now script should search for files
and reads 2 files from sub directories DirB and pass the content of the file and print the xml in result file. after that it need read sub dirC
files and pass the content to xml str and print/append them in the same resultfile.
Code:
C:\test
dirA
subDirB
fileB1
fileB2
SubDirC
fileC1
fileC2

Code:
#! /usr/bin/perl
use strict;
use warnings;
use File::Basename;
use File::Find;
use Class::CSV;
use Text::CSV;
use XML::TreeBuilder;
use Data::Dumper;
 
#variable/arrays declartions
my ($csv,$inputfilename,$resultfile,$fh,$line,$dir,$fp,$base_dir,$dh,$file,$mycontainerless,$paramsfile,$connection,$resultxmlfile,$config);
my ($tree,$container,$row,$contree,$index,$tag,$tagname);
my (@dir,@dir_names,@filenames,@taglist,@columns);
 
#path declartions
$base_dir = 'C:\Test';
$paramsfile = 'C:\Test\input.csv';
 
@dir = $base_dir;
 
 
#read through the given directory path to look for dir/subdir's
while (@dir) {
$dir = pop(@dir);
opendir($dh, $dir);
 
while($file = readdir($dh)) {
next if $file eq '.';
next if $file eq '..';
 
$file = "$dir/$file";
if (-d $file) {
push(@dir, $file);
push(@dir_names, basename($file));
} 
elsif (-f $file) {
$file =~ s/.*\///;
$file =~ s/\.[^.]+$//;
push(@filenames, $file);
}
}
}
&write_datato_csvfile;
#sleep(25);
print "<!-- ***************************-->\n";
print @dir_names;
print "\n";
print "<!-- ***************************-->\n";
print @filenames;
print "\n";
print "<!-- ***********************************-->\n";
&write_output_xml;
 
# sub functions
#sub function to write input file data into csv format
sub write_datato_csvfile{
open $resultfile, '>>', 'C:\Test\input.csv' or die "Can't open file: $!";
foreach $fp (glob("$dir/*.bat")) {
print $fp;
print "\n";
open ($fh, "<", $fp) or die "can't read open '$fp':";
while ($line = <$fh>) {
$line=~ s/ /,/g;
print $resultfile $line;
print $resultfile "\n";
}
close $fh or die "can't read close '$fp':";
}
#close $fh or die "can't read close '$fp':";
}
 
 
#sub function to write xml format in output file
sub write_output_xml{
 
open ($resultxmlfile, '>>', 'C:\Test\resultxml.xml') or die "Can't open file: $!";
print $resultxmlfile '<?xml version="1.0" encoding="utf-16"?>';
print $resultxmlfile "\n<!-- *************************************-->\n";
print $resultxmlfile "<!-- * *-->\n";
print $resultxmlfile "<!-- * PuTTY Configuration Manager save file- All right reserved. *-->\n";
print $resultxmlfile "<!-- * *-->\n";
print $resultxmlfile "<!-- *****************************-->\n";
print $resultxmlfile "<!-- The following lines can be modified at your own risks. -->\n";
 
 
 
#&write_datato_csvfile;
#sleep(30);
#my $time = 15;
#while($time--){
#sleep(1);
#}
$resultfile = 'C:\Test\input.csv';
my $csv = Text::CSV->new();
open (CSV, "<", $resultfile) or die $!;
 
print "Reading input csv file.... \n";
while (<CSV>) {
next if ($. == 0);
if ($csv->parse($_)) {
@columns = $csv->fields();
#print "$columns[1] \n";
} else {
my $err = $csv->error_input;
print "Failed to parse line: $err";
}
 
 
 
print $resultxmlfile <<EOF
<configuration version="0.1.1.2" savepassword="True">
<root type="database" name="$dir_names[0]" expanded="True">
<container type="folder" name="$dir_names[1]" expanded="True">
<container type="folder" name="$dir_names[2]" expanded="True">
<connection type="PuTTY" name="$filenames[0]">
<connection_info>
<name>$filenames[0]</name>
<protocol>SSH</protocol>
<host>$columns[5]</host>
<port>$columns[4]</port>
<session>Default Settings</session>
<commandline>$columns[9]</commandline>
<description /> 
</connection_info>
<login>
</login>
<password />
<prompt />
</login>
<timeout>
<connectiontimeout>1000</connectiontimeout>
<logintimeout>750</logintimeout>
<passwordtimeout>750</passwordtimeout>
<commandtimeout>750</commandtimeout>
</timeout>
<command>
<command1 />
<command2 />
<command3 />
<command4 />
<command5 />
</command> 
<options>
<loginmacro>False</loginmacro>
<postcommands>False</postcommands>
<endlinechar>10</endlinechar>
</options>
</connection>
EOF
;
}
close CSV;
print $resultxmlfile "</root>\n";
print $resultxmlfile "</configuration>\n";
#close CSV;
print "Output xml file has been generated successfully.. \n";
}


Last edited by Franklin52; 12-10-2012 at 05:44 AM.. Reason: fixed code tags
# 2  
Old 12-14-2012
Can we see the output?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Output path for file from search file

Hello Everyone, I am new to Unix and trying to learn as much as I can. But I do not need to create urgent scripts that I can use instantly. :(:( One of the scripts I am trying to write is essentially something that takes in a .txt file with product names. The product names are... (3 Replies)
Discussion started by: UN007
3 Replies

2. Shell Programming and Scripting

Printing Terminal Output to a Error File

I am having a bash script which is basically invoking a python program to validate the Source Query results against the target query results. I am placing all the queries in a .sql file. I want to write to a Error log file incase if the syntax is wrong or if the column is not present in the... (4 Replies)
Discussion started by: ronitreddy
4 Replies

3. UNIX for Dummies Questions & Answers

Help with printing advance output format from a file

Hi, below 'awk' code was given for my thread 'Help with printing output format from a file ' earlier, however script is not resulting expected output with below file content. cat test_tes123.dml record string("\001") emp_num; /* CHAR(11) NOT NULL*/ date("YYYYMMDD")... (1 Reply)
Discussion started by: AAHinka
1 Replies

4. UNIX for Dummies Questions & Answers

Help with printing output format from a file

Hi, I need help in printing data in below format from file extensions with .dml, i have listed details below file name is test_temp.dml, location in /home/users/test01/test_temp.dml file content: sample_type= record decimal(",") test_type; date("DD-MM-YYYY")(",") test_date... (2 Replies)
Discussion started by: AAHinka
2 Replies

5. UNIX for Dummies Questions & Answers

awk - Rename output file, after processing, same as input file

I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this? ... (12 Replies)
Discussion started by: High-T
12 Replies

6. UNIX for Dummies Questions & Answers

Take output of processing in text file

Hi ALL, I am presently using perl script mukesh.pl I just want to catch its output into another text file . So I am using > File.txt . I am getting output but i want the whole processing of the script into that file please let me know . Thanks in advance Cheers Mukesh (1 Reply)
Discussion started by: mumakhij
1 Replies

7. Shell Programming and Scripting

Printing the output of a db2 query on to an unix file

I want to print the output of a db2 query, on to an unix file in a manner that the columns are separated by 'commas'. Please help me out..thanx in advance (1 Reply)
Discussion started by: prasan_ven
1 Replies

8. Shell Programming and Scripting

processing an input file and then the output file

Hi. I am new to scripting and could really do with some advice on the best way to put a script together. Here is the scenario I am working to; - i will get files via ftp to a tmp directory on the server - all files will have a unique file name but with the same extension (.USM) - for each... (5 Replies)
Discussion started by: yabai
5 Replies

9. Shell Programming and Scripting

Writing output into different files while processing file using AWK

Hi, I am trying to do the following using AWK program. 1. Read the input data file 2. Parse the record and see if it contains errors 3. If the record contains errors, then write it into Reject file, else, write into usual output file or display it on the screen Here is what I have done -... (6 Replies)
Discussion started by: vidyak
6 Replies

10. Filesystems, Disks and Memory

Recursive file processing

I'm developing a generic script to catenate all files found in a given directory. The problem I've got is that the depth of the given directory is unknown, so I end up with some catenated directories. My unix isn't that hot and I'm at a loss. Heres an extract from the script cd $1 for i in... (4 Replies)
Discussion started by: k_mufasa
4 Replies
Login or Register to Ask a Question