Copying the Header & footer Information to the Outfile.
Hi
I am writing a perl script which checks for the specific column values from a file and writes to the OUT file.
So the feed file has a header information and footer information.
I header information isaround107 lines i.e.
Starts with
Code:
START-OF-FILE
.......
so on ....
TIMESTARTED=Thu Aug 25 01:03:50 BST 2011
START-OF-DATA
# PRODUCT=Corp/Pfd
After the last line "# PRODUCT=Corp/Pfd" the actual data would start.
The footer information is 4 lines i.e.
Code:
END-OF-DATA
DATARECORDS=1275983
TIMEFINISHED=Thu Aug 25 02:27:02 BST 2011
END-OF-FILE
Now, My perl script is as below:
Code:
#!/usr/bin/perl
$file='file';
open(FILE,$file)|| die ("could not open file $file: $!"); # note minor changes in this line, too
open(OUT1,'>','badfile');
open(OUT2,'>','goodfile');
my @fields;
$line = $_;
while (<FILE>) {
$line = $_;
@fields = split (/\|/, $line);
<<<<<< 1) Here Before going to check the column values, I need to write the HEADER and FOOTER information to the Goodfile. >>>>>>>>>
if( $fields[32] eq "N.A." && $fields[33] eq "N.A." && $fields[34] eq "N.A." && $fields[38] eq "N.A." && ($fields[62] eq "N.A." || $fields[62] eq " "))
{
print OUT1 $line; -----> Badfile
}
else
{
print OUT2 $line; ----> Goodfile
}
}
close FILE;
close OUT1;
close OUT2;
1)Here Before going to check the column values, I need to write the HEADER and FOOTER information to the Goodfile
2) Also, I need to calculate the Number of Records in the Good file and then change the FOOTER Information as:
Code:
END-OF-DATA
DATARECORDS=1275983 --> New Rowcount from the Goodfile
TIMEFINISHED=Thu Aug 25 02:27:02 BST 2011
END-OF-FILE
Could anyone please help me out in solving this. Help would be really appreciated.
The simplest way is to read the whole file in array, split it to four parts, process then and write the result in the output file. Because it's really simple and quick then perhaps you should do it in that way. There are a lot of things in the world else you can do or improve or learn.
But... There is always but, you know. :-) It is definitely not "unix way". Why?
Well. From the famous "The UNIX Time-Sharing System": "... there have always been fairly severe size constraints on the system and its software. Given the partially antagonistic desires for reasonable efficiency and expressive power, the size constraint has encouraged not only economy, but also a certain elegance of design."
You don't believe if I say what recourses did have the first Unix hosts. So I wouldn't - but the word "severe" says for itself. At those time the famous "unix philosophy" was born.
Doug McIlroy summarized it in this way: "This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
And what relation does all this stuff have to your question? Just see:
1. You need the header:
Code:
sed -n '/^START-OF-FILE/,/^START-OF-DATA/p' INPUTFILE >/tmp/header.$$
2. The footer:
Code:
sed -n '/^END-OF-DATA/,$p' INPUTFILE >/tmp/footer.$$
3. You can process your file with your perl script but print the name of your good file in the end of the script:
Code:
goodfile=$(perl process.pl)
Or you can print both names - good and bad one and then split them. Or you can give this name as the argument to the script. You just need to know this name.
4. What is the number of records(lines) in the goodfile?
Code:
goodrecs=$(wc -l "$goodfile")
5. The new footer:
Code:
sed 's/^DATARECORDS=.*$/DATARECORDS='"$goodrecs"'/' /tmp/footer.$$ >/tmp/newfooter.$$
rm /tmp/header.$$ /tmp/*footer.$$ # maybe the goodfile too
The beauty of the shell programming that you can do it incremental, in small pieces. You can test and debug your steps separately. And then, when you get the result, you just append your steps in a small, elegant, and really unix program - a shell script.
Regards,
Andrey (yazu)
===
Well. Sorry for my English. This post was really my English exercise. :-)
Really appreciate for your post. Thanks a lot for your answer and thoughts.
Code:
The simplest way is to read the whole file in array, split it to four parts, process then and write the result in the output file. Because it's really simple and quick then perhaps you should do it in that way. There are a lot of things in the world else you can do or improve or learn.
yes you are correct. I did tried the logic to save the entire file into an array and then tried to divide the parts.
But I was struck to do the following points Inside the script:
1) How to write the footer information into the goodfile inside the perl script.
2) Thought of using a counter to calculate the number of lines and then how do I substitute the number in the footer information.
Really appreciate your thoughts using Unix and I did learn a lot from your post.
Is there any way we can do the same in Perl Script itself.
cat INPUTFILE
START-OF-FILE
.......
so on ....
TIMESTARTED=Thu Aug 25 01:03:50 BST 2011
START-OF-DATA
# PRODUCT=Corp/Pfd
a
b
1
c
d
3
END-OF-DATA
DATARECORDS=1275983
TIMEFINISHED=Thu Aug 25 02:27:02 BST 2011
END-OF-FILE
Good lines are numbers and all others are bad lines. So here a sketch:
Code:
perl -e ' :(
use warnings;
use strict;
my $goodfile = "goodfile";
my $footer_len = 4;
my $datarec_line = 1;
my (@whole, @header, @footer, @goodlines, @badlines);
my $line;
@whole = <>;
do {
$line = shift @whole;
push @header, $line;
} while $line !~ /^START-OF-DATA/;
@footer = splice @whole, -$footer_len;
for (@whole) {
if (/\d/) {
push @goodlines, $_;
} else {
push @badlines, $_;
}
}
$footer[$datarec_line] =~ s/\d+/scalar @goodlines/e;
open my $fh, ">", $goodfile;
print $fh @header, @footer, @goodlines;
close $fh;
print @badlines
' INPUTFILE
Good records go to the goodfile and bad ones to the stdout. The footer is before good records.
You can change this sketch (the definition of good lines, the order of output, the output of bad lines) as you want.
Hi I have below requirements on the script below :
(1) I receive 2 pipe seperated file called OUT.psv and DIFF.psv with a column header.I concatenate the 2 files and create a final.psv file. I want to add another header as START_FILE to the final.psv file . How to achieve this ?
(2) I have... (5 Replies)
This is my file(Target.txt)
name|age|locaction
abc|23|del
xyz|24|mum
jkl|25|kol
The file should be like this
1|03252012
1|name|age|location
2|abc|23|del
2|xyz|24|mum
2|jkl|25|kol
2|kkk|26|hyd
3|4
Column 1 is row indicator
for row 1 and 2, column indicator is 1,for data rows... (1 Reply)
I have two files which are getting sent to a UNIX server in order to be bcp'd into a database. The bcp is failing because there's a header and footer row on the file which give the date of the file and the number of rows in it. That's because the file is also being used for another process, so we... (1 Reply)
Hi All,
I need to write a script that In my file I have to check header and footer records are available or not. If it is available I have to run the script, otherwise I should not. But current script it is checking only the data inside the script. It is avoiding to check Header and Footer... (1 Reply)
Hi,
TASK 1:
I have been using this code to print the information of files kept at "/castor/cern.ch/user/s/sudha/forPooja" in some text file name FILE.txt.
rfdir /castor/cern.ch/user/s/sudha/forPooja | grep data | awk '{print "rfio:///castor/cern.ch/user/s/sudha/forPooja/"$9}' > FILE.txt
... (6 Replies)
Hi,
I need to check whether the incoming file has a header and footer using a UNIX script. The pattern of the header and footer is fixed as follows:
Header: Name,Date
Footer: Count, Total
Name,Date
-------------------------
-------------------------
-------------------------
Count,... (5 Replies)
Hi, Guys,
I want add header and footer in a file. I can add footer using following command:
echo "Footer" >>file.
I don't know how to add header.
Thanks in advance (4 Replies)
Hi All,
I have several txt files i need to enter specific header and footer (both are separate) to all these files how can i do this? plz help..
Regards,
Raghav (4 Replies)
Hi Gurus,
My requirement is, I am passing a file1.dat into this(rowcnt.sh) script,but returning a wrong value of -2.(it should be 4).Becoz my file1.dat contains 6records incl: Header & Footer.(6-2=4)
wrong output:
-------
#sh rowcnt.sh file1.dat
-2 actual_cnt except HDR & FTR
should be:... (3 Replies)
I would like to remove some lines from begining of file (header) and some lines from end of file (footer).
The header/footer lines generated by web-browser when the user upload a file to my webserver.
Example:
-----------------------------7d62af20c052c
Content-Disposition: form-data;... (2 Replies)