Shell Programming and Scripting

Hi all,
I have a directory have all of the .stat and .dat file :
they are is a pipe separate flat file.
Example:
log-20061202.stat contain 1st line and last line of log-20061202.dat with record count of that day.
Example:
Total record = 240
Tom|02-12-2006|1600 W.Santa Clara|SanJose|95123|1001|ENG <--first
Mike|02-12-2006|23 Clayton Rd|San Francisco|94127|6666|PHY <-- last

log-20061202.dat have
NAME|DATETIME|Address|City|Zip|StudentID|Class
Tom|02-12-2006|1600 W.Santa Clara|SanJose|95123|1001|ENG <-- first
John|02-13-2006|234 Wlliam Rd|Oakland|94321|2324|MATH
..............................................
Mike|02-12-2006|23 Clayton Rd|San Francisco|94127|6666|PHY <--last


For each new log-yyyymmdd.dat file compare the record count from the stat file with the record count of the dat file. Raise an error if they do not match.

For each log-yyyymmdd.stat file have the first and last record of the dat file, confirm the first and last record of the dat file are in stat file. Raise an error if they do not match.


For each record that we are going to output file, the file to output is controled by the DATE field of the .dat file not the timestamp contained in the file name.

The output will have name yyyymmdd.dat and contain
NAME|STUDENTID|CLASS

This is a hard problem for me, please help me to learn more how to use ksh shell to solve it. Thanks for sharing.

Your problem seems like is devided into two questions.

1st being checking the record count for that you can simply do this:

Code:

RUNDATE=`date +%Y%m%d`
COUNTSTAT=`grep "Total record" log-${RUNDATE}.stat | awk -F"=" '{print $2}'` 
COUNTLOG=`wc -l log-${RUNDATE}.dat`

if [ ${COUNTSTAT} -ne ${COUNTLOG} ]
then
     echo "Count of dat ${COUNTLOG} doesn't match count of stat file ${COUNTSTAT}"
fi

Then coming to your second question:

Here the idea is to store First record and last record in different variables and then search for them in dat file

Code:

FIRSTLINE=`head -2 log-${RUNDATE}.stat |tail -1`
LASTLINE=`tail -1 log-${RUNDATE}.stat`

for i in $FIRSTLINE
do
LINECOUNT=`grep -n $i  log-20061202.dat | awk -F":" '{print $1}`
if [ ${LINECOUNT} -eq 1 ] 
then
     echo "FIRST RECORD MATCHED"
else
     echo "FIRST RECORD DIDN'T MATCH"
      exit
fi

for i in $LASTLINE
do
LINECOUNT=`grep -n $i  log-20061202.dat | awk -F":" '{print $1}`
COUNTLOG=`wc -l log-${RUNDATE}.dat`
LASTACTLINE=`expr ${COUNTLOG} - 1`
if [ ${LINECOUNT} -eq ${LASTACTLINE} ] 
then
     echo "LAST RECORD MATCHED"
else
     echo "LAST RECORD DIDN'T MATCH"
     exit
fi

Hi Anubhav,

If you set RUNDATE=`date +%Y%m%d`
then it will check todaydate, but if it run to test for directory and compare yyyymmdd.stat with yyyymmdd.dat then it can not run . For example they check it yesterday or the day before yesterday then it will not work

Thanks for sharing,

Then it will be best to read the date in as a parameter. So when you run the script

Run it as follows:

eg abc.ksh <YYYYMMDD>

and in the code read $1 in rundate

Code:

RUNDATE=$1

Hope that helps!!

The problem I hate is
In .dat file the DATE Field is $2 and and have mm-dd-yyyy hh:mm:ss
like 02-21-2006 12:05:06 , how do I get only 20060221 ???

Can you show me the code one more time and I learn it. Thanks

Code:

echo 'Mike|02-12-2006 12:05:06|23 Clayton Rd|San Francisco|94127|6666|PHY' | nawk -F'|' '{split(substr($2, 1, index($2, " ")-1), t, "-"); print t[3] t[1] t[2]}'

or better yet:

Code:

echo 'Mike|02-12-2006 12:05:06|23 Clayton Rd|San Francisco|94127|6666|PHY' | nawk -F'|' '{split($2, t, "[ -]"); print t[3] t[1] t[2]}'

So you mean
for file in *dat; do
YYYYMMDD=grep "|" $file | nawk -F"|" '{split(substr($2, 1, index($2, " ")-1), t, "-") | print t[3] t[1] t[2]}'

It did not work