Writing to a log file incorrectly

heprox · #1 (**permalink**) 11-16-2006

I have this script:

Code:

#!/bin/ksh
######### Environment Setup #########
PATH=/gers/nurev/menu/pub/sbin:/gers/nurev/menu/pub/bin:/gers/nurev/menu/pub/mac
:/gers/nurev/menu/adm/sbin:/gers/nurev/menu/adm/bin:/gers/nurev/menu/adm/mac:/ge
rs/nurev/custom:/gers/nurev/fix:/gers/nurev/src_rev/fix:/gers/nurev/opt/path:/ge
rs/nurev/bin:/g/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/sbin:.
ORACLE_HOME=/gers/nurev
ORACLE_SID=nurev
export PATH
export ORACLE_HOME
export ORACLE_SID
 
########## Global Variables ##########
DATE=$(date +%m%d%y)
TIME=$(date +%H%M)
DatafilesDir="/gers/nurev/datafiles"
PrintDir="/gers/nurev/print"
TempDir="/gers/nurev/tmp"
Logfile="/tmp/test_str_process_$DATE.log"
arrived=1
 
########## Function to Verify File is Completely Uploaded ###########
# Function : is_file_arrived file
# Arg(s)   : file = file to verify
# Output   : None
# Status   : 0 = yes file arrived, 1 = no
# Env.     : IFA_WAIT : interval (secs) for file size check (def=5)
#
 
is_file_arrived() {
   [ -z "$1" ] && return 1
   local file=$1
   arrived=1
   local size1 size2
   if [ -f "$file" -a -z "$(fuser $file 2> /dev/null)" ] ; then
      size1=$(ls -l $file 2>/dev/null | awk '{print $5}')
      sleep ${IFA_WAIT:-5}
      size2=$(ls -l $file 2>/dev/null | awk '{print $5}')
      [ ${size1:-1} -eq ${size2:-2} ] && arrived=0
   else
      log_it "Store file $file not done uploading at $TIME on $DATE"
   fi
   log_it "Arrived is $arrived"
   return $arrived
 
}
 
######### GERS Processing of File ###########
 
processFile ()
{
   local fileName=$1
   local fileExtension=$2
   local fileNewName="$DatafilesDir/str${fileExtension}.asc"
   local fileODIName="str${fileExtension}.pos"
   mv -Eignore $fileName $fileNewName
   prepup $fileNewName $fileExtension
   mv -Eignore  $PrintDir/$fileODIName $TempDir/$fileODIName
   save2tmp $fileExtension
   call_siu $fileExtension
   log_it "Store file $fileName processed at $TIME on $DATE"
}
 
########## Log File Function ###########
log_it()
{
  printf "%s\n" "$*" >> "$Logfile"
 
}
 
 
########## Main Processing ###########
 
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   for fileName in $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

aigles · #2 (**permalink**) 11-16-2006

When generating file list, ksh returns the pattern itself if no file match the pattern.
If the is no file matching the pattern $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
the for loop is executed one time with the value /gers/nurev/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] (as you can see in the logfile).

Replace the for in loop by :

Code:

ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null | \
while read filename
do
   . . . 
done

heprox · #3 (**permalink**) 11-18-2006

I replaced the "for in" loop but it didn't process the files correctly, the section now looks like:

Code:

nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null \
   while read filename
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

heprox · #4 (**permalink**) 11-18-2006

I also tried to just stop the string "/gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]" from being passed to the "is_file_arrived" function and thus it wouldn't get logged:

Code:

nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   for fileName in $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
   do
      fileExtension=${fileName#*.}
      if [ $fileName <> "/gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0
-9][0-9][0-9]" ] ; then
         is_file_arrived "$fileName"
         if [ $arrived = 0 ]; then
            if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
               nsec=1
               processFile $fileName $fileExtension
            else
               log_it "Store $fileExtension is already in process"
            fi
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

aigles · #5 (**permalink**) 11-19-2006

Quote:

Originally Posted by heprox

I replaced the "for in" loop but it didn't process the files correctly, the section now looks like:

Code:

nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null \
   while read filename
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

You miss the pipeline character between ls and while statements

Code:

nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null | \
   while read filename
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

heprox · #6 (**permalink**) 11-18-2006

I got it to log correctly by having it validate the existance of the input file again before sending it to the "is_file_arrived" function:

Code:

#!/bin/ksh
######### Environment Setup #########
PATH=/gers/genret/menu/pub/sbin:/gers/genret/menu/pub/bin:/gers/genret/menu/pub/
mac:/gers/genret/menu/adm/sbin:/gers/genret/menu/adm/bin:/gers/genret/menu/adm/m
ac:/gers/genret/custom:/gers/genret/fix:/gers/genret/src_rev/fix:/gers/genret/op
t/path:/gers/genret/bin:/g/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/sbin:.
 
ORACLE_HOME=/gers/genret
ORACLE_SID=genret
export PATH
export ORACLE_HOME
export ORACLE_SID
 
########## Global Variables ##########
DATE=$(date +%m%d%y)
TIME=$(date +%H%M)
DatafilesDir="/gers/genret/datafiles"
PrintDir="/gers/genret/print"
TempDir="/gers/genret/tmp"
Logfile="/tmp/str_process_$DATE.log"
 
########## Function to Verify File is Completely Uploaded ###########
# Function : is_file_arrived file
# Arg(s)   : file = file to verify
# Output   : None
# Status   : 0 = yes file arrived, 1 = no
# Env.     : IFA_WAIT : interval (secs) for file size check (def=5)
#
 
is_file_arrived() {
   [ -z "$1" ] && return 1
   local file=$1
   arrived=1
   local size1 size2
   if [ -f "$file" -a -z "$(fuser $file 2> /dev/null)" ] ; then
      size1=$(ls -l $file 2>/dev/null | awk '{print $5}')
      sleep ${IFA_WAIT:-5}
      size2=$(ls -l $file 2>/dev/null | awk '{print $5}')
      [ ${size1:-1} -eq ${size2:-2} ] && arrived=0
   else
      log_it "Store file $file not done uploading at $TIME on $DATE"
   fi
   log_it "Arrived is $arrived"
   return $arrived
 
}
 
######### GERS Processing of File ###########
 
processFile ()
{
   local fileName=$1
   local fileExtension=$2
   local fileNewName="$DatafilesDir/str${fileExtension}.asc"
   local fileODIName="str${fileExtension}.pos"
   mv -Eignore $fileName $fileNewName
  #prepup $fileNewName $fileExtension
  #mv -Eignore  $PrintDir/$fileODIName $TempDir/$fileODIName
  #save2tmp $fileExtension
  #call_siu $fileExtension
   log_it "Store file $fileName processed at $TIME on $DATE"
}
 
########## Log File Function ###########
log_it()
{
  printf "%s\n" "$*" >> "$Logfile"
 
}
 
 
########## Main Processing ###########
 
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   for fileName in $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
   do
      fileExtension=${fileName#*.}
      if [ -f $fileName ]; then
         is_file_arrived "$fileName"
         if [ $arrived = 0 ]; then
            if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
               nsec=1
               processFile $fileName $fileExtension
            else
               log_it "Store $fileExtension is already in process"
            fi
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
writing to a file	amitrajvarma	Shell Programming and Scripting	1	03-10-2008 01:29 AM
Hostname displays incorrectly	shamik	Linux	15	12-01-2007 10:43 AM
Reading a file and writing the file name to a param file.	thebeginer	UNIX for Advanced & Expert Users	1	10-05-2007 05:38 PM
Writing to another file	dreams5617	Shell Programming and Scripting	3	07-05-2006 02:39 PM
Writing to a file in different way	dhananjaysk	Shell Programming and Scripting	4	04-11-2006 12:54 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by