Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Writing to a log file incorrectly

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Writing to a log file incorrectly
# 1  
Old 11-16-2006
Question Writing to a log file incorrectly
I have this script:

Code:
#!/bin/ksh
######### Environment Setup #########
PATH=/gers/nurev/menu/pub/sbin:/gers/nurev/menu/pub/bin:/gers/nurev/menu/pub/mac
:/gers/nurev/menu/adm/sbin:/gers/nurev/menu/adm/bin:/gers/nurev/menu/adm/mac:/ge
rs/nurev/custom:/gers/nurev/fix:/gers/nurev/src_rev/fix:/gers/nurev/opt/path:/ge
rs/nurev/bin:/g/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/sbin:.
ORACLE_HOME=/gers/nurev
ORACLE_SID=nurev
export PATH
export ORACLE_HOME
export ORACLE_SID
 
########## Global Variables ##########
DATE=$(date +%m%d%y)
TIME=$(date +%H%M)
DatafilesDir="/gers/nurev/datafiles"
PrintDir="/gers/nurev/print"
TempDir="/gers/nurev/tmp"
Logfile="/tmp/test_str_process_$DATE.log"
arrived=1
 
########## Function to Verify File is Completely Uploaded ###########
# Function : is_file_arrived file
# Arg(s)   : file = file to verify
# Output   : None
# Status   : 0 = yes file arrived, 1 = no
# Env.     : IFA_WAIT : interval (secs) for file size check (def=5)
#
 
is_file_arrived() {
   [ -z "$1" ] && return 1
   local file=$1
   arrived=1
   local size1 size2
   if [ -f "$file" -a -z "$(fuser $file 2> /dev/null)" ] ; then
      size1=$(ls -l $file 2>/dev/null | awk '{print $5}')
      sleep ${IFA_WAIT:-5}
      size2=$(ls -l $file 2>/dev/null | awk '{print $5}')
      [ ${size1:-1} -eq ${size2:-2} ] && arrived=0
   else
      log_it "Store file $file not done uploading at $TIME on $DATE"
   fi
   log_it "Arrived is $arrived"
   return $arrived
 
}
 
######### GERS Processing of File ###########
 
processFile ()
{
   local fileName=$1
   local fileExtension=$2
   local fileNewName="$DatafilesDir/str${fileExtension}.asc"
   local fileODIName="str${fileExtension}.pos"
   mv -Eignore $fileName $fileNewName
   prepup $fileNewName $fileExtension
   mv -Eignore  $PrintDir/$fileODIName $TempDir/$fileODIName
   save2tmp $fileExtension
   call_siu $fileExtension
   log_it "Store file $fileName processed at $TIME on $DATE"
}
 
########## Log File Function ###########
log_it()
{
  printf "%s\n" "$*" >> "$Logfile"
 
}
 
 
########## Main Processing ###########
 
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   for fileName in $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done


...the script works well, except for that it logs information even when there is no file in the "for" loop, so the log looks like:

Code:
Arrived is 0
Store file /gers/nurev/datafiles/upload.0031 processed at 0127 on 111606
Arrived is 0
Store file /gers/nurev/datafiles/upload.0032 processed at 0127 on 111606
Store file /gers/nurev/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] n
ot done uploading at 0127 on 111606
Arrived is 1
Store file /gers/nurev/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] n
ot done uploading at 0127 on 111606
Arrived is 1

I would like for it to only log activity when it finds and actual file to process?
# 2  
Old 11-16-2006
When generating file list, ksh returns the pattern itself if no file match the pattern.
If the is no file matching the pattern $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
the for loop is executed one time with the value /gers/nurev/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] (as you can see in the logfile).

Replace the for in loop by :
Code:
ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null | \
while read filename
do
   . . . 
done


Jean-Pierre.
# 3  
Old 11-18-2006
Question
I replaced the "for in" loop but it didn't process the files correctly, the section now looks like:

Code:
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null \
   while read filename
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

It didn't create a log file at all, but the process did:

Code:
+ ls /gers/genret/datafiles/UPLOAD.0007 /gers/genret/datafiles/UPLOAD.0012 /gers
/genret/datafiles/UPLOAD.0016 /gers/genret/datafiles/UPLOAD.0017 /gers/genret/da
tafiles/UPLOAD.0039
+ 2> /dev/null
+ fileExtension=
+ is_file_arrived
+ [ 1 = 0 ]
+ read filename
+ fileExtension=
+ is_file_arrived
+ [ 1 = 0 ]
+ read filename
+ fileExtension=
+ is_file_arrived
+ [ 1 = 0 ]
+ read filename
+ fileExtension=
+ is_file_arrived
+ [ 1 = 0 ]
+ read filename
+ fileExtension=
+ is_file_arrived
+ [ 1 = 0 ]
+ read filename
+ sleep 1

I'm thoroughly confused now...
# 4  
Old 11-18-2006
I also tried to just stop the string "/gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]" from being passed to the "is_file_arrived" function and thus it wouldn't get logged:

Code:
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   for fileName in $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
   do
      fileExtension=${fileName#*.}
      if [ $fileName <> "/gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0
-9][0-9][0-9]" ] ; then
         is_file_arrived "$fileName"
         if [ $arrived = 0 ]; then
            if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
               nsec=1
               processFile $fileName $fileExtension
            else
               log_it "Store $fileExtension is already in process"
            fi
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

but it still generates entries in the log file? The look like:

Code:
Arrived is 0
Store file /gers/genret/datafiles/UPLOAD.0007 processed at 1218 on 111806
Arrived is 0
Store file /gers/genret/datafiles/UPLOAD.0012 processed at 1218 on 111806
Arrived is 0
Store file /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
processed at 1218 on 111806

and when I run the script I get:

Code:
+ PATH=/gers/genret/menu/pub/sbin:/gers/genret/menu/pub/bin:/gers/genret/menu/pu
b/mac:/gers/genret/menu/adm/sbin:/gers/genret/menu/adm/bin:/gers/genret/menu/adm
/mac:/gers/genret/custom:/gers/genret/fix:/gers/genret/src_rev/fix:/gers/genret/
opt/path:/gers/genret/bin:/g/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/sbin:.
+ ORACLE_HOME=/gers/genret
+ ORACLE_SID=genret
+ export PATH
+ export ORACLE_HOME
+ export ORACLE_SID
+ + date +%m%d%y
DATE=111806
+ + date +%H%M
TIME=1218
+ DatafilesDir=/gers/genret/datafiles
+ PrintDir=/gers/genret/print
+ TempDir=/gers/genret/tmp
+ Logfile=/tmp/test_str_process_111806.log
+ arrived=1
+ nsec=1
+ date +%H%M
+ [[ 1218 -lt 2340 ]]
+ fileExtension=0007
+ [ /gers/genret/datafiles/UPLOAD.0007 ]
+ 0<> /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
+ is_file_arrived /gers/genret/datafiles/UPLOAD.0007
+ [ 0 = 0 ]
+ [ ! -f /gers/genret/tmp/poll_0007.txt ]
+ nsec=1
+ processFile /gers/genret/datafiles/UPLOAD.0007 0007
+ fileExtension=0012
+ [ /gers/genret/datafiles/UPLOAD.0012 ]
+ 0<> /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
+ is_file_arrived /gers/genret/datafiles/UPLOAD.0012
+ [ 0 = 0 ]
+ [ ! -f /gers/genret/tmp/poll_0012.txt ]
+ nsec=1
+ processFile /gers/genret/datafiles/UPLOAD.0012 0012
+ sleep 1
+ nsec=15
+ date +%H%M
+ [[ 1218 -lt 2340 ]]
+ fileExtension=[0-9][0-9][0-9][0-9]
+ [ /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] ]
+ 0<> /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
+ is_file_arrived /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9
][0-9]
+ [ 0 = 0 ]
+ [ ! -f /gers/genret/tmp/poll_[0-9][0-9][0-9][0-9].txt ]
+ nsec=1
+ processFile /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-
9] [0-9][0-9][0-9][0-9]
Usage: mv [-i | -f] [-E{force|ignore|warn}] [--] src target
   or: mv [-i | -f] [-E{force|ignore|warn}] [--] src1 ... srcN directory
+ sleep 1
+ nsec=15
+ date +%H%M
+ [[ 1218 -lt 2340 ]]
+ fileExtension=[0-9][0-9][0-9][0-9]
+ [ /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] ]
+ 0<> /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
+ is_file_arrived /gers/genret/datafiles/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9
][0-9]

# 5  
Old 11-18-2006
I got it to log correctly by having it validate the existance of the input file again before sending it to the "is_file_arrived" function:

Code:
#!/bin/ksh
######### Environment Setup #########
PATH=/gers/genret/menu/pub/sbin:/gers/genret/menu/pub/bin:/gers/genret/menu/pub/
mac:/gers/genret/menu/adm/sbin:/gers/genret/menu/adm/bin:/gers/genret/menu/adm/m
ac:/gers/genret/custom:/gers/genret/fix:/gers/genret/src_rev/fix:/gers/genret/op
t/path:/gers/genret/bin:/g/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/sbin:.
 
ORACLE_HOME=/gers/genret
ORACLE_SID=genret
export PATH
export ORACLE_HOME
export ORACLE_SID
 
########## Global Variables ##########
DATE=$(date +%m%d%y)
TIME=$(date +%H%M)
DatafilesDir="/gers/genret/datafiles"
PrintDir="/gers/genret/print"
TempDir="/gers/genret/tmp"
Logfile="/tmp/str_process_$DATE.log"
 
########## Function to Verify File is Completely Uploaded ###########
# Function : is_file_arrived file
# Arg(s)   : file = file to verify
# Output   : None
# Status   : 0 = yes file arrived, 1 = no
# Env.     : IFA_WAIT : interval (secs) for file size check (def=5)
#
 
is_file_arrived() {
   [ -z "$1" ] && return 1
   local file=$1
   arrived=1
   local size1 size2
   if [ -f "$file" -a -z "$(fuser $file 2> /dev/null)" ] ; then
      size1=$(ls -l $file 2>/dev/null | awk '{print $5}')
      sleep ${IFA_WAIT:-5}
      size2=$(ls -l $file 2>/dev/null | awk '{print $5}')
      [ ${size1:-1} -eq ${size2:-2} ] && arrived=0
   else
      log_it "Store file $file not done uploading at $TIME on $DATE"
   fi
   log_it "Arrived is $arrived"
   return $arrived
 
}
 
######### GERS Processing of File ###########
 
processFile ()
{
   local fileName=$1
   local fileExtension=$2
   local fileNewName="$DatafilesDir/str${fileExtension}.asc"
   local fileODIName="str${fileExtension}.pos"
   mv -Eignore $fileName $fileNewName
  #prepup $fileNewName $fileExtension
  #mv -Eignore  $PrintDir/$fileODIName $TempDir/$fileODIName
  #save2tmp $fileExtension
  #call_siu $fileExtension
   log_it "Store file $fileName processed at $TIME on $DATE"
}
 
########## Log File Function ###########
log_it()
{
  printf "%s\n" "$*" >> "$Logfile"
 
}
 
 
########## Main Processing ###########
 
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   for fileName in $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9]
   do
      fileExtension=${fileName#*.}
      if [ -f $fileName ]; then
         is_file_arrived "$fileName"
         if [ $arrived = 0 ]; then
            if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
               nsec=1
               processFile $fileName $fileExtension
            else
               log_it "Store $fileExtension is already in process"
            fi
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

# 6  
Old 11-19-2006
Quote:
Originally Posted by heprox
I replaced the "for in" loop but it didn't process the files correctly, the section now looks like:

Code:
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null \
   while read filename
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

You miss the pipeline character between ls and while statements
Code:
nsec=1
while [[ "$(date +%H%M)" -lt 2340 ]]
do
   ls $DatafilesDir/[Uu][Pp][Ll][Oo][Aa][Dd].[0-9][0-9][0-9][0-9] 2>/dev/null | \
   while read filename
   do
      fileExtension=${fileName#*.}
      is_file_arrived "$fileName"
      if [ $arrived = 0 ]; then
         if [ ! -f "$TempDir/poll_$fileExtension.txt" ]; then
            nsec=1
            processFile $fileName $fileExtension
         else
            log_it "Store $fileExtension is already in process"
         fi
      fi
   done
   sleep $nsec
   case $nsec in
      1)   nsec=15;;
      15)  nsec=45;;
      45)  nsec=90;;
      90)  nsec=300;;
      300) nsec=600;;
      *) nsec=900;;
   esac
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Large File masking incorrectly happening Ç delimeter issue

The OS version is Red Hat Enterprise Linux Server release 6.10 I have a script to mask some columns with **** in a data file which is delimeted with Ç , I am using awk for the masking , when I try to mask a small file the awk works fine and masks the required column , but when the file is... (6 Replies)
Discussion started by: LinuxUser8092
6 Replies

2. UNIX for Beginners Questions & Answers

Ls incorrectly says file not found in sftp.

Hi all When I sftp to an Oracle cloud server, to a folder where there are more than 10k files, and list a small subset of files, it works OK. When I try list more than 10k files, it says file not found. Example output below. (FYI ls/mget with 6k files works OK on this server) Has anyone... (3 Replies)
Discussion started by: pdinsdale
3 Replies

3. UNIX for Dummies Questions & Answers

Syslog Messages from Remote Server are not writing to Log File Anymore

Hello All, Server: SUSE Linux Enterprise Server 11.3 (x86_64) Syslog-ng Version: syslog-ng 2.0.9 We have configured a Cisco router to send it's log messages to this server listed above. This has been working just perfectly for the last couple months, but we had never setup the log... (9 Replies)
Discussion started by: mrm5102
9 Replies

4. Shell Programming and Scripting

Need help in writing a script that do internal grep on a log file..and extract acct no's from it..

I need to write a script, which takes the input a log file and create output file with acct no's line by line from selected records with text like (in red) : 18:51:18 | 217863|Acct 0110855565|RC 17608| 16 Subs| 1596 UsgRecs| 2 Secs| 430 CPUms| prmis2:26213 <MoveUsage d aemon needs to run... (7 Replies)
Discussion started by: rkrish
7 Replies

5. Shell Programming and Scripting

Totals in a file - incorrectly displaying

Afternoon, I have a script which creates/modifies data into a formatted csv. The trailer record should display 2 columns, the first is a static entry of "T" to identify it as a trailer record. The 2nd is a total of amounts in a column throughout the entire file. My total isn't displaying... (8 Replies)
Discussion started by: mcclunyboy
8 Replies

6. Shell Programming and Scripting

Format the output from sqlplus while writing to log file.

Hi I have developed bash script to connect to database and execute .sql files. I am logging some statements in to log file using echo. While logging I am adding the date in front of the log statements which makes sense. I am unable to add date in front of output from the sqlplus and sqlldr,... (8 Replies)
Discussion started by: murtymvvs
8 Replies

7. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

8. UNIX for Dummies Questions & Answers

Log File Writing and Reading

Hi all, I have the following shell script code which tries to sftp and writes the log into the log file. TestConnection () { echo 'Connection to ' $DESTUSERNAME@$DESTHOSTNAME $SETDEBUG if ]; then rm $SCRIPT ; fi touch $SCRIPT echo "cd" $REMOTEDIR >> $SCRIPT echo "quit" >>... (10 Replies)
Discussion started by: valluvan
10 Replies

9. UNIX for Advanced & Expert Users

writing the log file into syslog

Hi, we are trying to enable auditing for few oracle 9i database. and right now it writies into adump directory. As adump can be read/write by oracle user so could it be possible to write into syslog while oracle keeps writing to adump . thanks in advance. Pk (2 Replies)
Discussion started by: p4cldba
2 Replies

10. Shell Programming and Scripting

Which Process is writing this Log file!!

Hello , Well I have some /tmp files which are growing very quickly..Can anyone suggest me a way to find which process is logging into this file :confused: ? Thanks very much in Advance!! Mohammed (2 Replies)
Discussion started by: Mohammed
2 Replies
Login or Register to Ask a Question