Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Script to pick the files from source folder

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Script to pick the files from source folder
# 1  
Old 02-03-2012
Script to pick the files from source folder
Hi,

I wrote a script to pick files based on some logic. when I execute my script, I get the below message ( I bolded the lines in the script where this error message shows up), it says that files are not available and it is a valid scenario if the files are not available and I don't know how to get rid of this message.. I am guessing this is a warning not an error as my script continues after this message..can someone help me on how to get rid of this message?

Message:

ls: FERAL_DATA_*_20121213.FERAL_DAT.TXT: No such file or directory
ls: FERAL_CONTROL_20121213.FERAL_CTRL.TXT: No such file or directory

below is my script..as I am new to unix scripting, I have posted the script to get suggestions from experts.. like I may missed something.. or could have done it in a better way.. please help me!


PHP Code:
# pickdate_paramfile1.sh
###script to pick the files based on the date of run and generate a parameter file #####
#!/bin/sh
. /opt/informatica/infa_shared/SCRIPTS/env_file.ksh
date_str=`date "+%Y%m%d%H%M%S"`
ENVID=$1
logfile=$LOGS/${ENVID}_pickupdate_${date_str}.log
echo Script $0 started at `date` >${logfile}
#### Check if one parameter is received#######
if [ $# -ne 1 ]; then
echo "input parameter for the script is missing" >>${logfile} 2>&1
exit 1
fi
######Connect to DB to fetch the date of run#####
DATE_OF_RUN_DB=`sqlplus -s <<sqlEOF
$ORACLE_USERNAME/$ORACLE_PASSWORD@$ORACLE_DB
set feedback off
set heading off
set linesize 5000
set pagesize 0
set echo off
whenever sqlerror exit failure
whenever oserror exit failure
SELECT TO_CHAR(MAX(LAST_RUN_DATE) +1,'YYYYMMDD') FROM ENV_LOAD_HISTORY WHERE ENV_ID='$ENVID' AND STATUS_OF_RUN ='SUCCESS';
sqlEOF`
if [ $? -ne 0 ]; then
echo "error connecting to database" >>${logfile}
exit 1
fi
#######If DATE_OF_RUN_DB is empty throw message and abend the script
if  [ -z "$DATE_OF_RUN_DB" ]
then
echo "No date of run value fetched from DB" >>${logfile} 2>&1
exit 1
fi
echo "Date of run fetched from DB is - $DATE_OF_RUN_DB" >>${logfile}
####Get the files based on the date received from the DB#####
cd $SOURCE_FOLDER
Data_Files=`ls ${ENVID}_DATA_*_${DATE_OF_RUN_DB}.${ENVID}_DAT.TXT`
control_file=`ls ${ENVID}_CONTROL_${DATE_OF_RUN_DB}.${ENVID}_CTRL.TXT`
#####if control file is not available for that date then look for the files with the least date value in the landing folder and decide the date of run#####
if  [ -z "${control_file}" ]
then
echo "Control File is missing for date $DATE_OF_RUN_DB" >>${logfile} 2>&1
cd $SOURCE_FOLDER
Actual_Date=`ls ${ENVID}_CONTROL_*.${ENVID}_CTRL.TXT| sort -t"_" -k3.8 | head -1 | cut -d"_" -f3| cut -d"." -f1`
####pick files based on the date value
Data_Files_New=`ls $SOURCE_FOLDER/${ENVID}_DATA_*_${Actual_Date}.${ENVID}_DAT.TXT`
control_file_New=`ls $SOURCE_FOLDER/${ENVID}_CONTROL_${Actual_Date}.${ENVID}_CTRL.TXT`
##### if no files for that date then throw error message and notify via email & abend the script
if  [ -z "${Data_Files_New}" -o -z "${control_file_New}"  ]
then
echo "Files are missing for date $Actual_Date" >>${logfile} 2>&1
#### include email notification script#####
exit 1
fi
DATE_OF_RUN=${Actual_Date}
CONTROL_FILE_FINAL=${control_file_New}
echo "Files are $Data_Files_New $control_file_New" >>${logfile}
cp ${Data_Files_New} ${control_file_New} $WORK
if [ $? -ne 0 ]; then
echo "Files are not copied" >>${logfile}
exit 1
else echo "Files are copied to work directory" >>${logfile}
fi
else
DATE_OF_RUN=${DATE_OF_RUN_DB}
CONTROL_FILE_FINAL=${control_file}
echo "Files are $Data_Files $control_file" >>${logfile}
cp ${Data_Files} ${control_file} $WORK
if [ $? -ne 0 ]; then
echo "Files are not copied" >>${logfile}
exit 1
else
echo "Files are copied to work directory" >>${logfile}
fi
fi
echo "date of run = $DATE_OF_RUN" >>${logfile} 2>&1
echo "Control file name is $CONTROL_FILE_FINAL" >>${logfile} 2>&1
#######If DATE_OF_RUN is empty throw message and abend the script
if  [ -z "${DATE_OF_RUN}" ]
then
echo "No date of run value fetched unable to proceed" >>${logfile} 2>&1
exit 1
fi
######Create a workflow parameter file#####
parmfile=$PARM_FILES/${ENVID}_parm.parm
parmfile1=$PARM_FILES/${ENVID}_PARM_I.TXT
##### Check if <EOF> tag is available in control file####
if [ `tail -1 "$CONTROL_FILE_FINAL"` = "<EOF>" ]
then echo "<EOF> tag is available in the $CONTROL_FILE_FINAL" >>${logfile} 2>&1
#####copy contents from control file to parameter file####
touch ${parmfile}
cd $SOURCE_FOLDER
cp $CONTROL_FILE_FINAL $parmfile1
sed '$d' $parmfile1 > $parmfile
rm $parmfile1
echo "DATE_OF_RUN=$DATE_OF_RUN" >>${parmfile}
echo "ENVID=$ENVID" >>${parmfile}
echo "parameter file created is $parmfile"  >>${logfile}
else  echo "<EOF> tag missing in $CONTROL_FILE_FINAL" >>${logfile} 2>&1
exit 1
fi 
Last edited by Vijay81; 02-03-2012 at 02:57 PM..
# 2  
Old 02-05-2012
Did you set SOURCE_FOLDER variable in /opt/informatica/infa_shared/SCRIPTS/env_file.ksh ?

Could you please give us the output of :

Code:
env



as well as

Code:
cat /opt/informatica/infa_shared/SCRIPTS/env_file.ksh

?

Did you check that your files at OS level really contains a pattern that match the date that is picked up from your DB ?

---------- Post updated at 10:38 PM ---------- Previous update was at 10:33 PM ----------

Is you database oracle or informatica ?

You should have

Code:
#!/bin/sh
# pickdate_paramfile1.sh
###script to pick the files based on the date of run and generate a parameter file #####

and NOT:

Code:
# pickdate_paramfile1.sh
###script to pick the files based on the date of run and generate a parameter file #####
#!/bin/sh

the shebang line #!/bin/sh should be the very first line in your script
# 3  
Old 02-06-2012
hi CTSGNB,

thanks for respnding.

below are the details you asked for:

env:
PHP Code:
_=/bin/env
G_BROKEN_FILENAMES=1
HISTSIZE=1000
HOME=/home/wertty
INPUTRC=/etc/inputrc
LANG=en_US.UTF-8
LESSOPEN=|/usr/bin/lesspipe.sh %s
LOGNAME=wertty
LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:
MAIL=/var/spool/mail/wertty
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
PWD=/opt/informatica/infa_shared/SrcFiles/logs
SHELL=/bin/ksh
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_TTY=/dev/pts/0
TERM=xterm
USER=wertty
_AST_FEATURES=UNIVERSE - ucb 
cat /opt/informatica/infa_shared/SCRIPTS/env_file.ksh

PHP Code:
#!/usr/bin/ksh
# Define environment variables for running Informatica workflow
#--------------------------------------------------------------
export SOURCE_FOLDER=/opt/informatica/infa_shared/SrcFiles
export TARGET_FOLDER=/opt/informatica/infa_shared/TgtFiles
export PARM_FILES=/opt/informatica/infa_shared/PARMS
export CACHE=/opt/informatica/infa_shared/Cache
export LOGS=/opt/informatica/infa_shared/SrcFiles/logs
export SESSION_LOGS=/opt/informatica/infa_shared/SessLogs
export WORKFLOW_LOGS=/opt/informatica/infa_shared/WorkflowLogs
export ARCHIVE=/opt/informatica/infa_shared/SrcFiles/arch
export SCRIPT_LOGS=/opt/informatica/infa_shared/SrcFiles/script_logs
export TEMP=/opt/informatica/infa_shared/Temp
export SCRIPTS=/opt/informatica/infa_shared/SCRIPTS
export WORK=/opt/informatica/infa_shared/SrcFiles/work
#----------------------------------------------------------------
# Database credentials
#----------------------------------------------------------------
export ORACLE_HOME=
export ORACLE_DB=ERDW_DEV
export ORACLE_USERNAME=resdt
export ORACLE_PASSWORD=opiutn3 
My database is ORACLE.


I am on LINUX.

Data_Files and control files variables can have no values.. it is not a must that they should have values.. means it a valid scenario not to have files in the directory.But at the same time, I want to get rid of the message that says
ls: FERAL_DATA_*_20121213.FERAL_DAT.TXT: No such file or directory
ls: FERAL_CONTROL_20121213.FERAL_CTRL.TXT: No such file or directory
thanks!

Last edited by Vijay81; 02-06-2012 at 02:04 PM..
# 4  
Old 02-06-2012
Do this :

Code:
cd /opt/informatica/infa_shared/SrcFiles
ls FERAL_DATA_*_20121213.FERAL_DAT.TXT

... and tell me what output and/or error message you get ?

also show what command line do you use when you launch your script ?
Last edited by ctsgnb; 02-06-2012 at 12:29 PM..
# 5  
Old 02-06-2012
I get the same error message:

$ cd /opt/informatica/infa_shared/SrcFiles
$ ls FERAL_DATA_*_20121213.FERAL_DAT.TXT
ls: FERAL_DATA_*_20121213.FERAL_DAT.TXT: No such file or directory

I am not sure if I got your question about the command line... as it is a .sh script, I execute it like- ./scriptname.sh parameter

thanks!
# 6  
Old 02-06-2012
It means what it says -- there's no files matching that in the current directory. What files do you have, or where?
# 7  
Old 02-06-2012
Quote:
I execute it like- ./scriptname.sh parameter
And the "parameter" is ... ?
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request for Shell script to move files from Subfolder to Parent folder and delete sub folder

Hi Team, I am new to shell script and there is a requirement where files should be moved from Subfolder to parent folder. Eg: parent folder --> /Interface/data/test/IN Sub folder -->/Interface/data/test/IN/Invoice20180607233338 Subfolder will be always with timestamp... (6 Replies)
Discussion started by: srivarun15
6 Replies

2. UNIX for Dummies Questions & Answers

Script to delete files in different folder

Hi Experts, i need a little help. i have different folder that contain files that need to be deleted. but those folder contains huge amoung of same with 3 different extention. what i used to do is to delete them using the rm commande rm *.ext *.ext1 *.ext3 what i want to do is to have... (1 Reply)
Discussion started by: yprudent
1 Replies

3. Shell Programming and Scripting

Pick files after specified Time from a folder in Perl

Hi All I am stuck in a problem and wants help What i want to do is I have a directory(say name of that directory is outdir) In that directory(outdir) the files come after some gap of time by some processes. What i want to do is i want to list all the files in that directory after the given... (2 Replies)
Discussion started by: parthmittal2007
2 Replies

4. Shell Programming and Scripting

need a shell script to extract the files from source file and check whether those files existonserve

Hi, I am new to shell scripting.Please help me on this.I am using solaris 10 OS and shell i am using is # echo $0 -sh My requirement is i have source file say makefile.I need to extract files with extensions (.c |.cxx |.h |.hxx |.sc) from the makefile.after doing so i need to check whether... (13 Replies)
Discussion started by: muraliinfy04
13 Replies

5. Shell Programming and Scripting

Need script using cp to transfer files to another folder

Hi guys , how are you doing? I am tryng to make a script using cp that i would like to do this: echo "Enter the name files to tranfer" then user has to enter via keyboard this kind of output file1 file 2 file 3 file x (i mean the number of the files could vary and their name too)... (1 Reply)
Discussion started by: REX:)
1 Replies

6. UNIX for Dummies Questions & Answers

Script for checking the files in a folder

Hi , I am using the below script for checking for a file in a folder. if ; then echo 0 else echo 1 fi Is there any way we can check for files which are starting with GL*.csv.What I am trying to do is , I have to check in a folder for the GL*.csv files if there are any files they I... (6 Replies)
Discussion started by: wangkc
6 Replies

7. Shell Programming and Scripting

Running a script with all files in a folder

Hi I have many files in a folder (like 200) and I want to run a Perl script through all of them. One way would be to do it one by one (which will take forever). But I am sure there is a faster way of doing it. So inside the folder i have files named in no particular way eg. YUI456... (3 Replies)
Discussion started by: phil_heath
3 Replies

8. Shell Programming and Scripting

script for Finding files in a folder and copying to another folder

Hi all, I have a folder '/samplefolder' in which i have some files like data0.txt, data1.txt and data2.txt. I have to search the folder for existence of the file data0.txt first and if found have to copy it to some other file; next i have to search the folder for existence of file... (5 Replies)
Discussion started by: satish2712
5 Replies

9. Shell Programming and Scripting

Script to pick out files from a file which are older than 15 days

Hi , I have a file abc.txt which actually contains a directory and file list -rwxrwxr-x 1 dmadmin mmasapp 334 Dec 03 2001 aafs_006.ksh -rwxrwxr-x 1 dmadmin mmasapp 1270 Nov 13 2001 mcl_deposit_ftp.ksh -rwxrwxr-x 1 dmadmin mmasapp 925 Oct 31 2001 ... (1 Reply)
Discussion started by: viv1
1 Replies

10. Shell Programming and Scripting

Suppres error message when moving files from empty source folder

Hi, I would like to write a shell script that moves files from one folder to another without retrieving the error 'can not acces/find file or folder' when the source folder is empty. Any ideas, Thx in advance, Steven. (2 Replies)
Discussion started by: Steven
2 Replies
Login or Register to Ask a Question