I am writing a script to automate FTP files from AIX to Mainframe. ( Binary mode, passive connection)
FTP the currently available files(poll a directory to find the list) and put it to mainframe in one connection instance
Verify if all the files are copied to Mainframe successfully
If the FTP command fails due to some issue, try to resend the files and if the issue persist after 5 attempts to resend, inform particular user by email.
If the retry succeeds(send only those files that was not sent., this is in case of partially succeeded transfer; say 5 files , 2 transferred and command failed due to some issue and other 3 not sent... in that case send only those 3 files).
Now again verification of file(s) transferred to mainframe
Below is the script that I have written so far
Issue/Doubt/Question/Help required:
Not able to do FTP with out giving password. Please let me know how to use .netrc here. Also please suggest any good alternative approach
I don't have any idea about verifying the transferred file to Mainframe
Requesting for help / guidance;
Pseudo code/ outline:
Code:
#! /usr/bin/ksh
########################################
#-- Defining Variable --#
########################################
DATE=`date +%Y%m%d`
POLL_DIR=/home/abcd/processing
TMP_DIR=/home/abcd/processing/tmpdir
LOG_DIR=/home/abcd/processing/log
KSH_REC=$POLL_DIR/`basename $0`"_rec" ##-- Recovery file for the script
# This will have current state of the script and will be used at while
# restarting the script after failure(before failing the job updated the rec file)
HOST=`hostname`
LOCK_FILE=`basename $0`_$DATE.lock
STOPPER_FILE=`basename $0`_$DATE.stop
MASTER_LOG_FILE=$LOG_DIR/ftp_to_mvs_$DATE.log
PROCESSED_FILES=$TMP_DIR/processed_files_$DATE.dat
SOURCE_SERVER=`hostname`
REMOTE_SERVER='remoteserver.abcd.com'
REMOTE_PORT=12345 ##-- Dummy, not sharing the details here in open forum.
REMOTE_USER=userid
REMOTE_USER_PASS=userpasswd
POLL_COUNTER=0
########################################
#-- Defining Functions --#
########################################
##--------------------------------------
#-- (1) function FTP to MVS
##--------------------------------------
f_FTP_MVS()
{
echo 'user $REMOTE_USER $REMOTE_USER_PASS' > $TMP_BATCH_FTP
echo 'type binary' >> $TMP_BATCH_FTP
echo 'passive' >> $TMP_BATCH_FTP
cat $TMP_BATCH_FILE >> $TMP_BATCH_FTP
echo 'bye' >> $TMP_BATCH_FTP
ftp -sinv $REMOTE_SERVER $REMOTE_PORT << `cat $TMP_BATCH_FTP`
}
##------------------------------------------------------
#-- (2) function to verify files transferred to MVS
##------------------------------------------------------
f_VERIFY_FTP_MVS()
{
##Use nlist or something to verify the transfer
}
if [[ ! -f $
[[ -f $PROCESSED_FILES ]] && rm -f $PROCESSED_FILES; touch $PROCESSED_FILES || touch $PROCESSED_FILES; ## Remove and touch if it exist
## on first run; if not touch it. This is one time activity
while [ 1 ] ### Future enhancement --> Include a parameter for Date and compare with system time,
# script should execute only for current day 00:00:01 hrs to 23:59:59 hrs else graceful exit;
# include STOPPER FILEs for convenient stoping by support team
do
DATE_TS=`date +%Y%m%d%H%M%S`
** TMP_LOG_FILE=$LOG_DIR/ld_rdy_$DATE_TS.log #--- Remove this file when processed
** TMP_FILE_ld_rdy=$TMP_DIR/ld_rdy_$DATE_TS.dat #--- Remove this file when processed
** TMP_BATCH_FILE=$TMP_DIR/ftp_batch_$DATE_TS.dat #--- Remove this file when processed
** TMP_BATCH_FTP=$TMP_DIR/ld_rdy_$DATE_TS.ftp
FTP_RETRY_COUNTER=0
if [ `ls -lrt $POLL_DIR/*.ld_rdy | awk 'END {print NR}'` -gt 0 ] ##** Change the condition with 1. LS and 2. grep "not found"
then
### Creating the list of currently available load ready file(s) to process(which is to be FTP'ed to MVS
ls -lrt $POLL_DIR/*.ld_rdy | awk '{print $NF}' > $TMP_FILE_ld_rdy
awk '{print "put", $0}' $TMP_FILE_ld_rdy | sed 's/.ld_rdy//g' > $TMP_BATCH_FILE
###########################
### FTP To MVS Part ###
###########################
echo "POLL COUNTER\t:\t"$POLL_COUNTER > $TMP_LOG_FILE
echo "DATE TIME \t:\t"$DATE_TS >> $TMP_LOG_FILE
echo "Trying to FTP below file(s) to Mainframe: "$TARGET_SERVER >> $TMP_LOG_FILE
sed 's/.ld_rdy//g' $TMP_FILE_ld_rdy >> $TMP_LOG_FILE
f_FTP_MVS
### Check for success or failure of FTP command
if [[ $? -ne 0 ]]
then
### Statements following failed FTPs command
### Step 1: Update the log for failure and
echo "\nFailure in transfering file to targer server" >> $TMP_LOG_FILE
FTP_RETRY_COUNTER=$(($FTP_RETRY_COUNTER+1))
sleep 10;
### Step 2: Check if FTP connection can be established
## if not send an email to Support(if Counter is more than 2) stating there is FTP connection issue.
## ----- Create a Function and call it here.
### Step 3: If connection can be established then try re-sending the unprocessed file(S). -- Write a Function for this
## i. Create Processed and unprocessed file list
## ii. Try re-sending the unprocessed file
## iii. If succeeds., update the log and come out of this loop and start Polling
## iv. If fails, Update FTP_RETRY_COUNTER to ++1
## and Call the STEP 2 - Connection check Function if
## FTP_RETRY_COUNTER is less than or equal to 2
## v. If FTP failed again, send a note to support to check on the FTP connection
### Step 4: Verify the files transferred to MVS
echo "exit 1 ||
echo "POLL COUNTER\t:\t"$POLL_COUNTER > $TMP_LOG_FILE
echo "DATE TIME \t:\t"$DATE_TS >> $TMP_LOG_FILE
echo "File(s) not available in the polling directory at this moment.\nPOLLING DIR\t:\t"$POLL_DIR"\n" >> $TMP_LOG_FILE
else
echo "Below files were successfully transferred to target server"; `cat TMP_FILE_ld_rdy | sed 's/.ld_rdy//g'cd`
fi
else
echo "POLL COUNTER\t:\t"$POLL_COUNTER > $TMP_LOG_FILE
echo "DATE TIME \t:\t"$DATE_TS >> $TMP_LOG_FILE
echo "File(s) not available in the polling directory at this moment.\nPOLLING DIR\t:\t"$POLL_DIR"\n" >> $TMP_LOG_FILE
fi
done
Last edited by rbatte1; 12-21-2017 at 09:27 AM..
Reason: Converted text lists to formatted lists with LIST=1 tags, corrected spelling and split long lines to make it more readable
There is an example part way down the page. The file should be readable only to the (process) owner much like SSH keys. The alternate is to use SFTP and SSH keys if the server supports it.
Do either of these help?
As for verifying, I think that SFTP includes that in its return code. For plain FTP, you could get the file back and check that the one sent matches the one that has made the return journey.
Not able to do FTP with out giving password. Please let me know how to use .netrc here. Also please suggest any good alternative approach
[/CODE]
There's nothing wrong with using a .netrc to avoid the need to supply a ftp password in the script if you are comfortable with using ftp and its security risks in the first place.
For a .netrc file to work it MUST be (1) residing in the home directory of the user under which the script (or cron job) is running, and (2) MUST be owned by that user, and (3) MUST have access rights so that only that user can read/write the file (typically '600').
Records in the file MUST be formatted correctly like:
if the ftp nodename is fred, the userid is joe and the password is smith.
Then when ftp is called:
$ ftp fred
ftp will first lookup the node 'fred' in /etc/hosts (or any other resolution method available) to retrieve an ip address. If that is successful ftp looks for a .netrc file in the script users home directory. If it finds .netrc it will retrieve userid/password combination for 'fred' (if it exists) and use it to make the ftp connection.
It's dead easy to see if this is working. Login as the user and interactively issue the command:
$ ftp <nodename>
and if .netrc is configured correctly it will connect instantly without requesting userid, password or anything.
No such thing, passwords is how ftp works. Even the .netrc file contains passwords, though is safer than keeping them in your scripts themselves. It looks like this:
If i use SFTP will I be able to opt for "Binary type and passive mode" of transfer?
Those features deal with limitations in FTP's protocol, and aren't relevant anywhere else. You don't have those options, because sftp isn't ftp. You also don't need them for the same reason. sftp doesn't possess these FTP features at all, so you can consider it to be always "binary" and "passive".
You don't have those options, because sftp isn't ftp. But you don't even need them, because sftp isn't ftp. The issues those solve don't exist in sftp in the first place.
Ok Sure. That was additional question on top of that.
This has resolved the first blocker.
Now how do I verify the transferred file to Mainframe; below are the scenarios:
1. FTP the currently available files(poll a directory to find the list) and put it to mainframe in one connection instance
2. Verify if all the files are copied to Mainframe successfully
3. If the FTP command fails due to some issue, try to resend the files and if the issue persist after 5 attempts to resend, inform particular user by email.
4.If the retry succeeds(send only those files that was not sent., this is in case of partially succeeded transfer; say 5 files , 2 transferred and command failed due to some issue and other 3 not sent... in that case send only those 3 files).
5. Now again verification of file(s) transferred to mainframe
I dont have any idea about how to verify., looking for your advise / discussion.
This discussion thread is an extension to what was discussed in Shell scripting section.
Please refer the post for the requirement:
Requirement Post - Click Here
The whole thread - Click Here
I would like to know how I can use NDM to transfer file from AIX to Mainframe and to verify the... (3 Replies)
Hi,
I have a task of setting up connectivity between Oracle 10g (AIX) and Mainframe (1 library). Went through couple of documents, forums, blogs etc. MY understanding is ODBC Generic Connectivity is free from Oracle side.
Question: (may be dumb to you)
1. Has anybody done this and would... (3 Replies)
Hi All,
The issue is that, we have a dataset in mianframe whose record length is 153 characters. And a batch job ftpies it to the unix server(SunOS) as a test file. But the ftpied file in unix does not have a record length of 153 chars. Each record of 153 chars gets splited into two line of... (8 Replies)
Hi,
I'm not sure if this has been solved in this forum already but please do help me out if possible. Basically, I've already setup a passwordless SSH connection between 2 AIX IDs (say ID-1 and ID-2) with a Mainframe server ID (say MVSID). I'm able to successfully do an SSH from the AIX server to... (1 Reply)
Hi Experts...
Greetings for the day..!
I just want to FTP the files to mainframe system..
my code is not working..and also i need to put the files in a particular directory in a specific naming format...
ftp -i -n ${HOST_NAME} << END_FTP
user ${USER_NAME} ${PASSWORD}
put ${FILE_NAME}... (3 Replies)
Hi All,
I am new to this site, I have a requirement where in i have to FTP a file from mainframe to Unix box.
The catch here is there are few Spanish characters like N with tilde(~) and a with ` etc., all other characters are coming fine but those mentioned above are not coming in a proper... (1 Reply)
suppose i have a file named xyz(-1) and i have to transfer(ftp) it on a Mainframe from unix,how should i do it as whenever i try to do so it says use MVS naming conventions (1 Reply)
hi
suppose i have a file named xyz(-1) and i have to transfer(ftp) it on a Mainframe from unix,how should i do it as whenever i try to do so it says use MVS naming conventions (1 Reply)
Hi
when i am trying to ftp to a newly cretaed mainframe GDG for the first time,i am getting the following error...
Verbose mode on.
200 Representation type is Ebcdic NonPrint
200 Port request OK.
550-SVC99 Return code=4 S99INFO=0 S99ERROR=38668 HEX=970C S99ERSN code X'00004379').
550... (1 Reply)