Now how do I verify the transferred file to Mainframe
If you're using FTP, some FTP clients support ls -l, but it sounds like you need a lot more than that.
Quote:
Below are the scenarios:
1. FTP the currently available files(poll a directory to find the list) and put it to mainframe in one connection instance
2. Verify if all the files are copied to Mainframe successfully
3. If the FTP command fails due to some issue, try to resend the files and if the issue persist after 5 attempts to resend, inform particular user by email.
4.If the retry succeeds(send only those files that was not sent., this is in case of partially succeeded transfer; say 5 files , 2 transferred and command failed due to some issue and other 3 not sent... in that case send only those 3 files).
5. Now again verification of file(s) transferred to mainframe
If sftp's an option, have you considered rsync? You can consider it to be "cp on steroids". It's an extremely common UNIX utility which connects with ssh (same protocol as sftp) and does a lot of what you're asking for, especially the verification of files.
i.e. you can do rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder and it will copy and verify. It will even skip already-existing files based on checksum.
If you're using FTP, some FTP clients support ls -l, but it sounds like you need a lot more than that.
If sftp's an option, have you considered rsync? You can consider it to be "cp on steroids". It's an extremely common UNIX utility which connects with ssh (same protocol as sftp) and does a lot of what you're asking for, especially the verification of files.
i.e. you can do rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder and it will copy and verify. It will even skip already-existing files based on checksum.
It would make your task very easy.
Thanks Much Corona688., appreciate your help.
I have to implement this functionality using FTP.
Ok., since you have given me the above code I would like to ask few questions.
A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
If so can you please throw some light?
B. Is using SFTP less secure when compared to FTP -s to transfer files from AIX to Mainframe.? Will there be any loss in data .i.e., will the data get corrupted?
And per my understanding if we go with SFTP method, we can used rsync to copy as well as verify the file.
Even in case of failure, restartability is assured(i.e, only files that are not copied will be copied to the mainframe).
Update:- I will not be able to use SFTP
Last edited by TechGyaann; 12-21-2017 at 05:28 PM..
The nice thing about FTP - and also the problem with FTP - is that it has zero features.
Can you ask FTP if the file uploaded successfully? Nope.
Can you ask FTP for a file checksum or anything like that to verify it? Nope.
Can it tell you the most recent file? Nope.
Can you ask it if a file is there at all? Nope.
Can it tell you anything? Nope. Well, a directory listing. That's it.
So, you can depend on FTP being reliably unhelpful. But there's old-fashioned ways to trick it into doing what you want:
This prevents partial uploads from appearing in /remote/complete. If the connection is broken early, the file will never get renamed.
Then you can get the directory listing from /remote/complete to see if the file you wanted is in it.
Quote:
A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
It will require brute force.
Connect with FTP, saving all output to a file.
Get directory listing.
Quit.
Process directory listing in script to remove all lines except the directory listing.
Check if the directory listing contains the file you uploaded.
That sort of thing.
Quote:
B. Is using SFTP less secure when compared to FTP -s to transfer files from AIX to Mainframe.?
Do you even have ftp -s? That's a very rare feature.
Quote:
Will there be any loss in data .i.e., will the data get corrupted?
sftp is less likely to corrupt your data than FTP.
FTP's "binary" feature is only there because, in the beginning, ftp was expected to translate text files automatically (i.e. EBCDIC to ASCII) so the recipient could read them. FTP mangles files because it's trying to process all downloads as text! And it always does that unless you force it not to with binary! sftp just doesn't do that, at all, period. It transfers the file as-is, always.
Wow.. that's lot of learning from just few interaction..
Quote:
This prevents partial uploads from appearing in /remote/complete. If the connection is broken early, the file will never get renamed.
Then you can get the directory listing from /remote/complete to see if the file you wanted is in it.
Quote:
A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
It will require brute force.
Connect with FTP, saving all output to a file.
Get directory listing.
Quit.
Process directory listing in script to remove all lines except the directory listing.
Check if the directory listing contains the file you uploaded.
That sort of thing.
This is what I understood.
Step 1. Open a Control channel & data channel.. put the file
Step 2. Create a nlist to get the info of files transferred (if any other way please let me know)
Step 3. Quit
Step 4. Compare the nlist output and the list of files that was supposed to be transferred. If not transfer only rest of the files.
Please advise if any discrepancy in above steps using FTP method
Using SFTP method:
U said its interactive, may I please request you to give me examples or example scripts or few links for example/ to learn.
Also tell me the steps here
More over for SFTP should we use only port 21 or the same control port used for FTP can be used to connect to Mainframe. (Asking this question because I am not able to connect to mainframe using sftp)
Step 2. Create a nlist to get the info of files transferred (if any other way please let me know)
If your FTP server and client support nlist, that's the best way to get the file list. Not all of them do.
Quote:
Please advise if any discrepancy in above steps using FTP method
You've got the idea, I think.
Quote:
Using SFTP method:
U said its interactive, may I please request you to give me examples or example scripts or few links for example/ to learn.
Nobody said that, and I'm not sure what you mean by it.
Quote:
More over for SFTP should we use only port 21 or the same control port used for FTP can be used to connect to Mainframe. (Asking this question because I am not able to connect to mainframe using sftp)
not connecting or not prompting for password.
sftp is not ftp. Period. Full stop. It is not related to FTP at all, and does not inter-operate with it. Trying to connect to your FTP server with sftp is as nonsensical as connecting to your mail server with it. It's just not going to work. There is no control channel, either. You just connect to your destination directly.
sftp is actually a use of ssh protocol -- secure shell, i.e. port 22.
This discussion thread is an extension to what was discussed in Shell scripting section.
Please refer the post for the requirement:
Requirement Post - Click Here
The whole thread - Click Here
I would like to know how I can use NDM to transfer file from AIX to Mainframe and to verify the... (3 Replies)
Hi,
I have a task of setting up connectivity between Oracle 10g (AIX) and Mainframe (1 library). Went through couple of documents, forums, blogs etc. MY understanding is ODBC Generic Connectivity is free from Oracle side.
Question: (may be dumb to you)
1. Has anybody done this and would... (3 Replies)
Hi All,
The issue is that, we have a dataset in mianframe whose record length is 153 characters. And a batch job ftpies it to the unix server(SunOS) as a test file. But the ftpied file in unix does not have a record length of 153 chars. Each record of 153 chars gets splited into two line of... (8 Replies)
Hi,
I'm not sure if this has been solved in this forum already but please do help me out if possible. Basically, I've already setup a passwordless SSH connection between 2 AIX IDs (say ID-1 and ID-2) with a Mainframe server ID (say MVSID). I'm able to successfully do an SSH from the AIX server to... (1 Reply)
Hi Experts...
Greetings for the day..!
I just want to FTP the files to mainframe system..
my code is not working..and also i need to put the files in a particular directory in a specific naming format...
ftp -i -n ${HOST_NAME} << END_FTP
user ${USER_NAME} ${PASSWORD}
put ${FILE_NAME}... (3 Replies)
Hi All,
I am new to this site, I have a requirement where in i have to FTP a file from mainframe to Unix box.
The catch here is there are few Spanish characters like N with tilde(~) and a with ` etc., all other characters are coming fine but those mentioned above are not coming in a proper... (1 Reply)
suppose i have a file named xyz(-1) and i have to transfer(ftp) it on a Mainframe from unix,how should i do it as whenever i try to do so it says use MVS naming conventions (1 Reply)
hi
suppose i have a file named xyz(-1) and i have to transfer(ftp) it on a Mainframe from unix,how should i do it as whenever i try to do so it says use MVS naming conventions (1 Reply)
Hi
when i am trying to ftp to a newly cretaed mainframe GDG for the first time,i am getting the following error...
Verbose mode on.
200 Representation type is Ebcdic NonPrint
200 Port request OK.
550-SVC99 Return code=4 S99INFO=0 S99ERROR=38668 HEX=970C S99ERSN code X'00004379').
550... (1 Reply)