FTP from AIX to Mainframe


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting FTP from AIX to Mainframe
# 8  
Old 12-21-2017
Quote:
Originally Posted by TechGyaann
Now how do I verify the transferred file to Mainframe
If you're using FTP, some FTP clients support ls -l, but it sounds like you need a lot more than that.

Quote:
Below are the scenarios:
1. FTP the currently available files(poll a directory to find the list) and put it to mainframe in one connection instance
2. Verify if all the files are copied to Mainframe successfully
3. If the FTP command fails due to some issue, try to resend the files and if the issue persist after 5 attempts to resend, inform particular user by email.
4.If the retry succeeds(send only those files that was not sent., this is in case of partially succeeded transfer; say 5 files , 2 transferred and command failed due to some issue and other 3 not sent... in that case send only those 3 files).
5. Now again verification of file(s) transferred to mainframe
If sftp's an option, have you considered rsync? You can consider it to be "cp on steroids". It's an extremely common UNIX utility which connects with ssh (same protocol as sftp) and does a lot of what you're asking for, especially the verification of files.

i.e. you can do rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder and it will copy and verify. It will even skip already-existing files based on checksum.

It would make your task very easy.

Code:
exec 2>errorlog
N=0
while ! rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder
do
        N=$((N+1))
        if [ "$N" -ge 5 ]
        then
                echo "rsync couldn't transfer files, see errlog" >&2
                exit 1
        fi
done

echo "rsync successfully transferred files"

# 9  
Old 12-21-2017
Quote:
Originally Posted by Corona688
If you're using FTP, some FTP clients support ls -l, but it sounds like you need a lot more than that.

If sftp's an option, have you considered rsync? You can consider it to be "cp on steroids". It's an extremely common UNIX utility which connects with ssh (same protocol as sftp) and does a lot of what you're asking for, especially the verification of files.

i.e. you can do rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder and it will copy and verify. It will even skip already-existing files based on checksum.

It would make your task very easy.

Code:
exec 2>errorlog
N=0
while ! rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder
do
        N=$((N+1))
        if [ "$N" -ge 5 ]
        then
                echo "rsync couldn't transfer files, see errlog" >&2
                exit 1
        fi
done

echo "rsync successfully transferred files"

Thanks Much Corona688., appreciate your help.
I have to implement this functionality using FTP.

Ok., since you have given me the above code I would like to ask few questions.

A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
If so can you please throw some light?

B. Is using SFTP less secure when compared to FTP -s to transfer files from AIX to Mainframe.? Will there be any loss in data .i.e., will the data get corrupted?
And per my understanding if we go with SFTP method, we can used rsync to copy as well as verify the file.
Even in case of failure, restartability is assured(i.e, only files that are not copied will be copied to the mainframe).



Update:- I will not be able to use SFTP Smilie

Last edited by TechGyaann; 12-21-2017 at 05:28 PM..
# 10  
Old 12-21-2017
Quote:
Originally Posted by TechGyaann
I have to implement this functionality using FTP.
The nice thing about FTP - and also the problem with FTP - is that it has zero features.
  • Can you ask FTP if the file uploaded successfully? Nope.
  • Can you ask FTP for a file checksum or anything like that to verify it? Nope.
  • Can it tell you the most recent file? Nope.
  • Can you ask it if a file is there at all? Nope.
  • Can it tell you anything? Nope. Well, a directory listing. That's it.

So, you can depend on FTP being reliably unhelpful. Smilie But there's old-fashioned ways to trick it into doing what you want:
Code:
ftp <<EOF
cd /remote/incomplete
put localfilename
rename /remote/incomplete/uploadedfile /remote/complete/uploadedfile

This prevents partial uploads from appearing in /remote/complete. If the connection is broken early, the file will never get renamed.

Then you can get the directory listing from /remote/complete to see if the file you wanted is in it.

Quote:
A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
It will require brute force.
  1. Connect with FTP, saving all output to a file.
  2. Get directory listing.
  3. Quit.
  4. Process directory listing in script to remove all lines except the directory listing.
  5. Check if the directory listing contains the file you uploaded.

That sort of thing.

Quote:
B. Is using SFTP less secure when compared to FTP -s to transfer files from AIX to Mainframe.?
Do you even have ftp -s? That's a very rare feature.
Quote:
Will there be any loss in data .i.e., will the data get corrupted?
sftp is less likely to corrupt your data than FTP.

FTP's "binary" feature is only there because, in the beginning, ftp was expected to translate text files automatically (i.e. EBCDIC to ASCII) so the recipient could read them. FTP mangles files because it's trying to process all downloads as text! And it always does that unless you force it not to with binary! sftp just doesn't do that, at all, period. It transfers the file as-is, always.
# 11  
Old 12-21-2017
Wow.. that's lot of learning from just few interaction..

Quote:

Code:
ftp <<EOF
cd /remote/incomplete
put localfilename
rename /remote/incomplete/uploadedfile /remote/complete/uploadedfile

This prevents partial uploads from appearing in /remote/complete. If the connection is broken early, the file will never get renamed.

Then you can get the directory listing from /remote/complete to see if the file you wanted is in it.

Quote:
A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
It will require brute force.
Connect with FTP, saving all output to a file.
Get directory listing.
Quit.
Process directory listing in script to remove all lines except the directory listing.
Check if the directory listing contains the file you uploaded.

That sort of thing.
This is what I understood.

Step 1. Open a Control channel & data channel.. put the file
Step 2. Create a nlist to get the info of files transferred (if any other way please let me know)
Step 3. Quit
Step 4. Compare the nlist output and the list of files that was supposed to be transferred. If not transfer only rest of the files.


Please advise if any discrepancy in above steps using FTP method



Using SFTP method:

U said its interactive, may I please request you to give me examples or example scripts or few links for example/ to learn.

Also tell me the steps here

More over for SFTP should we use only port 21 or the same control port used for FTP can be used to connect to Mainframe. (Asking this question because I am not able to connect to mainframe using sftp)

Code:
sftp -oPort=12345 '@MUSR'@mainframe.server

not connecting or not prompting for password.
# 12  
Old 12-21-2017
Quote:
Originally Posted by TechGyaann
Step 2. Create a nlist to get the info of files transferred (if any other way please let me know)
If your FTP server and client support nlist, that's the best way to get the file list. Not all of them do.
Quote:
Please advise if any discrepancy in above steps using FTP method
You've got the idea, I think.

Quote:
Using SFTP method:

U said its interactive, may I please request you to give me examples or example scripts or few links for example/ to learn.
Nobody said that, and I'm not sure what you mean by it.

Quote:
More over for SFTP should we use only port 21 or the same control port used for FTP can be used to connect to Mainframe. (Asking this question because I am not able to connect to mainframe using sftp)

Code:
sftp -oPort=12345 '@MUSR'@mainframe.server

not connecting or not prompting for password.
sftp is not ftp. Period. Full stop. It is not related to FTP at all, and does not inter-operate with it. Trying to connect to your FTP server with sftp is as nonsensical as connecting to your mail server with it. It's just not going to work. There is no control channel, either. You just connect to your destination directly.

sftp is actually a use of ssh protocol -- secure shell, i.e. port 22.
This User Gave Thanks to Corona688 For This Post:
# 13  
Old 12-21-2017
Thanks Corona688...

Got your point.

The point is we dont have SSH option to Mainframe. so SFTP, SCP and SSH options ruled out. Smilie

So we are left with FTP and NDM.

Last edited by TechGyaann; 12-21-2017 at 06:35 PM.. Reason: Adding more points
# 14  
Old 12-21-2017
No idea. At first glance that's an AIX thing. Maybe someone else knows, also try asking in the AIX subforum.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

AIX Forum: FTP from AIX to Mainframe

This discussion thread is an extension to what was discussed in Shell scripting section. Please refer the post for the requirement: Requirement Post - Click Here The whole thread - Click Here I would like to know how I can use NDM to transfer file from AIX to Mainframe and to verify the... (3 Replies)
Discussion started by: TechGyaann
3 Replies

2. AIX

ODBC Connectivity between Oracle10g on AIX and Mainframe

Hi, I have a task of setting up connectivity between Oracle 10g (AIX) and Mainframe (1 library). Went through couple of documents, forums, blogs etc. MY understanding is ODBC Generic Connectivity is free from Oracle side. Question: (may be dumb to you) 1. Has anybody done this and would... (3 Replies)
Discussion started by: jvmani_1
3 Replies

3. UNIX for Advanced & Expert Users

FTP issues between mainframe and UNIX

Hi All, The issue is that, we have a dataset in mianframe whose record length is 153 characters. And a batch job ftpies it to the unix server(SunOS) as a test file. But the ftpied file in unix does not have a record length of 153 chars. Each record of 153 chars gets splited into two line of... (8 Replies)
Discussion started by: hareeshkumaru
8 Replies

4. AIX

Problems with SSH/SFTP between AIX and Mainframe

Hi, I'm not sure if this has been solved in this forum already but please do help me out if possible. Basically, I've already setup a passwordless SSH connection between 2 AIX IDs (say ID-1 and ID-2) with a Mainframe server ID (say MVSID). I'm able to successfully do an SSH from the AIX server to... (1 Reply)
Discussion started by: sambeet
1 Replies

5. Shell Programming and Scripting

FTP files to target Mainframe system

Hi Experts... Greetings for the day..! I just want to FTP the files to mainframe system.. my code is not working..and also i need to put the files in a particular directory in a specific naming format... ftp -i -n ${HOST_NAME} << END_FTP user ${USER_NAME} ${PASSWORD} put ${FILE_NAME}... (3 Replies)
Discussion started by: spkandy
3 Replies

6. UNIX for Dummies Questions & Answers

FTP'ing EBCDIC Characters from Mainframe

Hi All, I am new to this site, I have a requirement where in i have to FTP a file from mainframe to Unix box. The catch here is there are few Spanish characters like N with tilde(~) and a with ` etc., all other characters are coming fine but those mentioned above are not coming in a proper... (1 Reply)
Discussion started by: harikiranr
1 Replies

7. UNIX for Advanced & Expert Users

ftp file from unix to mainframe

thanks (2 Replies)
Discussion started by: ashishabhishek
2 Replies

8. UNIX and Linux Applications

ftp from unix to Mainframe

suppose i have a file named xyz(-1) and i have to transfer(ftp) it on a Mainframe from unix,how should i do it as whenever i try to do so it says use MVS naming conventions (1 Reply)
Discussion started by: ashishabhishek
1 Replies

9. HP-UX

ftp from unix to mainframe

hi suppose i have a file named xyz(-1) and i have to transfer(ftp) it on a Mainframe from unix,how should i do it as whenever i try to do so it says use MVS naming conventions (1 Reply)
Discussion started by: ashishabhishek
1 Replies

10. Shell Programming and Scripting

mainframe FTP error

Hi when i am trying to ftp to a newly cretaed mainframe GDG for the first time,i am getting the following error... Verbose mode on. 200 Representation type is Ebcdic NonPrint 200 Port request OK. 550-SVC99 Return code=4 S99INFO=0 S99ERROR=38668 HEX=970C S99ERSN code X'00004379'). 550... (1 Reply)
Discussion started by: sam99
1 Replies
Login or Register to Ask a Question