06-13-2017
Best way to transfer files to remote FTPS server instead of local FTPS server
Hi,
I am working on an application which runs on an Informatica Red-Hat 5.10 Linux Server.
The application involves several Informatica ETL workflows which generate 100s of Text files with lot of data. Many of the files will each be up to 5 GB in size.
Currently the Informatica server itself acts as an FTPS server and so these files are stored on the local file system by the ETL workflows and my application's downstream consumer applications use this Informatica server as their FTPS server and pull these data files from this server to their local client machines.
But going forward, due to space limitations and performance issues on Informatica server we are planning on acquiring a new Redhat Linux FTPS server and have these data files (which are gigabytes in size) generated by the ETL workflows on the local Informatica servers, pushed to the new remote FTPS server instead of storing them locally. Then I will have all downstream consumers connect to the new remote FTPS server to pull the data files they need. So then the informtica server will become a purely app server and no longer have the overhead of being an FTPS server
My question is what do you think is the best approach/strategy you would suggest for me to have the huge data files generated locally on my Informatica server shipped/transferred automatically to the new remote Linux FTPS server everytime they are generated. Since these files are huge, I know there will be latency in pushing the files from local Informatica server to remote FTPS server across network unlike before when they were being stored locally. Which Unix strategy or idea do you think would be best in minimizing these latency issues in shipping these huge files across the network to the remote FTPS server or not impact the SLAs' for file availability on the FTPS server too significantly so that downstream consumers applications do not see significant delays in file availability times. Lftp, rsync are few of the tools I have used. But are good enough for this purpose or is there some more creative approach that I could use. I know slight delays are inevitable since there is always a difference between storing files locally versus shipping them across the network to a remote FTPS server. But I am curious as to which approach will minimize this latency. Any inputs/ideas would be greatly appreciated.
thanks
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I am facing a weired problem in my FTP script. I want to transfer multiple files from remote server to local server everyday, using mget * in my script. I also, want to send an email for successful or failed FTP. My script works for file transfer, but it don't send any mail. There is... (2 Replies)
Discussion started by: berlin_germany
2 Replies
2. HP-UX
want to remove the thread
thanks (2 Replies)
Discussion started by: indira
2 Replies
3. Shell Programming and Scripting
want to remove this thread.
thanks (2 Replies)
Discussion started by: indira
2 Replies
4. UNIX for Dummies Questions & Answers
Good evening
I need your help pease
I know there are 2 commands(diff, or cp) to compare files in a directory. but the question arises:
1. can i compare zip files or ive got to unzip them?
2. how can i compare 2 files from a local to a remote server? (is there any special commad or ive got... (4 Replies)
Discussion started by: alexcol
4 Replies
5. UNIX for Dummies Questions & Answers
Hi experts,
I 'm newbie to unix world, now I have task to copy the latest files from remote server to my local. I believe this must be very common request in this community. I want you do it one more time for me please.
My requirement is something like this:
I receive files in the below... (3 Replies)
Discussion started by: parpaa
3 Replies
6. Solaris
Hi,
I am trying to automate the process of fetching files from remote server to local server through sftp. I have the username and password for the remote solaris server. But I need to give password manually everytime i run the script.
Can anyone help me in automating the script such that it... (3 Replies)
Discussion started by: ssk250
3 Replies
7. UNIX for Dummies Questions & Answers
Hi All,
I am new to SHell scripting, can someone please help me with the below requirement.
1) LFTP a file to a remote server which supports FTPS. My current enviroment is Sun Solaris 5.10
2) I need to incorporate this in a shell which is currently sending files to a server that accepts... (1 Reply)
Discussion started by: subbu
1 Replies
8. Shell Programming and Scripting
Hi All
I need to transfer a file from a UNIX server to a windows server.
I saw that it is possible to do this using scp command by looking at the forum listed below:
... (2 Replies)
Discussion started by: vx04
2 Replies
9. Shell Programming and Scripting
Hi Guyz ,,
I'm an ERP functional guy , I really need to automate a script which required a shell script but have a little knowledge in shell scripting.
I need my generated files to be zipped first in one directory lets say (xyz) and then it needs to transfer another ftp server in... (3 Replies)
Discussion started by: Shogundion
3 Replies
10. Shell Programming and Scripting
Hi Team,
I am currently working on an integration project where in we have planned to receive files from an external source onto our system via FTPS(SSL). I am new to this part and would like to know the points for consideration and the points to learn to get this done. The files we receive from... (1 Reply)
Discussion started by: Rads
1 Replies
LEARN ABOUT MOJAVE
curlopt_sslengine_default
CURLOPT_SSLENGINE_DEFAULT(3) curl_easy_setopt options CURLOPT_SSLENGINE_DEFAULT(3)
NAME
CURLOPT_SSLENGINE_DEFAULT - make SSL engine default
SYNOPSIS
#include <curl/curl.h>
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_SSLENGINE_DEFAULT, long val);
DESCRIPTION
Pass a long set to 1 to make the already specified crypto engine the default for (asymmetric) crypto operations.
This option has no effect unless set after CURLOPT_SSLENGINE(3).
DEFAULT
None
PROTOCOLS
All TLS based protocols: HTTPS, FTPS, IMAPS, POP3S, SMTPS etc.
EXAMPLE
TODO
AVAILABILITY
Only if the SSL backend is OpenSSL built with engine support.
RETURN VALUE
CURLE_OK - Engine set as default.
CURLE_SSL_ENGINE_SETFAILED - Engine could not be set as default.
CURLE_NOT_BUILT_IN - Option not built in, OpenSSL is not the SSL backend.
CURLE_UNKNOWN_OPTION - Option not recognized.
CURLE_OUT_OF_MEMORY - Insufficient heap space.
SEE ALSO
CURLOPT_SSLENGINE(3), CURLOPT_SSLCERT(3),
libcurl 7.54.0 April 18, 2016 CURLOPT_SSLENGINE_DEFAULT(3)