06-13-2017
Best way to transfer files to remote FTPS server instead of local FTPS server
Hi,
I am working on an application which runs on an Informatica Red-Hat 5.10 Linux Server.
The application involves several Informatica ETL workflows which generate 100s of Text files with lot of data. Many of the files will each be up to 5 GB in size.
Currently the Informatica server itself acts as an FTPS server and so these files are stored on the local file system by the ETL workflows and my application's downstream consumer applications use this Informatica server as their FTPS server and pull these data files from this server to their local client machines.
But going forward, due to space limitations and performance issues on Informatica server we are planning on acquiring a new Redhat Linux FTPS server and have these data files (which are gigabytes in size) generated by the ETL workflows on the local Informatica servers, pushed to the new remote FTPS server instead of storing them locally. Then I will have all downstream consumers connect to the new remote FTPS server to pull the data files they need. So then the informtica server will become a purely app server and no longer have the overhead of being an FTPS server
My question is what do you think is the best approach/strategy you would suggest for me to have the huge data files generated locally on my Informatica server shipped/transferred automatically to the new remote Linux FTPS server everytime they are generated. Since these files are huge, I know there will be latency in pushing the files from local Informatica server to remote FTPS server across network unlike before when they were being stored locally. Which Unix strategy or idea do you think would be best in minimizing these latency issues in shipping these huge files across the network to the remote FTPS server or not impact the SLAs' for file availability on the FTPS server too significantly so that downstream consumers applications do not see significant delays in file availability times. Lftp, rsync are few of the tools I have used. But are good enough for this purpose or is there some more creative approach that I could use. I know slight delays are inevitable since there is always a difference between storing files locally versus shipping them across the network to a remote FTPS server. But I am curious as to which approach will minimize this latency. Any inputs/ideas would be greatly appreciated.
thanks
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I am facing a weired problem in my FTP script. I want to transfer multiple files from remote server to local server everyday, using mget * in my script. I also, want to send an email for successful or failed FTP. My script works for file transfer, but it don't send any mail. There is... (2 Replies)
Discussion started by: berlin_germany
2 Replies
2. HP-UX
want to remove the thread
thanks (2 Replies)
Discussion started by: indira
2 Replies
3. Shell Programming and Scripting
want to remove this thread.
thanks (2 Replies)
Discussion started by: indira
2 Replies
4. UNIX for Dummies Questions & Answers
Good evening
I need your help pease
I know there are 2 commands(diff, or cp) to compare files in a directory. but the question arises:
1. can i compare zip files or ive got to unzip them?
2. how can i compare 2 files from a local to a remote server? (is there any special commad or ive got... (4 Replies)
Discussion started by: alexcol
4 Replies
5. UNIX for Dummies Questions & Answers
Hi experts,
I 'm newbie to unix world, now I have task to copy the latest files from remote server to my local. I believe this must be very common request in this community. I want you do it one more time for me please.
My requirement is something like this:
I receive files in the below... (3 Replies)
Discussion started by: parpaa
3 Replies
6. Solaris
Hi,
I am trying to automate the process of fetching files from remote server to local server through sftp. I have the username and password for the remote solaris server. But I need to give password manually everytime i run the script.
Can anyone help me in automating the script such that it... (3 Replies)
Discussion started by: ssk250
3 Replies
7. UNIX for Dummies Questions & Answers
Hi All,
I am new to SHell scripting, can someone please help me with the below requirement.
1) LFTP a file to a remote server which supports FTPS. My current enviroment is Sun Solaris 5.10
2) I need to incorporate this in a shell which is currently sending files to a server that accepts... (1 Reply)
Discussion started by: subbu
1 Replies
8. Shell Programming and Scripting
Hi All
I need to transfer a file from a UNIX server to a windows server.
I saw that it is possible to do this using scp command by looking at the forum listed below:
... (2 Replies)
Discussion started by: vx04
2 Replies
9. Shell Programming and Scripting
Hi Guyz ,,
I'm an ERP functional guy , I really need to automate a script which required a shell script but have a little knowledge in shell scripting.
I need my generated files to be zipped first in one directory lets say (xyz) and then it needs to transfer another ftp server in... (3 Replies)
Discussion started by: Shogundion
3 Replies
10. Shell Programming and Scripting
Hi Team,
I am currently working on an integration project where in we have planned to receive files from an external source onto our system via FTPS(SSL). I am new to this part and would like to know the points for consideration and the points to learn to get this done. The files we receive from... (1 Reply)
Discussion started by: Rads
1 Replies
LEARN ABOUT DEBIAN
bup-on
bup-on(1) General Commands Manual bup-on(1)
NAME
bup-on - run a bup server locally and client remotely
SYNOPSIS
bup on <hostname> index ...
bup on <hostname> save ...
bup on <hostname> split ...
DESCRIPTION
bup on runs the given bup command on the given host using ssh. It runs a bup server on the local machine, so that commands like bup save
on the remote machine can back up to the local machine. (You don't need to provide a --remote option to bup save in order for this to
work.)
See bup-index(1), bup-save(1), and so on for details of how each subcommand works.
This 'reverse mode' operation is useful when the machine being backed up isn't supposed to be able to ssh into the backup server. For
example, your backup server can be hidden behind a one-way firewall on a private or dynamic IP address; using an ssh key, it can be autho-
rized to ssh into each of your important machines. After connecting to each destination machine, it initiates a backup, receiving the
resulting data and storing in its local repository.
For example, if you run several virtual private Linux machines on a remote hosting provider, you could back them up to a local (much less
expensive) computer in your basement.
EXAMPLES
# First index the files on the remote server
$ bup on myserver index -vux /etc
bup server: reading from stdin.
Indexing: 2465, done.
bup: merging indexes (186668/186668), done.
bup server: done
# Now save the files from the remote server to the
# local $BUP_DIR
$ bup on myserver save -n myserver-backup /etc
bup server: reading from stdin.
bup server: command: 'list-indexes'
PackIdxList: using 7 indexes.
Saving: 100.00% (241/241k, 648/648 files), done.
bup server: received 55 objects.
Indexing objects: 100% (55/55), done.
bup server: command: 'quit'
bup server: done
# Now we can look at the resulting repo on the local
# machine
$ bup ftp 'cat /myserver-backup/latest/etc/passwd'
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
...
SEE ALSO
bup-index(1), bup-save(1), bup-split(1)
BUP
Part of the bup(1) suite.
AUTHORS
Avery Pennarun <apenwarr@gmail.com>.
Bup unknown- bup-on(1)