Managed file transfer


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Managed file transfer
# 1  
Old 04-29-2011
Managed file transfer

Hello All,

Firstly, the systems involved are Solaris 9/10 x86 and SPARC.

At present, we have an internally written file transfer system that we use to manage incoming transfers and distribute the files to relevant processing systems. This is based on log watching. Over the years its become evident we need to expand the reverse part of this where we drop return files to external parties as a lot of the internal developers are whacking SCP commands with keys to send files and I don't really like the fact they have access to some of these.

We are not prepared to fork out money on completely revamping our managed transfer system and I'm looking at adding a "feature set" for transferring files back to third parties based on a "hot" folder system reference a "transfer map" running under a "privileged" and "locked down" user account. What I'm thinking is a system that does this:

- Runs as a daemon
- Watches folders from a table map (text file) that's pipe delimited that contains source directory (watch directory), destination, user id, key, port, etc...
- Reports back on success/failure transfers
- Retries.

Additionally, I would really like to set this up so that the processing servers that contain the files that need to go back to third parties and assumingly where this watcher daemon runs shoots the files to a "DMZ jumpbox" first and then the DMZ system sends the files onto the relevent third parties. I just don't like internal processing servers sending files directly to third parties, especially that most of the time it's SCP (pretty much all of it) and an SSH tunnel needs to be created.

Differently to what I mentioned above, I'm also thinking of giving the developers an interface (a script) they can call to "queue" the transfer where the DMZ host gets triggered to pickup the file and spit it off to the third party instead of doing the "hot folder" system. The benefits of the hot system is that I can integrate other things beforehand like, "encrypt", "zip" with ease etc...

Does anyone have any opinions or have seen custom implementations using KSH/BASH scripting. I would prefer to do it this way as it's just a lot easier to manage and troubleshoot. I don't want to go down the Java/Perl path for managing this.

Thanks

Last edited by dcarrion87; 04-29-2011 at 09:03 AM.. Reason: Additional comments.
# 2  
Old 04-29-2011
You might want to have an RDBMS for config and logging, so you can report what sent how much how often when.

Many such systems let correspondents drop files in write-only dirs, or remove the files once they are valid and complete or after a time.

ssh gives remote login or command execution and names the whole facility, SCP moves files, tunnel allows remote connections, and sftp is just an ftp-like front end for ssh file transfer. This is not a tunnel sort of application. Tunneling is just what you want to prevent!

Yes, store and forward adds security. Also, you can online/offline archive copies for a while in case the lose them, and need a rerun (or to fulfill a court order Smilie). Compression in a nice economy of disk space, bzip2 or the like, after a few days, and zip archiving of small files saves disk pages and inodes.

You can poll the folders easily enough using your code. You are actually reinventing the wheel, so snoop around at things like Tidal to see what features you want. You might integrate a secure web service for administration, reports, operator manual activities and even external user status reports. It could all be scripted, using command line tools like isql for db access, or PERL is nice both for DB and web, or JAVA.
# 3  
Old 04-30-2011
Thanks for your response. It's good to hear opinions from others on matters such as this. I would really like to have something like Tidal or JScape (Managed File Transfer and Network Solutions) but it becomes quite difficult convincing the bosses to fork out and more so get the guys to move their systems over. We're fairly small scale in all this but have enough systems to be painful to migrate.

I think for now I'm going to go with polling "hot folders", move them onto DMZ host with some sort of ticket as part of our existing queuing systems and then let the DMZ host handle the transfer. If it fails transfer (from DMZ host to third party), it can create a queue file in a folder, which I can monitor out of Nagios to see if there are any pending transfers. At this point, the onus has moved off the internal processing systems, and if necessary certain departments can get spammed about "Failed to send to third party". If it fails copying from processing server to DMZ host then the files stay where they are anyway and I get spammed about "Failed to send to transfer system".

Hopefully this thread may help others who are facing similar situations in the area of manged transfer systems.
# 4  
Old 07-06-2011
GoAnywhere from Linoma Software can automate & encrypt your file transfers

I realize this post is a few months late but I thought I'd respond just in case there are others that may have an interest in this topic. I'm the Sales Manager for Linoma Software and we have a browser based cross platform Managed File Transfer solution suite called GoAnywhere (GoAnywhereMFT) which will handle your file transfer and encryption requirements. GoAnywhere Director can be used to automate (using our own scheduler or you can call it out using your existing scheduler or script) the file transfer process (push and pull), has file and folder monitoring, can encrypt (SSH, SSL, AES, AS2, PGP), has detailed audit logging, can translate files to/from databases, notifies you upon failures and can retry failed connections automatically and continue with the file transfer where it was cut off.

Our GoAnywhere Services product is the server side of the file exchange and includes the FTP, SFTP, FTPS and HTTPS servers. Services is used when your trading partner is initiating the file exchange.

The last product of the GoAnywhere product suite is called GoAnywhere Gateway. Gateway is a reverse proxy which allows you to move Services inside the network where it's more secure and only Gateway will be sitting in the DMZ to authenticate requests to access Services.

A fully functional trial can be downloaded from our website and a license of GoAnywhere starts out at of $4,995. For more information, please visit GoAnywhereMFT.
Login or Register to Ask a Question

Previous Thread | Next Thread

6 More Discussions You Might Find Interesting

1. AIX

Managed system's uptime

How to find Physical server uptime from HMC/ ASMI. Server was in standby mode. We have started the Lpar manually. Server rebooted automatically but no information updated in Lpars's errpt, alog.console or HMC prior to the reboot. (1 Reply)
Discussion started by: sunnybee
1 Replies

2. AIX

HEA configuration on managed node.

Folks, Please have a look to the attached screenshot from my managed node's HEA configuration option page. I would like to know - what does "Flow Control Enabled" checkbox help us with if opted for? Thanks! -- Souvik (3 Replies)
Discussion started by: thisissouvik
3 Replies

3. Shell Programming and Scripting

Avoiding file overwrite during file transfer using scp

Hi, I have written a small script to transfer a file from one unix server to other using scp command which is working fine. As I know with scp, if any file with the same name is already present on destination server, it would get overwritten without any notification to user. Could anyone help me... (14 Replies)
Discussion started by: dsa
14 Replies

4. Web Development

managed service reccomendations

Dunno If its ok to post this, but I am looking for a company to manage our linux server. If the company is a specialist in security issues then that would be ideal Thanks Ed (5 Replies)
Discussion started by: edzillion
5 Replies

5. Solaris

Smf managed service not starting

Hi Experts, While playing with smf in my local system ( which is not in production ) i am unable to restart the service svc:/network/nfs/server:default . I tried starting it in different way, however unable to restart the same. I was checking the dependency for that I disabled the... (11 Replies)
Discussion started by: kumarmani
11 Replies

6. Filesystems, Disks and Memory

FSCK on veritas managed disk

I've had a VXFS filesystem get corrupted and now it won't mount. Can I run a fsck -y on the raw disk device or should something be done within veritas? Veritas does not see the disk at the moment. (2 Replies)
Discussion started by: ozzmosiz
2 Replies
Login or Register to Ask a Question