Shell Programming and Scripting

Hello All,

Firstly, the systems involved are Solaris 9/10 x86 and SPARC.

At present, we have an internally written file transfer system that we use to manage incoming transfers and distribute the files to relevant processing systems. This is based on log watching. Over the years its become evident we need to expand the reverse part of this where we drop return files to external parties as a lot of the internal developers are whacking SCP commands with keys to send files and I don't really like the fact they have access to some of these.

We are not prepared to fork out money on completely revamping our managed transfer system and I'm looking at adding a "feature set" for transferring files back to third parties based on a "hot" folder system reference a "transfer map" running under a "privileged" and "locked down" user account. What I'm thinking is a system that does this:

- Runs as a daemon
- Watches folders from a table map (text file) that's pipe delimited that contains source directory (watch directory), destination, user id, key, port, etc...
- Reports back on success/failure transfers
- Retries.

Additionally, I would really like to set this up so that the processing servers that contain the files that need to go back to third parties and assumingly where this watcher daemon runs shoots the files to a "DMZ jumpbox" first and then the DMZ system sends the files onto the relevent third parties. I just don't like internal processing servers sending files directly to third parties, especially that most of the time it's SCP (pretty much all of it) and an SSH tunnel needs to be created.

Differently to what I mentioned above, I'm also thinking of giving the developers an interface (a script) they can call to "queue" the transfer where the DMZ host gets triggered to pickup the file and spit it off to the third party instead of doing the "hot folder" system. The benefits of the hot system is that I can integrate other things beforehand like, "encrypt", "zip" with ease etc...

Does anyone have any opinions or have seen custom implementations using KSH/BASH scripting. I would prefer to do it this way as it's just a lot easier to manage and troubleshoot. I don't want to go down the Java/Perl path for managing this.

Thanks

You might want to have an RDBMS for config and logging, so you can report what sent how much how often when.

Many such systems let correspondents drop files in write-only dirs, or remove the files once they are valid and complete or after a time.

ssh gives remote login or command execution and names the whole facility, SCP moves files, tunnel allows remote connections, and sftp is just an ftp-like front end for ssh file transfer. This is not a tunnel sort of application. Tunneling is just what you want to prevent!

Yes, store and forward adds security. Also, you can online/offline archive copies for a while in case the lose them, and need a rerun (or to fulfill a court order ). Compression in a nice economy of disk space, bzip2 or the like, after a few days, and zip archiving of small files saves disk pages and inodes.

You can poll the folders easily enough using your code. You are actually reinventing the wheel, so snoop around at things like Tidal to see what features you want. You might integrate a secure web service for administration, reports, operator manual activities and even external user status reports. It could all be scripted, using command line tools like isql for db access, or PERL is nice both for DB and web, or JAVA.

Thanks for your response. It's good to hear opinions from others on matters such as this. I would really like to have something like Tidal or JScape (Managed File Transfer and Network Solutions) but it becomes quite difficult convincing the bosses to fork out and more so get the guys to move their systems over. We're fairly small scale in all this but have enough systems to be painful to migrate.

I think for now I'm going to go with polling "hot folders", move them onto DMZ host with some sort of ticket as part of our existing queuing systems and then let the DMZ host handle the transfer. If it fails transfer (from DMZ host to third party), it can create a queue file in a folder, which I can monitor out of Nagios to see if there are any pending transfers. At this point, the onus has moved off the internal processing systems, and if necessary certain departments can get spammed about "Failed to send to third party". If it fails copying from processing server to DMZ host then the files stay where they are anyway and I get spammed about "Failed to send to transfer system".

Hopefully this thread may help others who are facing similar situations in the area of manged transfer systems.

I realize this post is a few months late but I thought I'd respond just in case there are others that may have an interest in this topic. I'm the Sales Manager for Linoma Software and we have a browser based cross platform Managed File Transfer solution suite called GoAnywhere (GoAnywhereMFT) which will handle your file transfer and encryption requirements. GoAnywhere Director can be used to automate (using our own scheduler or you can call it out using your existing scheduler or script) the file transfer process (push and pull), has file and folder monitoring, can encrypt (SSH, SSL, AES, AS2, PGP), has detailed audit logging, can translate files to/from databases, notifies you upon failures and can retry failed connections automatically and continue with the file transfer where it was cut off.

Our GoAnywhere Services product is the server side of the file exchange and includes the FTP, SFTP, FTPS and HTTPS servers. Services is used when your trading partner is initiating the file exchange.

The last product of the GoAnywhere product suite is called GoAnywhere Gateway. Gateway is a reverse proxy which allows you to move Services inside the network where it's more secure and only Gateway will be sitting in the DMZ to authenticate requests to access Services.

A fully functional trial can be downloaded from our website and a license of GoAnywhere starts out at of $4,995. For more information, please visit GoAnywhereMFT.