Managed file transfer Post: 302518292

Sponsored Content

Top Forums Shell Programming and Scripting Managed file transfer Post 302518292 by dcarrion87 on Friday 29th of April 2011 07:48:36 AM

04-29-2011

Registered User

Managed file transfer

Hello All,

Firstly, the systems involved are Solaris 9/10 x86 and SPARC.

At present, we have an internally written file transfer system that we use to manage incoming transfers and distribute the files to relevant processing systems. This is based on log watching. Over the years its become evident we need to expand the reverse part of this where we drop return files to external parties as a lot of the internal developers are whacking SCP commands with keys to send files and I don't really like the fact they have access to some of these.

We are not prepared to fork out money on completely revamping our managed transfer system and I'm looking at adding a "feature set" for transferring files back to third parties based on a "hot" folder system reference a "transfer map" running under a "privileged" and "locked down" user account. What I'm thinking is a system that does this:

- Runs as a daemon
- Watches folders from a table map (text file) that's pipe delimited that contains source directory (watch directory), destination, user id, key, port, etc...
- Reports back on success/failure transfers
- Retries.

Additionally, I would really like to set this up so that the processing servers that contain the files that need to go back to third parties and assumingly where this watcher daemon runs shoots the files to a "DMZ jumpbox" first and then the DMZ system sends the files onto the relevent third parties. I just don't like internal processing servers sending files directly to third parties, especially that most of the time it's SCP (pretty much all of it) and an SSH tunnel needs to be created.

Differently to what I mentioned above, I'm also thinking of giving the developers an interface (a script) they can call to "queue" the transfer where the DMZ host gets triggered to pickup the file and spit it off to the third party instead of doing the "hot folder" system. The benefits of the hot system is that I can integrate other things beforehand like, "encrypt", "zip" with ease etc...

Does anyone have any opinions or have seen custom implementations using KSH/BASH scripting. I would prefer to do it this way as it's just a lot easier to manage and troubleshoot. I don't want to go down the Java/Perl path for managing this.

Thanks

Last edited by dcarrion87; 04-29-2011 at 09:03 AM.. Reason: Additional comments.

dcarrion87

View Public Profile for dcarrion87

Find all posts by dcarrion87

6 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

FSCK on veritas managed disk

I've had a VXFS filesystem get corrupted and now it won't mount. Can I run a fsck -y on the raw disk device or should something be done within veritas? Veritas does not see the disk at the moment.

2. Solaris

Smf managed service not starting

Hi Experts, While playing with smf in my local system ( which is not in production ) i am unable to restart the service svc:/network/nfs/server:default . I tried starting it in different way, however unable to restart the same. I was checking the dependency for that I disabled the...

3. Web Development

managed service reccomendations

Dunno If its ok to post this, but I am looking for a company to manage our linux server. If the company is a specialist in security issues then that would be ideal Thanks Ed

4. Shell Programming and Scripting

Avoiding file overwrite during file transfer using scp

Hi, I have written a small script to transfer a file from one unix server to other using scp command which is working fine. As I know with scp, if any file with the same name is already present on destination server, it would get overwritten without any notification to user. Could anyone help me...

5. AIX

HEA configuration on managed node.

Folks, Please have a look to the attached screenshot from my managed node's HEA configuration option page. I would like to know - what does "Flow Control Enabled" checkbox help us with if opted for? Thanks! -- Souvik

6. AIX

Managed system's uptime

How to find Physical server uptime from HMC/ ASMI. Server was in standby mode. We have started the Lpar manually. Server rebooted automatically but no information updated in Lpars's errpt, alog.console or HMC prior to the reboot.

LEARN ABOUT ULTRIX

libcurl-easy

libcurl(3)						      libcurl easy interface							libcurl(3)

NAME

       libcurl-easy - easy interface overview

DESCRIPTION

       When  using  libcurl's  "easy"  interface  you init your session and get a handle (often referred to as an "easy handle"), which you use as
       input to the easy interface functions you use. Use curl_easy_init(3) to get the handle.

       You continue by setting all the options you want in the upcoming transfer, the most important among them  is  the  URL  itself  (you  can't
       transfer  anything without a specified URL as you may have figured out yourself). You might want to set some callbacks as well that will be
       called from the library when data is available etc. curl_easy_setopt(3) is used for all this.

       When all is setup, you tell libcurl to perform the transfer using curl_easy_perform(3).	It will then do the  entire  operation	and  won't
       return until it is done (successfully or not).

       After  the  transfer  has  been	made, you can set new options and make another transfer, or if you're done, cleanup the session by calling
       curl_easy_cleanup(3). If you want persistent connections, you don't cleanup immediately, but instead run ahead and perform other  transfers
       using the same easy handle.

libcurl 7.10.7							    12 Aug 2003 							libcurl(3)