Sponsored Content
Full Discussion: Managed file transfer
Top Forums Shell Programming and Scripting Managed file transfer Post 302518292 by dcarrion87 on Friday 29th of April 2011 07:48:36 AM
Old 04-29-2011
Managed file transfer

Hello All,

Firstly, the systems involved are Solaris 9/10 x86 and SPARC.

At present, we have an internally written file transfer system that we use to manage incoming transfers and distribute the files to relevant processing systems. This is based on log watching. Over the years its become evident we need to expand the reverse part of this where we drop return files to external parties as a lot of the internal developers are whacking SCP commands with keys to send files and I don't really like the fact they have access to some of these.

We are not prepared to fork out money on completely revamping our managed transfer system and I'm looking at adding a "feature set" for transferring files back to third parties based on a "hot" folder system reference a "transfer map" running under a "privileged" and "locked down" user account. What I'm thinking is a system that does this:

- Runs as a daemon
- Watches folders from a table map (text file) that's pipe delimited that contains source directory (watch directory), destination, user id, key, port, etc...
- Reports back on success/failure transfers
- Retries.

Additionally, I would really like to set this up so that the processing servers that contain the files that need to go back to third parties and assumingly where this watcher daemon runs shoots the files to a "DMZ jumpbox" first and then the DMZ system sends the files onto the relevent third parties. I just don't like internal processing servers sending files directly to third parties, especially that most of the time it's SCP (pretty much all of it) and an SSH tunnel needs to be created.

Differently to what I mentioned above, I'm also thinking of giving the developers an interface (a script) they can call to "queue" the transfer where the DMZ host gets triggered to pickup the file and spit it off to the third party instead of doing the "hot folder" system. The benefits of the hot system is that I can integrate other things beforehand like, "encrypt", "zip" with ease etc...

Does anyone have any opinions or have seen custom implementations using KSH/BASH scripting. I would prefer to do it this way as it's just a lot easier to manage and troubleshoot. I don't want to go down the Java/Perl path for managing this.

Thanks

Last edited by dcarrion87; 04-29-2011 at 09:03 AM.. Reason: Additional comments.
 

6 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

FSCK on veritas managed disk

I've had a VXFS filesystem get corrupted and now it won't mount. Can I run a fsck -y on the raw disk device or should something be done within veritas? Veritas does not see the disk at the moment. (2 Replies)
Discussion started by: ozzmosiz
2 Replies

2. Solaris

Smf managed service not starting

Hi Experts, While playing with smf in my local system ( which is not in production ) i am unable to restart the service svc:/network/nfs/server:default . I tried starting it in different way, however unable to restart the same. I was checking the dependency for that I disabled the... (11 Replies)
Discussion started by: kumarmani
11 Replies

3. Web Development

managed service reccomendations

Dunno If its ok to post this, but I am looking for a company to manage our linux server. If the company is a specialist in security issues then that would be ideal Thanks Ed (5 Replies)
Discussion started by: edzillion
5 Replies

4. Shell Programming and Scripting

Avoiding file overwrite during file transfer using scp

Hi, I have written a small script to transfer a file from one unix server to other using scp command which is working fine. As I know with scp, if any file with the same name is already present on destination server, it would get overwritten without any notification to user. Could anyone help me... (14 Replies)
Discussion started by: dsa
14 Replies

5. AIX

HEA configuration on managed node.

Folks, Please have a look to the attached screenshot from my managed node's HEA configuration option page. I would like to know - what does "Flow Control Enabled" checkbox help us with if opted for? Thanks! -- Souvik (3 Replies)
Discussion started by: thisissouvik
3 Replies

6. AIX

Managed system's uptime

How to find Physical server uptime from HMC/ ASMI. Server was in standby mode. We have started the Lpar manually. Server rebooted automatically but no information updated in Lpars's errpt, alog.console or HMC prior to the reboot. (1 Reply)
Discussion started by: sunnybee
1 Replies
yppush(1M)																yppush(1M)

NAME
yppush - force propagation of Network Information Service database SYNOPSIS
domain] host] maxm] mint] mapname Remarks The Network Information Service (NIS) was formerly known as Yellow Pages (YP). Although the name has changed, the functionality of the service remains the same. DESCRIPTION
copies a Network Information Service (NIS) map (database), mapname, from the map's master NIS server to each slave NIS server. It is usu- ally executed only on the master NIS server by shell script which is run either after changes are made to one or more of the master's NIS databases or when the NIS databases are first created. See ypmake(1M) and ypinit(1M) for more information on these processes. constructs a list of NIS server host names by reading the NIS map within the domain. Keys within the map are the host names of the machines on which the NIS servers run. then sends a "transfer map" request to the NIS server at each host, along with the information needed by the transfer agent (the program that actually moves the map) to call back When the transfer attempt is complete, whether successful or not, and the transfer agent sends a status message, the results can be printed to standard output. Messages are printed when a transfer is not possible, such as when the request message is undeliverable or when the timeout period on responses expires. Refer to ypfiles(4) and ypserv(1M) for an overview of Network Information Service. Options recognizes the following options: Copy mapname to the NIS servers in domain rather than to the domain returned by (see domainname(1)). Propagate the map(s) only to the specified host. Attempt to run maxm transfers in parallel to as many servers simultaneously. Without the option, attempts to transfer a map to each server, one at a time. When a network has many servers, such serial transfers can result in long delays to complete all transfers. A maxm value greater than 1 reduces total transfer time through better utilization of CPU time at the mas- ter. maxm can be any value from 1 through the number of NIS servers in the domain. Set the minimum timeout value to mint seconds. When transferring to one slave at a time, waits up to 80 seconds for the transfer to complete, after which it begins transferring to the next slave. When multiple parallel transfers are attempted by use of the option, it may be necessary to set the transfer timeout limit to a value larger than the default 80 seconds to prevent timeouts caused by network delays related to parallel transfers. Verbose mode: messages are printed when each server is called and when each response is received. If this option is omitted, only error messages are printed. WARNINGS
In the current implementation (Version 2 NIS protocol), the transfer agent is ypxfr(1M) which is started by the ypserv(1M) program at yppush's request (see ypxfr(1M) and ypserv(1M)). Starting with ONCplus version B.11.31.02, the NIS Version 1 protocol is no longer available. AUTHOR
was developed by Sun Microsystems, Inc. FILES
SEE ALSO domainname(1), ypserv(1M), ypxfr(1M), ypfiles(4). yppush(1M)
All times are GMT -4. The time now is 07:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy