Realtime data handling between 2 or more systems


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Realtime data handling between 2 or more systems
# 1  
Old 07-06-2008
Realtime data handling between 2 or more systems

Hi

I am in need to find a solution the following problem

I have more 3 unix based servers. Each handling many different services.
Then I have 1 windows based application that needs to collect the .log files in /tmp on the servers.

The plan I initially thought of will be to open ftp connection from windows to the servers on a fixed time interval and collect .log this way.

But now I need a more up-to-date data, so I need to find a solution whereby the servers will PUSH the data to the windows system, upon modification on the log files.

Please kindly recommend any middle-application or ideas on how to make this possible?
# 2  
Old 07-07-2008
Java

You've got quite a few options to select from here, the main thing to think about is your security requirements and data integrity (ie, is it ok to have a partial file etc?).

Once you know what you need from that, it should be easy to pick what you want to use.

As for mechanisms to transfer your files, here's a few ideas that might work for you:
  • samba: Run smbd on your three unix servers then map drives on the windows server to these shares. This will mean you don't actually need to store the files over on the windows server, you can read them over the network. Hard to get more up-to-date than that Smilie
  • ftp push: Run an ftpd on the windows server and have the unix servers push files over via ftp. Use send/expect to interact with ftp or use curl to do the transfer on the commandline.
  • ssh/scp: Run an sshd (eg cygwin) on your windows server so that you can automate file pushes from the unix servers. This will give you encrypted transfers and allow you to use ssh keys - that way you don't need to store a password on each of the servers.
  • email: Run an smtpd on the windows server and have it trigger your log application (or just a copy) on incomming emails for a certain address. Then on each unix server, uuencode/mpack/embed the logs to your email and fire them off.
  • Third party file transfer tools like connect direct or xcom: Most of these give you protection from trying to use partial files etc and have clients and servers for many different platforms. These generally cost money to use though :/

If you need encryption, then it's ssh, sftp, email combined with pgp, or the thrid party tools for you.
If you need passwordless connection, you can script pretty much anything but will have to store the password in the script (generally a bad thing) unless you go with samba, ssh, email, or the right thrid party tool.
If it's important that your windows app doesn't try to use a file that's only half copied, the only (fairly) sure solution is to script the use of a temp file that gets renamed after it's fully sent, use email, or a thirld party tool.

Hope that helps, sing out if you want further elaboration on a particular solution.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

4 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Handling embedded double quotes within column data

I have a text file where each field is enclosed in double quotes and separated by a comma. But in certain rows we have embedded double quotes within column data For e.g """TRUSPICE CENTRE"" CLAYTON AU" The above value is for a single column but there are embedded quotes within... (2 Replies)
Discussion started by: abhilashnair
2 Replies

2. UNIX for Dummies Questions & Answers

Large file data handling issue

I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long" I need to actually remove the last | symbol from this file. sed -e 's/\|*$//' filename is working fine for small files. But not working on this big... (13 Replies)
Discussion started by: Gurkamal83
13 Replies

3. What is on Your Mind?

From Systems Admin to Systems Eng.

I have been wondering how do Systems Administrators do the jump into Systems Engineering? Is it only a matter of time and experience or could I actually help myself get there? Opinions? Books I could read? Thanks a lot for your help! (0 Replies)
Discussion started by: svalenciatech
0 Replies

4. Shell Programming and Scripting

Data handling using AWK

Hi, I have requirement to fetch lines with a particular character in one column e.g. 2.5M asdsad 3.5M sadsadas 12323M ssdss i tried following so far #echo 2.3M asdsad | nawk -F " " '{print substr($1,length($1))}' M So far i have tried following # echo 2.3M asdsad | nawk... (4 Replies)
Discussion started by: mtomar
4 Replies
Login or Register to Ask a Question