UNIX for Beginners Questions & Answers

Hi Don Cragun,

I cannot share the file data over here due to company policies, and even there is no code written for this...
I just login into server through Putty and use tail -f server.log
to see the contents of the logs. And this file keeps only appending with new logs (ERROR, WARN and INFO lines), it does not edit the previous lines which have already written in past.

I just wanted to have this file to be copied/synced in my local windows drive so that I can view the logs realtime (but in Windows drive).

when you tail should be all you need to keep with with changes being added to the file on the remote server and should also be able to keep a local copy of the file up-to-date with text appended to that file on the remote system, can you elaborate how this can be achieved.

Thanks in advance and Sorry if I missed anything.

Basically, in modern environments one should have some kind of log shipping service/agents, and process those logs on central server.
There are free, open, proprietary etc. systems, which do the same thing, ship logs you want to central server.

As i see it you have couple of choices on Windows :

1. NFS, export the log directory via NFS from server and mount it on Windows box (windows 7 or later) - this is closest to 'real time' you will get.
2. rsync copy - not so 'real time' as NFS, but close, depending on the network bandwidth / cpu and scheduled times (every minute or 2 etc.)
3. Remote syslog on windows PC -> this will require your application to use syslog for logging with one of the local facilities and syslog configured to send logs to Windows PC.
Depending on again bandwidth, quantity of logging etc. can be a performance hit on the client you wish to pull logs off.

This is all, from my point of view, not production ready and could be / is considered hackery.
A regular solution would be to use software written for that purpose.

What do you consider as real time?
I consider that to be almost instant, in regards time needed to read from disk, send it over the wire, and write to disk in windows.
Those ops are measured in milliseconds.

Hopefully we can neglect the CPU time required for that operation, but it is software implementation dependent.
For instance, if you use rsync with compression and checksum, it is bound to take more cpu (and other) resources then an agent made especially for that purpose.
Also, running scheduled rsyncs on large log files will cause that entire file to be cached in memory, changing the memory footprint of the box.
Above can be irrelevant if there is enough memory, but can also cause log files to be cached instead of other things, making those other things visible to users as slowdowns

Hope that helps
Regards
Peasant.

Files do not work that way.

I didn't ask you to share any file data (other than the sample data you have already shown us and a clearer explanation of the format of the data you are trying to process).

I asked you to explain how you are currently accessing the remote file using applications running on Windows. From your response, I assume that you have no way to access the file. Using NFS to mount the remote file onto your windows server (acting as a cliient in this case) would allow a tail -f command on Windows to access your remote file seeing additions to that file in real-time. Anything else will introduce delays and complications.

As Corona688 said, unless you have established remote access to a file, it won't magically create a copy of itself on another server.

Wouldn't a continuously (or, rather, often enough to meet the "real-time" requirements) updated rsync-ed copy of the file suffice? Depending on the size one could also set up a cron job to do something similar with the complete file via [icode]ssh[i/code]/[icode]scp[i/code].

But all this means there at least some means to do what has to be done. But i am not allowed to touch the file, read it, transfer it or even access it but somehow i should process it is a matter of (stupid) regulations. You can't alleviate regulations via code, just by changing the regulations: "you want me to process the file, then give me access to it" is the correct way of doing things in such a case.

I hope this helps.

bakunin

Thanks @Bakunin for the insight.

I have the read only access over the file as said... I can view the file over Putty by tail -f command.
Can you help me how I can set the sync with NFS... a URL where I can learn how to setup NFS services to sync the file from server with local Windows machine.

Thanks a lot

To repeat: Files do not work that way. Even when you share the file -- which you can't without big changes in your server -- you'll be stuck running tail -f in a local terminal instead of a remote one, because text editors don't work that way either.
Files do not work that way. Files do not work that way. Because files don't work that way.

It would be worth your while to reconsider the problem from any other angle besides "magically growing file", or better yet, explain what you actually want. You might be quite surprised when you go through all the trouble of setting up an NFS only to discover that what you thought you needed hasn't actually solved your problems. tail -f in putty actually works surprisingly well.

Perhaps, is the problem the inconvenience of having to type passwords into putty? There are automatic login features you could be using. putty can do logging, also, and even network forwarding, allowing you to connect to services on the server.

Otherwise, you have information you wish to transmit from point A to point B in realtime. Perhaps netcat?

You could do full-out network or centralized logging, perhaps, and have an infrastructure to warn you of certain events.