Reading file from remote system and work on it locally
Hi!
I am still fairly new to shell programming, but I have taken an interest to it and want to try some new stuff.
I intend to make a shell script (using bash) to read a file on a remote system, then do some work on it on the local system and display it.
In the long run I want to have a script that does this automatically for me and I only need to initiate the script and it does updates say every 10 minutes or atoher user defined time intereval (I guess spesefied as an argument as well eventually)
But first things first. I need to be able to read the file and manipulate it. More spesific it is a comma seperated file contianing some valuses I am interested in. Not all of them, but value 2, 7 and 13 from each line I want to list on my local system.
I have so far desigend my script to take in the arguments of IPadress and filename:
As you see its so far very, very basic and not really useful as is. My next "milestone" so to speak is reading the file from a remote system. I do not want to manipulate the remote file at all, only read it and get it to the local system, there I will do the manipulation and print what I want, in a manner that disturbed the remote system the least.
But I am not entierly sure how to read the file from the remote system in a good way (after all I am still really new at shell scripting, I have only ever done some c++ programming before). Should I make a copy of it localy and use that further in my script? Or read the information I want form the remote system and only get thoose values to work with further?
I would appreciate some helps and hints here as to how to proceed. Assume I am a total newbie (and some examples if any would also help me understand)
If I have posted in the wrong section, I am sorry, but this looked to be the place were such things belong.
(Sorry if my english is not top notch, its not my native language)
Last edited by aaOzymandias; 02-22-2011 at 10:13 AM..
Reason: error in script
As always: it depends. It depends on the type of the remote system and the local system, and what services are available? Can you access the remote side using SSH/FTP/CIFS/NFS/...? What tool will you use to further process the file? Which side has more processing power available? How fast is the network between the two?
For example, with a fast network, a quick local machine, and SSH access I'd do something like this:
On a slow network, with enough processing capability on the remote side I'd let that machine do the processing and only transfer the results:
Thank you, a good question. And actually, there are two differet systems, one is far away with high latency, acrsoss the globe over satellite link, but has realtive high bandwidth. One in just a few meteres away, even higher bandwidth and very low latency. All of them have ssh (I use ssh all the time to access them), so I think using ssh is the perfered method. Processing locally might be better as well I think, despite being over saetllite link. (Realtively, the files I have interest in is not too big, usually not more than ~60 lines, but can reach up to ~1000 is some rare cases).
OK, so bandwidth is not a problem. Are there a lot of files (> ~5k)? Do they each have a header? The reason I'm asking is that for a high file count on a high latency line it's more efficient to do fewer invocations of ssh.
I'm working on Aix 6.1 and using ksh shell.
The below works fine on Linux bash or ksh shell .
while IFS= read -r dirpath ; do
echo "Hi"
done <<<"$var"
However, any such while loop that reads the input from file or variable using <<< fails on Aix system with the below error:
Below... (2 Replies)
I have 10 application servers in a distributed architecture generating their own application logs. Each server has application utility to continuously tail the log. for example following command follows tails and follows new logfiles as they are generated
server1$ logutility logtype
When I run... (8 Replies)
The code below works ok, however; I need to output the results to a local variable
the_path="/mnt/back/hang"
ssh -T -i /home/buddy/.ssh/id_rsa buddy@ginger << EOF
find ${the_path} -name "*.jpg" | wc -l > ## output to local variable
exit
EOF (3 Replies)
Hi Scott,
My previous post was not for any school or college projects. I am currently working with a IT company (Cannot provide more details than this). I am trying to implement the below script in my day-to-day work,
Apologies for the confusion in previous post :). My question remains same... (4 Replies)
Hi All,
I have below requirements for my project:
1. Building a shell script which connects to a remote server
2. running script on local machine as user 'A'
3. connecting to server using user 'B' with password
4. login with a powerbroker role 'P' (asks for same password as 'B') on that... (1 Reply)
I'm looking to use rsync to compare remote files and to copy the diff to a local directory, rather than transfer over the net.
The net connection is not fast enough to transfer these files (~1.8TB) and I'd like to sneakernet them instead.
Possible? (4 Replies)
Hi,
I want to invoke(run) a graphical application remotely, and the display should be in remote itself. (no X redirect).i want to do this through ssh.
like if i login to a remote machine and run firefox it should display there itself.
how can i do this..? (2 Replies)
Hi friends,
In my case, there are serveral PCs running Linux in a LAN.
I would like to to mount the directory /A_river of machine-A to the file system of another machine machine-B so that I can access files in that directory.
I do not know how to do this. The situation is complicated by... (2 Replies)