Reading file from remote system and work on it locally


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reading file from remote system and work on it locally
# 1  
Old 02-22-2011
Reading file from remote system and work on it locally

Hi!

I am still fairly new to shell programming, but I have taken an interest to it and want to try some new stuff.

I intend to make a shell script (using bash) to read a file on a remote system, then do some work on it on the local system and display it.

In the long run I want to have a script that does this automatically for me and I only need to initiate the script and it does updates say every 10 minutes or atoher user defined time intereval (I guess spesefied as an argument as well eventually)

But first things first. I need to be able to read the file and manipulate it. More spesific it is a comma seperated file contianing some valuses I am interested in. Not all of them, but value 2, 7 and 13 from each line I want to list on my local system.

I have so far desigend my script to take in the arguments of IPadress and filename:

Code:
#!/bin/sh

# Look for args, if worng number of agruments print error and exit
# ARGS is number of arguments scripts expects
ARGS=2

if [ $# -ne $ARGS ]
then
    echo "$0:Error, missing or too many arguments!"
    echo "Usage: $0 IP filename"
    exit 1
fi

IPADRESS=$1
FILENAME=$2
echo $IPADRESS
echo $FILENAME

As you see its so far very, very basic and not really useful as is. My next "milestone" so to speak is reading the file from a remote system. I do not want to manipulate the remote file at all, only read it and get it to the local system, there I will do the manipulation and print what I want, in a manner that disturbed the remote system the least.

But I am not entierly sure how to read the file from the remote system in a good way (after all I am still really new at shell scripting, I have only ever done some c++ programming before). Should I make a copy of it localy and use that further in my script? Or read the information I want form the remote system and only get thoose values to work with further?

I would appreciate some helps and hints here as to how to proceed. Assume I am a total newbie (and some examples if any would also help me understand) Smilie

If I have posted in the wrong section, I am sorry, but this looked to be the place were such things belong.

(Sorry if my english is not top notch, its not my native language)

Last edited by aaOzymandias; 02-22-2011 at 11:13 AM.. Reason: error in script
# 2  
Old 02-22-2011
As always: it depends. It depends on the type of the remote system and the local system, and what services are available? Can you access the remote side using SSH/FTP/CIFS/NFS/...? What tool will you use to further process the file? Which side has more processing power available? How fast is the network between the two?

For example, with a fast network, a quick local machine, and SSH access I'd do something like this:
Code:
ssh user@remote 'cat /path/to/file' | while read line; ...

On a slow network, with enough processing capability on the remote side I'd let that machine do the processing and only transfer the results:
Code:
ssh user@remote 'while read line; do ...; done < /path/to/file'

This User Gave Thanks to pludi For This Post:
# 3  
Old 02-22-2011
Thank you, a good question. And actually, there are two differet systems, one is far away with high latency, acrsoss the globe over satellite link, but has realtive high bandwidth. One in just a few meteres away, even higher bandwidth and very low latency. All of them have ssh (I use ssh all the time to access them), so I think using ssh is the perfered method. Processing locally might be better as well I think, despite being over saetllite link. (Realtively, the files I have interest in is not too big, usually not more than ~60 lines, but can reach up to ~1000 is some rare cases).
# 4  
Old 02-22-2011
OK, so bandwidth is not a problem. Are there a lot of files (> ~5k)? Do they each have a header? The reason I'm asking is that for a high file count on a high latency line it's more efficient to do fewer invocations of ssh.
This User Gave Thanks to pludi For This Post:
# 5  
Old 02-22-2011
There is only one file for each system.

The files look roughly like this:

Code:
value1,value2,...,value12,value13
value1,value2,...,value12,value13
...
value1,value2,...,value12,value13
value1,value2,...,value12,value13

Some of the values are strings with spaces included.
# 6  
Old 02-22-2011
Well, then all you need would be
Code:
ssh user@$IPADRESS "cat $FILENAME" | awk 'BEGIN{ FS=OFS="," } { print $2, $7, $13 }'

This User Gave Thanks to pludi For This Post:
# 7  
Old 02-22-2011
Thanks a bunch! Gonna try it out later today Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read file input in while loop does not work on AIX system

I'm working on Aix 6.1 and using ksh shell. The below works fine on Linux bash or ksh shell . while IFS= read -r dirpath ; do echo "Hi" done <<<"$var" However, any such while loop that reads the input from file or variable using <<< fails on Aix system with the below error: Below... (2 Replies)
Discussion started by: mohtashims
2 Replies

2. Shell Programming and Scripting

Ssh remote command doesn't work from script file

I have 10 application servers in a distributed architecture generating their own application logs. Each server has application utility to continuously tail the log. for example following command follows tails and follows new logfiles as they are generated server1$ logutility logtype When I run... (8 Replies)
Discussion started by: indianya
8 Replies

3. Shell Programming and Scripting

SSH - remote output locally

The code below works ok, however; I need to output the results to a local variable the_path="/mnt/back/hang" ssh -T -i /home/buddy/.ssh/id_rsa buddy@ginger << EOF find ${the_path} -name "*.jpg" | wc -l > ## output to local variable exit EOF (3 Replies)
Discussion started by: squrcles
3 Replies

4. Shell Programming and Scripting

Editing a file on remote server using shell script locally

Hi Scott, My previous post was not for any school or college projects. I am currently working with a IT company (Cannot provide more details than this). I am trying to implement the below script in my day-to-day work, Apologies for the confusion in previous post :). My question remains same... (4 Replies)
Discussion started by: Nishant Ladiwal
4 Replies

5. Shell Programming and Scripting

Editing a file on remote server using shell script locally

Hi All, I have below requirements for my project: 1. Building a shell script which connects to a remote server 2. running script on local machine as user 'A' 3. connecting to server using user 'B' with password 4. login with a powerbroker role 'P' (asks for same password as 'B') on that... (1 Reply)
Discussion started by: Nishant Ladiwal
1 Replies

6. Shell Programming and Scripting

Reading file from remote host?

flamingo:~ jnojr$ ssh -t macbook sudo -v Password: Connection to macbook closed. flamingo:~ jnojr$ ssh -t macbook sudo cat /etc/cma.conf | grep CMABuildNumber <CMABuildNumber>2918</CMABuildNumber> Connection to macbook closed. flamingo:~ jnojr$ ssh -t macbook sudo -k Connection to macbook... (3 Replies)
Discussion started by: jnojr
3 Replies

7. UNIX for Advanced & Expert Users

Using remote rsync, but copy locally?

I'm looking to use rsync to compare remote files and to copy the diff to a local directory, rather than transfer over the net. The net connection is not fast enough to transfer these files (~1.8TB) and I'd like to sneakernet them instead. Possible? (4 Replies)
Discussion started by: dfbills
4 Replies

8. UNIX and Linux Applications

invoke remote graphical application..and display locally

Hi, I want to invoke(run) a graphical application remotely, and the display should be in remote itself. (no X redirect).i want to do this through ssh. like if i login to a remote machine and run firefox it should display there itself. how can i do this..? (2 Replies)
Discussion started by: madhusudankh
2 Replies

9. UNIX for Dummies Questions & Answers

how to mount a file system of a remote machine to local file system

Hi friends, In my case, there are serveral PCs running Linux in a LAN. I would like to to mount the directory /A_river of machine-A to the file system of another machine machine-B so that I can access files in that directory. I do not know how to do this. The situation is complicated by... (2 Replies)
Discussion started by: cy163
2 Replies

10. Filesystems, Disks and Memory

reading unix file system from windows

I found a more appropriate section to post my question. Thanks! (0 Replies)
Discussion started by: cneill
0 Replies
Login or Register to Ask a Question

Featured Tech Videos