Copy down remote files and rename them to include the server name with full path


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Copy down remote files and rename them to include the server name with full path
# 1  
Old 01-14-2013
[SOLVED] - Copy down remote files and rename them to include the server name with full path

I need to pull down a good bit of files for another support team for an upgrade project. I have a server.list with all of the server names.

I need to do two parts:

FIRST:
I have this example, but it does not list the server name in front of each line.

Code:
#! /bin/bash
for server in $(< server.list);
do
echo "********* $server *********"
ssh $server "sudo find /u01/ -name \"filename\""
done

I need to get an output file that will look similar to this:
SERVER1:/path/path/filename
SERVER2:/path/path/filename
SERVER3:/path/path/filename
SERVER3:/path/path/filename
so on (and I might have multi files on one server)...

SECOND:
Once I have the "new" list of all servers and files I need to be able to take that output and copy the files down to a remote server and rename the file to include the server name and full path.

EX: SERVER1/path/path/filename
Script will use the above to ssh to SERVER1 and copy the file from the full path then rename it to SERVER1_path_path_filename

This is needed so that the support team will know which file belongs to which server and the full path as one server could have multi files with the same name.

Last edited by asnatlas; 01-16-2013 at 10:27 AM..
# 2  
Old 01-14-2013
This solution uses sed to prepend the server name to filelist and scp to copy the files locally. Most servers that support ssh also support scp so it should work for you.

Code:
#!/bin/bash
 
for server in $(< server.list)
do
   ssh $server "find /u01/ -name \"filename\"" | sed "s/^/$server:/"
done > new.list
 
OLDIFS=$IFS
IFS=":"
while read server filepath
do
   scp $server:$filepath $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

This User Gave Thanks to Chubler_XL For This Post:
# 3  
Old 01-15-2013
THANK YOU VERY MUCH Chubler_XL

I had to add " " around the $server:$filepath for some reason.

Code:
OLDIFS=$IFS
IFS=":"
while read server filepath
do
   scp "$server:$filepath" $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

Without the " " I was getting scp SERVER /path/path/file (No such file or directory)

Once I updated it to "$server:$filepath" I would get scp SERVER:/path/path/file

Thanks again for your help Smilie

---------- Post updated at 03:19 PM ---------- Previous update was at 11:02 AM ----------

I was getting the following error on some of the servers because I was using sudo find:

sudo: sorry, you must have a tty to run sudo

I just added ssh -t and that appears to have fixed it Smilie
# 4  
Old 01-29-2013
Very odd... When I tested this out on Jan 15th it worked just fine... I try to run it today and I get the following error:

scp: ambiguous target

When I check the cmd it appears to be adding a : to the end of the line / filename
I am getting the following:
/path/filename: No such file or directory

My current script is :

Code:
#!/bin/bash
set -x

OLDIFS=$IFS
IFS=":"
while read server filepath
do
  scp -t "$server:$filepath" $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

Any ideas ??
# 5  
Old 01-29-2013
You probably have a blank line in the new.list file try:

Code:
#!/bin/bash
set -x

OLDIFS=$IFS
IFS=":"
while read server filepath
do
  [ -n "$filepath" ] && scp -t "$server:$filepath" $server${filepath//\//_}
done < new.list
IFS=$OLDIFS

# 6  
Old 01-29-2013
I double checked and saved the original new.list to new.list-backup. I removed all but 15 lines (no blank lines or spaces):

server1.domain.net:/path/path/path/filename
server2.domain.net:/path/path/path/filename
server3.domain.net:/path/path/path/filename
... on to server 15

I tried adding your update
Code:
[ -n "$filepath" ] &&

and I am still receiving the "scp: ambiguous target" for each line...

Like I said this worked fine on the Jan 15th. I did not update the script and tried to run it today to check again and am now getting the "scp: ambiguous target" error.

When I run the script with set -x I am getting the following:

Quote:
+ OLDIFS='
'
+ IFS=:
+ read server filepath
+ '[' -n $'/path/path/filename\r' ']'
+ scp -t $'server.domain.net:/path/path/filename\r' $'server.domain.net_path_path_filename\r'
scp: ambiguous target
+ read server filepath
+ IFS='
'
# 7  
Old 01-29-2013
Looks like your data file is in DOS format (perhaps you edited it with windows notepad or something) try $ dos2unix new.list to clean it up.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Rename (move) multiple files on remote server using sftp

I want to rename (move) multiple files on remote server. I tried the following command to move all TXT files from my_dir directory to /new_dir. But it does not work. Any help? #!/bin/ksh sftp -dev3 << ABC cd my_dir $(for i in TXT; do echo "ls *.$i" ; rename $x /new_dir/$x;... (1 Reply)
Discussion started by: Soham
1 Replies

2. Shell Programming and Scripting

Rsync - how to copy hidden folder or hidden files when using full path

Hello. I use this command : rsync -av --include=".*" --dry-run "$A_FULL_PATH_S" "$A_FULL_PATH_D"The data comes from the output of a find command. And no full source directories are in use, only some files. Source example... (2 Replies)
Discussion started by: jcdole
2 Replies

3. UNIX for Dummies Questions & Answers

How to copy files from remote server to local?

Hi experts, I 'm newbie to unix world, now I have task to copy the latest files from remote server to my local. I believe this must be very common request in this community. I want you do it one more time for me please. My requirement is something like this: I receive files in the below... (3 Replies)
Discussion started by: parpaa
3 Replies

4. UNIX for Dummies Questions & Answers

Checking files in remote server and decide to copy file or not

Hi there, I have a problem in my script, I need to check whether file exists in remote server or not, if the file exists, then stop copy else copy the file to the server.. my code is something like this while read $server do if ssh $server "cd $directory_name; if ; then echo "Error:... (2 Replies)
Discussion started by: beezy
2 Replies

5. Shell Programming and Scripting

make file (include files path)

Hi All, In make file i want to include header files from my local directory and if it did not find in local directory i want to include from network directory. can any help me how i can do this?. here is the code INCLUDE=${include}/ this is point to network dir how i can add option that it... (1 Reply)
Discussion started by: goraya430
1 Replies

6. UNIX for Dummies Questions & Answers

Can I copy files on remote server with ftp ?

I just realize the only way is to download and upload again.. is not possible to copy them remotely with the ftp protocol ? thanks (2 Replies)
Discussion started by: aneuryzma
2 Replies

7. UNIX for Dummies Questions & Answers

Copy files from remote server

Hi Friends, Could you please help me as per my requirement mentioned below ? I have to copy files from one unix server to another unix server, and the files that i need to copy from the remote server are only those which are modified/created Today from abc directory on the remote server (1 Reply)
Discussion started by: ramask
1 Replies

8. Shell Programming and Scripting

copy files from remote server (B) to target server (A)?

Hi All, what is the comand to log off the remote server? I have 2 servers A, B. I need to find all files older than 7 days on server B and copy over to server A. My logic is: login the remote server: ================= ssh hostB cd /data/test find . -mtime -7 -ls | awk '{print... (4 Replies)
Discussion started by: Beginer0705
4 Replies

9. Shell Programming and Scripting

rename files in remote server via ftp

Hi All, I want to rename set of files in the remote server which iam unable to do . i can rename only one file at a time . for example ftp $REMOTESERVER { rename $NAME1 $NAME2 } is working whereas ftp $REMOTESERVER { rename $NAME1*.dat $NAME2*.data } is not working ...... (4 Replies)
Discussion started by: dineshr85
4 Replies
Login or Register to Ask a Question