Shell Programming and Scripting

I need to copy a directory and all sub directories/files from an ftp server (no SSH access) to my own dedicated server.

So I need a script that will login via ftp and copy everything (recurse through the directories).

The problem I have is that some of the sub directories are symlinks and so far I only get the symlink created on my server and the files in the sub directories are not copied.

I have googled and used different methods but none seem to work. Does anyone know of any script that will ignore the symlink creation and just get the file(s) or another way to do this ?

There must be a way to do this as I used Cyberduck on my MAC and the files are downloading ok but I need to do this via SSH on the server so that the trafic is server to server and not have to download to my MAC and upload to the server.

Some details on the client side would be helpful.

Besides the mirroring of the host, is there no chance of tar'ing it this would preserve your symlinks and more.

I only have FTP access to the remote server and therefore I cannot create a tar file.

I do not want the symlinks, I just want the files that are in all the directories regardless of the symlinks.

My local server runs CentOS5.

FWIW, some ftp servers allow getting a directory structure as a tar ball. Its a very handy feature.

This works for me, it will recreate the symlinks, but it won't follow them or download them as files. Obviously there are tons of args related to "--no-follow-ftp", "--exclude-directories" and what have you to tune this.

Code:

wget -r --ftp-user USER --ftp-password PASS ftp://HOST/

I already tried ftp using wget with -r option and this does not get me the files that are in the directories which have a symlink.

Code:

--retr-symlinks 

When retrieving FTP directories recursively, follow symbolic links and retrieve the linked-to files.

If your trying to

Code:

ln -s /privatestuff ~/insidemystuff

most modern ftp servers chroot your ftp session to your own stuff or a specific area.

$ mkdir ~/tmp
$ cd tmp
$ mkdir a
$ touch a/1
$ ln -s a b

wget -r --retr-symlinks --ftp-user USER --ftp-password PASS HOST/tmp

i end up with
HOST/tmp/a/1
HOST/tmp/b/1

Thanks I will give that a go when I get home after work.