Mirror a folder from the Internet to local machine

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Mirror a folder from the Internet to local machine
# 1  
Old 01-02-2018
Mirror a folder from the Internet to local machine

I have a machine in my LAN that runs Ubuntu and I have full access in it.
There is another PC on the Internet that has a public IP eg 3.3.3.3 and it servers as https server and access to it is by username and password. On the remote PC all i have is access through https using username/passwd.

At the moment I'm logging to the ubuntu and using wget to get folders that I want.
I would like to automate the boring stuff and every 1 hour to check if the server has new content compared to the ubuntu's /var/www/html/x3 and /var/www/html/y7 local folders.
The Internet server has as series of folders eg x1, x2,...,xn, y1,y2,...,yn. i do need to get only 2 particular folder and its contents and not the whole server.
If those 2 folders have different content, then download the new content and delete the old ones in the corresponding folder.
It is a one way sync. is there any script that could do the job with cron, please?
# 2  
Old 01-02-2018
I think looking at the man pages of rsync command would be a good start, as rsync is very versatile and may just do the job you want, I used to use it to synchronise /etc/passwd and other changing stuff in clusters...
# 3  
Old 01-02-2018
Hi. Thanks a lot for the quick reply. I thought of rsync, but the only access to the remote PC is through https.
as i see it i need a script that checks the current directory against the remote. if there is difference then wget the new and delete the old ones.
# 4  
Old 02-09-2018
There's only one way synchronizing with wget -m <URL>
which will copy what in server to your local storage. Try
wget --spider .. read its official man; GNU Wget 1.18 Manual
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with fetching the data from remote machine from my jumpbox(local machine)

Team, Presently i am running a script from my local box(i.e jumpbox) to all the remote machines.Basically fetching basic queries like pwd,mkdir,touch etc and i am able to successfully fetch it from my local machine.But when i want to check certain database related queries like the dbstat... (20 Replies)
Discussion started by: whizkidash
20 Replies

2. Red Hat

iptables applied in local machine, can't ssh remote machine after chain changed to DROP

I want to SSH to 192.168.1.15 Server from my machine, my ip was 192.168.1.99 Source Destination was UP, with IP 192.168.1.15. This is LAN Network there are 30 Machine's Connected to the network and working fine, I'm Playing around the local machine's because I need to apply the same rules in... (2 Replies)
Discussion started by: babinlonston
2 Replies

3. UNIX for Advanced & Expert Users

FTP While transfering files to local machine to remote machine

Hi Am using unix Ksh Am getting the problem while transferring zero size files through the script . When i transfer zero size files from local machine to remote machine manually i can able to do it . My question its beause of zero size files am not able to transfer through script ? or its... (2 Replies)
Discussion started by: Venkatesh1
2 Replies

4. Shell Programming and Scripting

Taking mirror copy of the application id and password into the new UNIX machine

I am working in Datastage Migration project. The applications has to be moved from legacy machines to the new machines. all the applications will be having their own application id and password (non expiry) created in the unix (5.3). Now the scripts and the datastage applications has to be moved... (1 Reply)
Discussion started by: kmanivan82
1 Replies

5. Shell Programming and Scripting

How to transfer files from unix machine to local machine using shell script?

Hi All.. Am new to Unix!! Am creating a shell script in which a scenario is like i have transfer the output file from unix machine (Server) to local directory (Windows xp). And also i have to transfer the input file from the local directory to Unix machine (Server) Any help from you... (1 Reply)
Discussion started by: vidhyaS
1 Replies

6. Solaris

Solaris Volume Manager: Break Mirror and use to make a 2nd machine

Hello: I have a machine built on 2 drives and mirrors have been created to a second set of 2 drives (4 total) on the same platform. SVM. Might anyone provide guidance to create a second machine by breaking the mirrored set, moving the two mirrors (2) to another machine (same platform type),... (1 Reply)
Discussion started by: 4dailyrunner
1 Replies

7. Solaris

How to create mirror disk in solaris machine?

hi, I'm newbie in Solaris 10. can someone explain me the steps of how to create mirror disk in Solaris machine. thanks in advance (5 Replies)
Discussion started by: Wong_Cilacap
5 Replies

8. Red Hat

To find the LATEST file from a dir on REMOTE machine and SCP to local machine?

Hi All, URGENT - Please help me form a scipt for this: I need the LATEST file from a dir on REMOTE machine to be SCP'd to a dir on local machine. (and I need to execute this from local server) I know that the below cmd is used to find the LATEST file from a dir. But this command is not... (3 Replies)
Discussion started by: me_ub
3 Replies

9. Solaris

x86 Sun Machine and Solaris Soft Mirror

Hi everyone, Normally it always easier in Sparc machine, i can set or manually use the boot-device in NVram to boot the mirrored disk. However I have a big trouble about x86 mirror for a long time. I have been doing anything i can: search document, google, ask the others. Recently i did as... (5 Replies)
Discussion started by: tien86
5 Replies

10. UNIX for Dummies Questions & Answers

Copying multiple folders to local machine (don't know folder names)

Hi. I'm trying to copy multiple folders from the remote machine to the local machine. I wrote a batch file to run an ftp window. The problem I am having is that the only command to copy files is mget *, and this copies only files, not folders. For example, ftp ts555 cd ts555/test ' test... (5 Replies)
Discussion started by: leenyburger
5 Replies
Login or Register to Ask a Question