Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

(w)get web server's directories + bash script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting (w)get web server's directories + bash script
# 1  
Old 08-20-2008
(w)get web server's directories + bash script
Hi everyone.

I need to write a script which will download files/folders (a huge collection) to the local file server (centOS 4.4 Server),
and check regularly (every 6 hours or so if any new files are present, or if the old ones were modified to update contents).

Any insights on how to tackle this problem is greatly appreciated.

Thx.
# 2  
Old 08-20-2008
sounds like the perfect job for rsync

rsync - Wikipedia, the free encyclopedia

Quote:
Originally Posted by Wikipedia
rsync is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion.

In daemon mode, rsync listens to the default TCP port of 873, serving files in the native rsync protocol. rsync can also be used to synchronize local directories, or via a remote shell such as RSH or SSH. In the latter case, the rsync client executable must be installed on both the local and the remote host.
# 3  
Old 08-20-2008
Quote:
Originally Posted by broli
sounds like the perfect job for rsync

rsync - Wikipedia, the free encyclopedia

Thanks for the reply, it certainly gave me an alternative how to handle local synch of files; however, I need to get the files through (authed) http off the internet.

What would be the easiest way to do it?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to block first bash script until second bash script script launches web server/site?

I'm new to utilities like socat and netcat and I'm not clear if they will do what I need. I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes. I'm using Cygwin bash on Windows 10. My... (3 Replies)
Discussion started by: siegfried
3 Replies

2. Shell Programming and Scripting

Need BASH Script Help to Move Files While Creating Directories

I've got this script to loop through all folders and move files that are more than 2 years old. I'm using the install command because it creates the necessary directories on the destination path and then I remove the source. I'd like to change the script to use the mv command since it is much... (4 Replies)
Discussion started by: consultant
4 Replies

3. Shell Programming and Scripting

Refresh web page in bash script

hello, I am trying to refresh my web page which is created in bash script. I have a HTML page which when press a button calls a bash script. this bash script created the same page with dynamic data. When pressing the button I am calling to a function that set time out of 7 seconds and and after... (1 Reply)
Discussion started by: SH78
1 Replies

4. Shell Programming and Scripting

Chose list of sub directories in a bash script file

Hi, I have a command in my bash script, searchDirectoryName.sh: DIR_NAME=$(find . -type d) echo "${DIR_NAME}" . ./Dir1 ./Dir1/1.0.2.1 ./Dir2 ./Dir2/1.1 ./Dir3 ./Dir3/2.2.1 How can I select only following directory names with second subdirectoies and without first ./ in the... (3 Replies)
Discussion started by: hce
3 Replies

5. Shell Programming and Scripting

Compare & Copy Directories : Bash Script Help

Beginner/Intermediate shell; comfortable in the command line. I have been looking for a solution to a backup problem. I need to compare Directory 1 to Directory 2 and copy all modified or new files/directories from Directory 1 to Directory 3. I need the directory and file structure to be... (4 Replies)
Discussion started by: Rod
4 Replies

6. Shell Programming and Scripting

bash script to rename multiple directories

Hello I have a directory structure with year in format 4 digits, e.g 2009, below which is month format 1 or 2 digits, e.g 1 or 12, blow which is day format 1 or 2 digits, e.g 1 or 31. I want to change the names of lots of directories to the be Year - 4 digits , e.g 2009 - No change here... (4 Replies)
Discussion started by: garethsays
4 Replies

7. Shell Programming and Scripting

Bash Script to Read & Write on different directories

Hi, root@server] df -h 121G 14G 101G 12% /home 147G 126G 14G 91% /backup We having our site files and images are storing in /backup/home/user/files/ through symbolic link created in /home directory pointing in /backup directory as following. root@server] cd /home... (1 Reply)
Discussion started by: mirfan
1 Replies

8. Shell Programming and Scripting

Bash script: issue changing directories in script

I am working on a script that checks two arguments at the command line. The first argument is a search pattern, the second can be a file or a directory, if it is a file a second script is called that checks it for the search pattern. If the second argument is a directory, it checks for the search... (5 Replies)
Discussion started by: Breakology
5 Replies

9. Shell Programming and Scripting

Script to telnet web server

A similar script I wrote long time ago, it was working, but it was deleted and now for some reasons my new script doesnt work. I have been searching for a while in google but no luck so far. Can anyone help me to spot syntax errors with this little script here: #!/bin/sh { sleep 2 ... (5 Replies)
Discussion started by: bsddaemon
5 Replies

10. Shell Programming and Scripting

Bash Script to display directories in Path?

Hello there, As a newbie: The directories in PATH can be hard to distinguish when printed out as one line with colon .Please, can i have a sample script to display them,one to a line. Thank you. (1 Reply)
Discussion started by: debut
1 Replies
Login or Register to Ask a Question