Help with WGET and renaming downloaded files :(


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with WGET and renaming downloaded files :(
# 1  
Old 08-18-2010
Help with WGET and renaming downloaded files :(

Hi everybody, I would greatly appreciate some expertise in this matter. I am trying find an efficient way to batch download files from a website and rename each file with the url it originated from (from the CLI). (ie. Instead of xyz.zip, the output file would be http://www.abc.com/xyz.zip) A method using WGET is preferable but not absolutely necessary. I'm just starting to get comfortable with the command line; in part because of some of the great help I've gotten here before, so once again I'm back looking for some help from this excellent community Smilie

Thanks

Last edited by o0110o; 08-18-2010 at 10:01 PM.. Reason: Revision
# 2  
Old 08-18-2010
Do you try -O option in wget?

Code:
  -O   --output-document=FILE   write documents to FILE.

# 3  
Old 08-18-2010
Quote:
Originally Posted by rdcwayx
Do you try -O option in wget?

Code:
  -O   --output-document=FILE   write documents to FILE.

Thanks for your response Smilie Will -O replace all the downloaded files filenames with their corresponding original url addresses?
# 4  
Old 08-18-2010
you can customize the output file name with -O option, or assign a var to it.
# 5  
Old 08-19-2010
Quote:
Originally Posted by rdcwayx
you can customize the output file name with -O option, or assign a var to it.
Okay, so far so good. Thanks for the tip. All I need to do now is figure out how I can automatically replace the filenames with their respective url addresses; is this somehow possible using grep or sed with the
Code:
wget -a logfile

switch enabled?
# 6  
Old 08-19-2010
You cannot name a file "http://www.abc.com/xyz.zip". Forward slashes are one of two illegal characters in unix filenames (null byte being the other). The best you can do without some kind of translation is to mirror the hierarchy, with each slash-delimited component in the url, except the last, being a directory. Even so, the "//" cannot be handled without some special treatment as a "//" in a pathname is treated identically to "/".

Last edited by alister; 08-19-2010 at 01:07 AM..
# 7  
Old 08-19-2010
Quote:
Originally Posted by alister
You cannot name a file "http://www.abc.com/xyz.zip". Forward slashes are one of two illegal characters in unix filenames (null byte being the other). The best you can do is mirror the hierarchy, with each slash-delimited component in the url, except the last, being a directory. Even so, the "//" cannot be handled without some special treatment as a "//" in a pathname is treated identically to "/".
Thanks for your feedback alistar Smilie Maybe you can answer this one for me: How would I go about printing the url of a particular photo onto that photo (ie. watermark) as soon as its downloaded? I have already had limited success with the "convert" command using predefined text, however, I still haven't figured out the auto-url-watermark capability that I'm after. Thanks again!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting multiple files off an ftp server once they have been downloaded

Hello, I have a server that I have to ftp files off and they all start SGRD and are followed by 6 numbers. SGRD000001 SGRD000002 SGRD000003 The script I have will run every 10 mins to pick up files as new ones will be coming in all the time and what I want to do is delete the files I have... (7 Replies)
Discussion started by: sph90457
7 Replies

2. Shell Programming and Scripting

Renaming multiple files in sftp server in a get files script

Hi, In sftp script to get files, I have to rename all the files which I am picking. Rename command does not work here. Is there any way to do this? I am using #!/bin/ksh For eg: sftp user@host <<EOF cd /path get *.txt rename *.txt *.txt.done ... (7 Replies)
Discussion started by: jhilmil
7 Replies

3. Shell Programming and Scripting

For loop till the files downloaded

Need assistance in writing a for loop script or any looping method. Below is the code where i can get all the files from the URL . There are about 80 files in the URL .Every day the files get updated . Script that i wanted is the loop must keep on running till it gets 80 files. It matches the count... (5 Replies)
Discussion started by: ajayram_arya
5 Replies

4. Shell Programming and Scripting

BASH scripting - Preventing wget messed downloaded files

hello. How can I detect within script, that the downloaded file had not a correct size. linux:~ # wget --limit-rate=20k --ignore-length -O /Software_Downloaded/MULTIMEDIA_ADDON/skype-4.1.0.20-suse.i586.rpm ... (6 Replies)
Discussion started by: jcdole
6 Replies

5. Shell Programming and Scripting

Specific image to be downloaded with wget

Hello All, I have gone through Google and came to know that we can download images from a site using wget. Now I am been asked to check whether an image is populated in a site or not. If yes, please send that image to an address as an attachment.. Say for example, the site is Wiki -... (6 Replies)
Discussion started by: sathyaonnuix
6 Replies

6. Shell Programming and Scripting

renaming files or adding a name in the beginning of all files in a folder

Hi All I have a folder that contains hundreds of file with a names 3.msa 4.msa 21.msa 6.msa 345.msa 456.msa 98.msa ... ... ... I need rename each of this file by adding "core_" in the begiining of each file such as core_3.msa core_4.msa core_21.msa (4 Replies)
Discussion started by: Lucky Ali
4 Replies

7. Shell Programming and Scripting

Extract urls from index.html downloaded using wget

Hi, I need to basically get a list of all the tarballs located at uri I am currently doing a wget on urito get the index.html page Now this index page contains the list of uris that I want to use in my bash script. can someone please guide me ,. I am new to Linux and shell scripting. ... (5 Replies)
Discussion started by: mnanavati
5 Replies

8. Web Development

php files are downloaded

Hello, I have setup Cherokee web server and php 5.2 in Opensolaris zone. Problem is that all .php files are downloaded from web server and not served when I use IP address instead of DNS name in web brovser. Example: test.mydomain.com <-- php works 192.168.0.10/index.php <--... (3 Replies)
Discussion started by: kreno
3 Replies

9. Shell Programming and Scripting

how to limit files downloaded by wget

I am trying to download a page and retrieve only wav and mp3 files via wget. the website is: Alarm Sounds | Free Sound Effects | Alarm Sound Clips | Sound Bites my command is : wget -rl 2 -e robots=off -A wav,mp3 http://soundbible.com/tags-alarm.html When not using the -A wav,mp3... (2 Replies)
Discussion started by: Narnie
2 Replies

10. UNIX for Advanced & Expert Users

question regarding ftp. Files downloaded are of size Zero.

I need to download some files from a remote server using ftp. I have ftp'd into the site. I then do an mget * to retrieve all of the data files. Everything seems to proceed normally and I am given feedback that the files were downloaded. Now if I go into the DOS Shell or Windows explorer, it list... (5 Replies)
Discussion started by: ralphisnow
5 Replies
Login or Register to Ask a Question