Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

wget output file names

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting wget output file names
# 1  
Old 07-16-2012
wget output file names
Hi,

I have a list of urls in my input.txt file like this

input.txt

Code:
http://unix.com/index.html?acc=OSR765454&file=filename1.gz
http://unix.com/index.html?acc=OBR765454&file=filename111.gz
http://unix.com/index.html?acc=ORS765454&file=filename1111.gz
http://unix.com/index.html?acc=OST76454&file=filename11111.gz
http://unix.com/index.html?acc=OS5454&file=filename1111111.gz


I am using this command
Code:
wget -i input.txt

My output files are coming with a weird file names like this

ls in file folder

Code:
index.html?acc=OSR765454&file=filename1.gz
index.html?acc=OBR765454&file=filename111.gz
index.html?acc=ORS765454&file=filename1111.gz
index.html?acc=OST76454&file=filename11111.gz
index.html?acc=OS5454&file=filename1111111.gz

How do I change this to just the filenames, like filename1.gz, filename11,gz...etc?

Thanks

---------- Post updated at 04:03 PM ---------- Previous update was at 03:18 PM ----------

Hi, I just figured out the solution.

Append -O desired_filename to the input urls.

For ex

Code:
http://unix.com/index.html?acc=OSR765454&file=filename1.gz -O filename1.gz

Now, issue this command

Code:
cat inputlist.txt | xargs wget inputlist.txt

You will get the filename as filename1.gz

Hope this helps someone.
Last edited by jacobs.smith; 07-16-2012 at 04:19 PM.. Reason: forgot code tags for wget command I have used
This User Gave Thanks to jacobs.smith For This Post:
Vryali
# 2  
Old 07-16-2012
Thanks for posting the solution!
This User Gave Thanks to Vryali For This Post:
jacobs.smith
# 3  
Old 07-16-2012
Useless Use of Cat

Code:
xargs ... < inputfile

This User Gave Thanks to Corona688 For This Post:
jacobs.smith
# 4  
Old 07-16-2012
Hi guys,

My solution doesn't work. It leaves u with only the last downloaded file.

Even @Corona's solution is still doing the same.

Any other possibilities?
# 5  
Old 07-16-2012
Code:
while read URL; do
  wget $URL -O ${URL##*=}
done < input.txt

Ends up being:
Code:
wget http://unix.com/index.html?acc=OSR765454&file=filename1.gz -O filename1.gz
wget http://unix.com/index.html?acc=OBR765454&file=filename111.gz -O filename111.gz
wget http://unix.com/index.html?acc=ORS765454&file=filename1111.gz -O filename1111.gz
wget http://unix.com/index.html?acc=OST76454&file=filename11111.gz -O filename11111.gz
wget http://unix.com/index.html?acc=OS5454&file=filename1111111.gz -O filename1111111.gz

# 6  
Old 07-16-2012
Hi Scott,

Thanks for ur time.

I have more than 400 links.

So, the best possible solution would be to put all those links in a file and use wget -i inputlist.txt by changing the output file names.
# 7  
Old 07-16-2012
Quote:
Originally Posted by jacobs.smith
So, the best possible solution would be to put all those links in a file and use wget -i inputlist.txt by changing the output file names.
Except that you can't tell wget how to filter the urls to generate the filenames. I believe scott's solution is the best you can do. Besides, doing the network i/o is probably going to be the bottleneck anyway, not the shell loop.

Alternatively, you can stick with your original approach, which generates the undesirable filenames, and rename the files afterwards (perhaps with the rename utility).

In my opinion, scott's solution is preferable. Keep it simple. Smilie

Regards,
Alister
Last edited by alister; 07-16-2012 at 09:25 PM..
This User Gave Thanks to alister For This Post:
jacobs.smith
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to list files names and sizes in a directory and output result to the file?

Hi , I'm trying to list the files and output is written to a file. But when I execute the command , the output file is being listed. How to exclude it ? /tmp file1.txt file2.txt ls -ltr |grep -v '-' | awk print {$9, $5} > output.txt cat output.txt file1.txt file2.txt output.txt (8 Replies)
Discussion started by: etldeveloper
8 Replies

2. Shell Programming and Scripting

Print the output with different file names

I have a python script that gives output called test.png. By using the following command I run the script every 2 seconds. What is the easiest way to save the output as follows ( test.png (1st output), tes1.png (second output), tes2.png ....) Command I i use while sleep 2; do python... (1 Reply)
Discussion started by: quincyjones
1 Replies

3. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

4. Shell Programming and Scripting

Custom wget output

The below hides the messy commands of wget #!/bin/bash cd 'C:\Users\cmccabe\Desktop\wget' wget -O getCSV.txt http://172.24.188.113/data/getCSV.csv progressfilt () { local flag=false c count cr=$'\r' nl=$'\n' while IFS='' read -d '' -rn 1 c do if $flag ... (5 Replies)
Discussion started by: cmccabe
5 Replies

5. Shell Programming and Scripting

Exclude certain file names while selectingData files coming in different names in a file name called

Data files coming in different names in a file name called process.txt. 1. shipments_yyyymmdd.gz 2 Order_yyyymmdd.gz 3. Invoice_yyyymmdd.gz 4. globalorder_yyyymmdd.gz The process needs to discard all the below files and only process two of the 4 file names available ... (1 Reply)
Discussion started by: dsravanam
1 Replies

6. Shell Programming and Scripting

Encapsulating output of CURL and/or WGET

i use curl and wget quite often. i set up alarms on their output. for instance, i would run a "wget" on a url and then search for certain strings within the output given by the "wget". the problem is, i cant get the entire output or response of my wget/curl command to show up correctly in... (3 Replies)
Discussion started by: SkySmart
3 Replies

7. Shell Programming and Scripting

ery weird wget/curl output - what should I do?

Hi, I'm trying to write a script to download RedHat's errata digest. It comes in a txt.gz format, and i can get it easily with firefox. HOWEVER: output is VERY strange when donwloading it in a script. It seems I'm getting a file of the same size - but partially text and partly binary! It... (5 Replies)
Discussion started by: jstilby
5 Replies

8. Shell Programming and Scripting

Searching for file names in a directory while ignoring certain file names

Sun Solaris Unix Question Haven't been able to find any solution for this situation. Let's just say the file names listed below exist in a directory. I want the find command to find all files in this directory but at the same time I want to eliminate certain file names or files with certain... (2 Replies)
Discussion started by: 2reperry
2 Replies

9. Shell Programming and Scripting

download a particular file using wget

Hi All I want to download srs8.3.0.1.standard.linux24_EM64T.tar.gz file from the following website : http://downloads.biowisdomsrs.com/srs83_dist/ But this website contains lots of zipped files I want to download the above file only discarding other zipped files. When I am trying the... (1 Reply)
Discussion started by: alphasahoo
1 Replies

10. UNIX for Dummies Questions & Answers

wget output question

Hello there, İ want to ask a very simple question. I want to read the output messages of wget both in terminal and also put them into a text file. i know that by using -o flag, i can log the messages into a text file but then i won't be able to see them on terminal. I'd appreciate any help... (1 Reply)
Discussion started by: sertansenturk
1 Replies
Login or Register to Ask a Question