Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Store filenames for wget in bash

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Store filenames for wget in bash
# 1  
Old 02-08-2016
Store filenames for wget in bash
I have a bash that downloads a list of files as a text file using wget. What I now need to do is store those files names and pass them to a download call also using wget.

List.txt in /home directory
Code:
FilterDuplicates.html
file1.bam
file2.bam
file3.bam
file1.vcf.gz
file2.vcf.gz
file3.vcf.gz

Download
Code:
wget --user=xxxxx--password=xxxxx --xxxx \
https://www.xxxx.com/xxx/file to download -o wget.log

where file to download would be FilterDuplicates.html, then after that downloads file1.bam, then file2.bam, then file3.bam, then file1.vcf.gz, then file2.vcf.gz, then file3.vcf.gz. Each one of the bam files is ~20GB all others are 10MB or less. Thank you Smilie.

edit:
It looks like I need to alter the command that downloads the list to include the url, but I'm not sure how.

Code:
wget -q --user=xxxx --password=xxxxx --xxxxx \
https://www.xxxx.com/list

list
Code:
https://www.xxxx.com/FilterDuplicates.html
https://www.xxxx.com/file1.bam
https://www.xxxx.com/file2.bam
https://www.xxxx.com/file3.bam
https://www.xxxx.com/file1.vcf.gz
https://www.xxxx.com/file2.vcf.gz
https://www.xxxx.com/file3.vcf.gz

Last edited by cmccabe; 02-08-2016 at 05:46 PM.. Reason: added edit
# 2  
Old 02-08-2016
Take a look at the -i flag in wget to download from a file.
This User Gave Thanks to Aia For This Post:
cmccabe
# 3  
Old 02-09-2016
Try either one or both:
Code:
# Method one: list contains only filenames
while read F;do
	wget -q --user=xxxx --password=xxxxx --xxxxx \
		https://www.xxxx.com/$F
done<<~/list.txt

# Method two: list contains full url
wget -q --user=xxxx --password=xxxxx --xxxxx -i list.txt

hth
This User Gave Thanks to sea For This Post:
cmccabe
# 4  
Old 02-09-2016
Thank you both Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash: Picking up filenames then moving to next directory

I have a collection of directories, for example as below I want to create a loop that goes in the first directory and picks up the *hhz*.sac.pzs filename in a variable, and the other files matching *hhz*.sac in another variable (however I do not want to pick the *hhz*.sac.pzs). This is because... (4 Replies)
Discussion started by: kristinu
4 Replies

2. Shell Programming and Scripting

Wget in bash using sed and awk

In the bash below when the program is opened the download function runs and downloads the getCSV file and on the screen "Downloading getCSV.csv:%" displays and when it completes the menu function is called. However, as of now the bash opens and closes after a few seconds and I'm not sure... (4 Replies)
Discussion started by: cmccabe
4 Replies

3. Shell Programming and Scripting

Wget in bash

I am attempting to write a bash that starts by using wget and getting the following errors: Stand-alone code that works: wget -O getCSV.txt http://172.24.xxx.xxx/data/getCSV.csv c:\cygwin\home\cmccabe\NGS.sh: line 2: $'\r': command not found : No such file or directorysh: line 3:... (2 Replies)
Discussion started by: cmccabe
2 Replies

4. Shell Programming and Scripting

How to take the filenames from a directory and store into a file??

hi, how can i take the file names from a directory and store only the filenames in the file. suppose i have a directory which contains the following files and subdirectories. $ ls -ltr total 16 -rw-rw-r-- 1 adm etc 4 Aug 6 20:37 s1.txt -rw-rw-r-- 1 adm etc 4 Aug 6 20:37 s2.txt... (11 Replies)
Discussion started by: Little
11 Replies

5. IP Networking

Using Apache2 to Store Files for Accessing with wget?

Hello All, I have a Virtual Machine that I basically use for just testing stuff on. It is running SLES 11.1 and Apache2. I was able to get Apache2 set-up and working... I was able to insert a basic index.html page (i.e. a simple "Hello World" html page) just to check and make sure I can... (4 Replies)
Discussion started by: mrm5102
4 Replies

6. Shell Programming and Scripting

New file should store all the 7 existing filenames and their record counts and ftp th

Hi, I need help regarding below concern. There is a script and it has 7 existing files(in a path say,. usr/appl/temp/file1.txt) and I need to create one new blank file say “file_count.txt” in the same script itself. Then the new file <file_count.txt> should store all the 7 filenames and... (1 Reply)
Discussion started by: pr293
1 Replies

7. Shell Programming and Scripting

Bash script - stripping away characters that can't be used in filenames

I want to create a temp file which is named based on a search string. The search string may contain spaces or characters that aren't supposed to be used in filenames so I want to strip those out. My thought was to use 'tr' with but the result is the opposite of what I want: $ echo "test... (5 Replies)
Discussion started by: mglenney
5 Replies

8. Shell Programming and Scripting

Whitespace in filenames in for loop in bash script

I'm trying to search all .odt files in a directory for a string in the text of the file. I've found a bash script that works, except that it can't handle whitespace in the filenames. #!/bin/bash if ; then echo "Usage: searchodt searchterm" exit 1 fi for file in $(ls *.odt); do ... (4 Replies)
Discussion started by: triplemaya
4 Replies

9. Shell Programming and Scripting

bash: reading filenames from file

Hi, I'm trying to write a script that reads filenames from a file and use these filenames in a loop. The filenames are all on one line and the problem is that these filenames have wildcards like * and braces like in them. Right now what I'm doing is something like this: echo "reading from... (0 Replies)
Discussion started by: warp17
0 Replies

10. Shell Programming and Scripting

How to script wget in bash?

The script below is giving me grief! The error message says /download.bash: line 16: syntax error near unexpected token `else' ./download.bash: line 16: `else wget "http://downloads.sourceforge.net/hibernate/hibernate-3.2.5.ga.zip?modtime=1185893922&big_mirror=1" ' I think it must be a... (1 Reply)
Discussion started by: siegfried
1 Replies
Login or Register to Ask a Question