Sponsored Content
Top Forums Shell Programming and Scripting Store filenames for wget in bash Post 302966132 by cmccabe on Monday 8th of February 2016 03:57:43 PM
Old 02-08-2016
Store filenames for wget in bash

I have a bash that downloads a list of files as a text file using wget. What I now need to do is store those files names and pass them to a download call also using wget.

List.txt in /home directory
Code:
FilterDuplicates.html
file1.bam
file2.bam
file3.bam
file1.vcf.gz
file2.vcf.gz
file3.vcf.gz

Download
Code:
wget --user=xxxxx--password=xxxxx --xxxx \
https://www.xxxx.com/xxx/file to download -o wget.log

where file to download would be FilterDuplicates.html, then after that downloads file1.bam, then file2.bam, then file3.bam, then file1.vcf.gz, then file2.vcf.gz, then file3.vcf.gz. Each one of the bam files is ~20GB all others are 10MB or less. Thank you Smilie.

edit:
It looks like I need to alter the command that downloads the list to include the url, but I'm not sure how.

Code:
wget -q --user=xxxx --password=xxxxx --xxxxx \
https://www.xxxx.com/list

list
Code:
https://www.xxxx.com/FilterDuplicates.html
https://www.xxxx.com/file1.bam
https://www.xxxx.com/file2.bam
https://www.xxxx.com/file3.bam
https://www.xxxx.com/file1.vcf.gz
https://www.xxxx.com/file2.vcf.gz
https://www.xxxx.com/file3.vcf.gz


Last edited by cmccabe; 02-08-2016 at 05:46 PM.. Reason: added edit
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to script wget in bash?

The script below is giving me grief! The error message says /download.bash: line 16: syntax error near unexpected token `else' ./download.bash: line 16: `else wget "http://downloads.sourceforge.net/hibernate/hibernate-3.2.5.ga.zip?modtime=1185893922&big_mirror=1" ' I think it must be a... (1 Reply)
Discussion started by: siegfried
1 Replies

2. Shell Programming and Scripting

bash: reading filenames from file

Hi, I'm trying to write a script that reads filenames from a file and use these filenames in a loop. The filenames are all on one line and the problem is that these filenames have wildcards like * and braces like in them. Right now what I'm doing is something like this: echo "reading from... (0 Replies)
Discussion started by: warp17
0 Replies

3. Shell Programming and Scripting

Whitespace in filenames in for loop in bash script

I'm trying to search all .odt files in a directory for a string in the text of the file. I've found a bash script that works, except that it can't handle whitespace in the filenames. #!/bin/bash if ; then echo "Usage: searchodt searchterm" exit 1 fi for file in $(ls *.odt); do ... (4 Replies)
Discussion started by: triplemaya
4 Replies

4. Shell Programming and Scripting

Bash script - stripping away characters that can't be used in filenames

I want to create a temp file which is named based on a search string. The search string may contain spaces or characters that aren't supposed to be used in filenames so I want to strip those out. My thought was to use 'tr' with but the result is the opposite of what I want: $ echo "test... (5 Replies)
Discussion started by: mglenney
5 Replies

5. Shell Programming and Scripting

New file should store all the 7 existing filenames and their record counts and ftp th

Hi, I need help regarding below concern. There is a script and it has 7 existing files(in a path say,. usr/appl/temp/file1.txt) and I need to create one new blank file say “file_count.txt” in the same script itself. Then the new file <file_count.txt> should store all the 7 filenames and... (1 Reply)
Discussion started by: pr293
1 Replies

6. IP Networking

Using Apache2 to Store Files for Accessing with wget?

Hello All, I have a Virtual Machine that I basically use for just testing stuff on. It is running SLES 11.1 and Apache2. I was able to get Apache2 set-up and working... I was able to insert a basic index.html page (i.e. a simple "Hello World" html page) just to check and make sure I can... (4 Replies)
Discussion started by: mrm5102
4 Replies

7. Shell Programming and Scripting

How to take the filenames from a directory and store into a file??

hi, how can i take the file names from a directory and store only the filenames in the file. suppose i have a directory which contains the following files and subdirectories. $ ls -ltr total 16 -rw-rw-r-- 1 adm etc 4 Aug 6 20:37 s1.txt -rw-rw-r-- 1 adm etc 4 Aug 6 20:37 s2.txt... (11 Replies)
Discussion started by: Little
11 Replies

8. Shell Programming and Scripting

Wget in bash

I am attempting to write a bash that starts by using wget and getting the following errors: Stand-alone code that works: wget -O getCSV.txt http://172.24.xxx.xxx/data/getCSV.csv c:\cygwin\home\cmccabe\NGS.sh: line 2: $'\r': command not found : No such file or directorysh: line 3:... (2 Replies)
Discussion started by: cmccabe
2 Replies

9. Shell Programming and Scripting

Wget in bash using sed and awk

In the bash below when the program is opened the download function runs and downloads the getCSV file and on the screen "Downloading getCSV.csv:%" displays and when it completes the menu function is called. However, as of now the bash opens and closes after a few seconds and I'm not sure... (4 Replies)
Discussion started by: cmccabe
4 Replies

10. Shell Programming and Scripting

Bash: Picking up filenames then moving to next directory

I have a collection of directories, for example as below I want to create a loop that goes in the first directory and picks up the *hhz*.sac.pzs filename in a variable, and the other files matching *hhz*.sac in another variable (however I do not want to pick the *hhz*.sac.pzs). This is because... (4 Replies)
Discussion started by: kristinu
4 Replies
RBASH(1)						      General Commands Manual							  RBASH(1)

NAME
rbash - restricted bash, see bash(1) RESTRICTED SHELL
If bash is started with the name rbash, or the -r option is supplied at invocation, the shell becomes restricted. A restricted shell is used to set up an environment more controlled than the standard shell. It behaves identically to bash with the exception that the follow- ing are disallowed or not performed: o changing directories with cd o setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV o specifying command names containing / o specifying a file name containing a / as an argument to the . builtin command o specifying a filename containing a slash as an argument to the -p option to the hash builtin command o importing function definitions from the shell environment at startup o parsing the value of SHELLOPTS from the shell environment at startup o redirecting output using the >, >|, <>, >&, &>, and >> redirection operators o using the exec builtin command to replace the shell with another command o adding or deleting builtin commands with the -f and -d options to the enable builtin command o using the enable builtin command to enable disabled shell builtins o specifying the -p option to the command builtin command o turning off restricted mode with set +r or set +o restricted. These restrictions are enforced after any startup files are read. When a command that is found to be a shell script is executed, rbash turns off any restrictions in the shell spawned to execute the script. SEE ALSO
bash(1) GNU Bash-4.0 2004 Apr 20 RBASH(1)
All times are GMT -4. The time now is 07:08 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy