Bash to download specific files and save in two folders


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash to download specific files and save in two folders
# 1  
Old 05-30-2015
Bash to download specific files and save in two folders

I am trying to download all files from a user authentication, password protected https site, with a particular extension (.bam). The files are ~20GB each and I am not sure if the below is the best way to do it. I am also not sure how to direct the downloaded files to a folder as well as external drive. Thank you Smilie.

Code:
while true
do
      printf "Files down loading please wait "
      wget --http-user cmccabe --http -passwd xxxx*** -r -A "*.bam" -q --show-progress --directory-prefix=C:\Users\cmccabe\Desktop\Example --directory-prefix=E:\WD\files https://test.xxxx.com/xxxxx.aspx
      printf "Files downloaded "

done


Last edited by cmccabe; 05-30-2015 at 12:10 PM.. Reason: added line of code
# 2  
Old 05-31-2015
Well I would probably try to somehow incorporate a "counter" on the file downloaded. Maybe a for() incremental loop and if its on file X or file Y run the wget. If not you should have your script cd into a different directory and then run the wget for file Z. I hope that makes sense...
# 3  
Old 06-01-2015
I'm not sure I understand completly, do I need a counter if I am downloaded the same 3 files to two different folders simaltanously? Thank you Smilie.
# 4  
Old 06-01-2015
He ment to 'count' the files beeing downloaded, as in this simplified code:

Code:
U1=http://someadress.com
U2=http://someadress.com
U3=http://someadress.com

cd /path/to/downloaddir_1

for url in $U1 $U2 $U3
do  wget -o ${url/##*/} $url
    # The copy can be done in background so the next download begins.
    cp ${url/##*/} /path/to/downloaddir_2 &
done

hth
# 5  
Old 06-01-2015
Since the name of the file will change daily but the extension will always be the same (.bam), can that just be referenced in the 'count'

Code:
U1=http://someadress.com/*.bam
U2=http://someadress.com/*.bam
U3=http://someadress.com/*.bam

Also, how can the wget auto-login to the site to download the files? Thank you Smilie.
# 6  
Old 06-01-2015
As i said simplified, i didnt want to copy paste what you already posted.
So you could focus on what has changed.

You do not have 'local access' to remote files, so * will not work.
Code:
U1=http://someadress.com/file1.bam
U2=http://someadress.com/file2.bam
U3=http://someadress.com/file3.bam

You could shorten it to the file names and 'prefix' a hardcoded url at the wget command.

Code:
U1=file1.bam
U2=file2.bam
U3=file3.bam

for url in $U1 $U2 $U3
do  # Append the login stuff from your original post, i have no experience to login with wget
    wget -o $url http://someadress.com/$url
    # The copy can be done in background so the next download begins.
    cp $url /path/to/downloaddir_2/ &
done

# 7  
Old 06-01-2015
Quote:
Originally Posted by cmccabe
Also, how can the wget auto-login to the site to download the files? Thank you Smilie.
That depends entirely on the site in question.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Find files in specific folders

Hi Team, I am new to the linux commands and I really need help . I would be really thankful if I can get some inputs. I have below folders in the path "/home/temp" 20170428 20170427 20170429 changes tempI need to get the files generated in the last 15 mins in all the above folders... (4 Replies)
Discussion started by: JackJinu
4 Replies

2. Shell Programming and Scripting

Bash directory loop, but only choose those folders with specific word in it

Hello, how in bash i can get directory loop, but only choose those folders with specific word in it, so it will only echo those with specific word #!/bin/bash for filename in /home/test/* do if ; then echo $filename; fithx! (4 Replies)
Discussion started by: ZerO13
4 Replies

3. Shell Programming and Scripting

Bash to list all folders in a specific directory

The below bash is trying to list the folders in a specific directory. It seems close but adds the path to the filename, which basename could strip off I think, but not sure why it writes the text file created? This list of folders in the directory will be used later, but needs to only be the... (5 Replies)
Discussion started by: cmccabe
5 Replies

4. Shell Programming and Scripting

Bash to move specific files from folders in find file

I have a directory /home/cmccabe/nfs/exportedReports that contains multiple folders in it. The find writes the name of each folder to out.txt. A new directory is then created in a new location /home/cmccabe/Desktop/NGS/API, named with the date. What I am trying to do, unsuccessfully at the moment,... (7 Replies)
Discussion started by: cmccabe
7 Replies

5. Shell Programming and Scripting

Bash to tell download where specific files are stored

The bash below will download all the files in download to /home/Desktop/folder. That works great, but within /home/Desktop/folder there are several folders bam, other, and vcf, is there a way to specify by extention in the download file where to download it to? For example, all .pdf and .zip... (2 Replies)
Discussion started by: cmccabe
2 Replies

6. Shell Programming and Scripting

How to delete all the files and folders inside all the directories except some specific directory?

hi, i have a requirement to delete all the files from all the directories except some specific directories like archive and log. for example: there are following directories such as A B C D Archive E Log F which contains some sub directories and files. The requirement is to delete all the... (7 Replies)
Discussion started by: Little
7 Replies

7. Shell Programming and Scripting

FAQ how to download multiple specific files via command line

Hi, This evening i would like to download multiple pcap captures files in the wireshark wiki sites. My aim is to download the capture files .pcap .cap and etc on the wireshark site SampleCaptures - The Wireshark Wiki. i already used wget, lynx, htget but still their problem downloading..it seems... (1 Reply)
Discussion started by: jao_madn
1 Replies

8. UNIX for Dummies Questions & Answers

Monitoring specific files and folders

I want a mechanism to monitor a folder full of files that are sensitive. I want to log all accesses,modifications and changes to any file within the folder in a log file which should give me access/modify/change times,the user id of the process which tried and the pid. Even some idea of what to... (1 Reply)
Discussion started by: Vivek788
1 Replies

9. Shell Programming and Scripting

Copying specific files from remote m/c to specific folders

Hi All, I am trying to rsync some of the latest files from remote m/c to my local linux box. Folder structure in my remote m/c looks like this /pub/Nightly/Package/ROLL/WIN /pub/Nightly/Package/SOLL/sol /pub/Nightly/Package/SOLL/linux Each of the folder contains gzip files which on daily... (0 Replies)
Discussion started by: jhoomsharabi
0 Replies

10. Shell Programming and Scripting

decompressed files to specific folders

Hi all, I have a script below which meant to decompress *.tr.gz to allocated folder. e.g. hello.tar.gz should be decompress to a subdirectory called hello and all the files should be in hello subdirectory. How do i do it? Am i missing anything? Condition: there are existing tar.gz in teh... (3 Replies)
Discussion started by: c00kie88
3 Replies
Login or Register to Ask a Question