Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Wget to download multiple source code

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Wget to download multiple source code
# 1  
Old 09-29-2014
Wget to download multiple source code
Can a modified command be used to download multiple source codes from specific sites and output each into a separate output file?. All the sites are in a text file (attached):

Code:
 wget -qO- http://www.genedx.com/test-catalog/available-tests/edar-gene-sequencing/ | cat > output.txt

file.txt (222 Bytes) 
Last edited by cmccabe; 09-29-2014 at 05:20 PM..
# 2  
Old 09-29-2014
yes, yes it can...
Code:
 cat file.txt | while read a
do
//generate a uniquefilename//
wget ${a} >  //uniquefilename//
done

# 3  
Old 09-29-2014
Nothing needs cat's help to read a file.

Code:
while read LINE
do
...
done < inputfile

But in this case, you don't even have to do the loop:

Code:
wget -i urls.txt

And wget should generate the unique filenames for you, along with full paths if you do -x:

Code:
wget -x -i urls.txt

This User Gave Thanks to Corona688 For This Post:
sea
# 4  
Old 10-03-2014
Code:
 wget -x -i file.txt

The code connects to each website and creates a separate file, but can it download the source code so that it can be parsed?

For example,

Code:
 wget -qO- http://www.genedx.com/test-catalog/available-tests/edar-gene-sequencing/ | cat > output.txt

downloads and creates a output file of the source code for the particular test.

Then, in a separate command
Code:
 awk -F '<[^>]*>' '{$1=$1}1' OFS='' output

the <> tags are removed, so the desired information can be obtained. Its not a perfect solution but it works. Thanks.
# 5  
Old 10-07-2014
If I have the file attached (source), which is a list of 10 sites:

can that file be used in to output each of the sites source code in a separate file?

Code:
 wget -qO- Source.txt | cat > output.txt

Thanks.
Source.txt (853 Bytes) 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

2. Shell Programming and Scripting

Download multiple files uing wget

Need Assistance . Using wget how can i download multiple files from http site. Http doesnt has wild card (*) but FTP has it . Any ideas will be appreciative. wget --timeout=120 --append-output=output.txt --no-directories --cut-dirs=1 -np -m --accept=grib2 -r http://sample.com/... (4 Replies)
Discussion started by: ajayram_arya
4 Replies

3. Shell Programming and Scripting

How to cancel wget download after 1%?

I am running a video download test and automating that. I wanna know how to stop a wget download session when downloads reached 1% Thanks in advance, Tamil (11 Replies)
Discussion started by: tamil.pamaran
11 Replies

4. Shell Programming and Scripting

Files download using wget

Hi, I need to implement below logic to download files daily from a URL. * Need to check if it is yesterday's file (YYYY-DD-MM.dat) * If present then download from URL (sample_url/2013-01-28.dat) * Need to implement wait logic if not present * if it still not able to find the file... (1 Reply)
Discussion started by: rakesh5300
1 Replies

5. UNIX and Linux Applications

download file using wget

I need to download the following srs8.3.0.1.standard.linux26_32.tar.gz file from the following website: http://downloads.biowisdomsrs.com/srs83_dist There are many gzip files along with the above one in the above site but I want to download the srs8.3.0.1.standard.linux26_32.tar.gz only from... (1 Reply)
Discussion started by: alphasahoo
1 Replies

6. Shell Programming and Scripting

download a particular file using wget

Hi All I want to download srs8.3.0.1.standard.linux24_EM64T.tar.gz file from the following website : http://downloads.biowisdomsrs.com/srs83_dist/ But this website contains lots of zipped files I want to download the above file only discarding other zipped files. When I am trying the... (1 Reply)
Discussion started by: alphasahoo
1 Replies

7. UNIX for Dummies Questions & Answers

help find wget, less, unzip source code

Where can I find the source code to basic unix core utilities like less, wget, and unzip? I'm on a HP-UX system that is missing a lot of basic tools. I Don't have admin access to the box. Google searches won't give me the source code. I would like to install some of the missing tools, like... (4 Replies)
Discussion started by: goldfish
4 Replies

8. UNIX for Dummies Questions & Answers

Using wget to download a file

Hello Everyone, I'm trying to use wget recursively to download a file. Only html files are being downloaded, instead of the target file. I'm trying this for the first time, here's what I've tried: wget -r -O jdk.bin... (4 Replies)
Discussion started by: thoughts
4 Replies

9. UNIX for Dummies Questions & Answers

source code download

hai This is A.gowri kumar from india.I am very facinated with this operating system. so i want to download the whole source code of this operating system (UNIX). So from where can i Download the source code. Please help me. (1 Reply)
Discussion started by: tapanagkumar
1 Replies

10. Programming

multiple source code

Is there any site that has the source code for just about all the apps that usually come default installed on most *nix systems (su, grep, find, etc...). Im average at c/c++ programming and feal like taking on a new challenge, understanding the source of well know apps. (2 Replies)
Discussion started by: minion
2 Replies
Login or Register to Ask a Question