Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Wget from multiple paths
Top Forums Shell Programming and Scripting Wget from multiple paths Post 302938091 by kumaran_5555 on Thursday 12th of March 2015 02:52:42 AM
Old 03-12-2015
wget can recursively pull file from a webpage - provided that webpage has links to other files. This is how recursion happens.

You can't enumerate from a home page to all of its subpages if you don't have any links in the home page. Logically there is no way for wget to find what all pages in that domain (brute force search is simply not practical).

If you have links to other pages, then you can use
Code:
--accept-regex urlregex

Code:
wget

to restrict what links are recursively pulled.

In your case, if you have one web page which provides links to say "path1, path2 ..." and each pathX provides further links, you can do what you want through
Code:
wget -r

 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

Multiple Paths to SAN with LVM in Linux?

I'm setting up a CentOS 5 server that will be connected to an HP EVA4000. My previous experience is with an HP VA7400 and HP-UX. In HP-UX I can add "alternate paths" to a volume group in order to have redundant paths to the SAN via dual fiber channel HBAs just by either adding them with... (3 Replies)
Discussion started by: deckard
3 Replies

2. Shell Programming and Scripting

Enabling a script to run in multiple paths

I have a script that i need to run from different paths. for example mypc/path1/path2/, mypc/path1/path2/path3/, and mypc/path1/path2/path3/path4 How do i set up that script so that it can execute in any of the above paths or how can i make it run on any path on my computer?? (5 Replies)
Discussion started by: dowell
5 Replies

3. Shell Programming and Scripting

find multiple paths

How do i find files in more than one directory? I searched through forums, but could not land into the right thread. I tried something like find dir1|dir2 -name file1 but it doesn't work. Please suggest. (5 Replies)
Discussion started by: krishmaths
5 Replies

4. Shell Programming and Scripting

KSH - Find paths of multiple files in CC (dir and sub-dir))

Dear Members, I have a list of xml files like abc.xml.table prq.xml.table ... .. . in a txt file. Now I have to search the file(s) in all directories and sub-directories and print the full path of file in a output txt file. Please help me with the script or command to do so. ... (11 Replies)
Discussion started by: Yoodit
11 Replies

5. Shell Programming and Scripting

Replace directory paths in multiple files at once

I need to update about 2400 files in a directory subtree, with a new directory path inside the files I need to change this occurence in all files: /d2/R12AB/VIS/apps/tech_st/10.1.2 with this: /u01/PROD/apps/apps_st/10.1.3 I know how to change single words using "find . -type f -print0 |... (6 Replies)
Discussion started by: wicus
6 Replies

6. Shell Programming and Scripting

Script to delete files older than x days and also taking an input for multiple paths

Hi , I am a newbie!!! I want to develop a script for deleting files older than x days from multiple paths. Now I could reach upto this piece of code which deletes files older than x days from a particular path. How do I enhance it to have an input from a .txt file or a .dat file? For eg:... (12 Replies)
Discussion started by: jhilmil
12 Replies

7. Shell Programming and Scripting

Storing multiple file paths in a variable

I am working on a script for Mac OS X that, among many other things, gets a list of all the installed Applications. I am pulling the list from the system_profiler command and formatting it using grep and awk. The problem is that I want to be able to use each result individually later in the script.... (3 Replies)
Discussion started by: cranfordio
3 Replies

8. Shell Programming and Scripting

Download multiple files uing wget

Need Assistance . Using wget how can i download multiple files from http site. Http doesnt has wild card (*) but FTP has it . Any ideas will be appreciative. wget --timeout=120 --append-output=output.txt --no-directories --cut-dirs=1 -np -m --accept=grib2 -r http://sample.com/... (4 Replies)
Discussion started by: ajayram_arya
4 Replies

9. Shell Programming and Scripting

Wget to download multiple source code

Can a modified command be used to download multiple source codes from specific sites and output each into a separate output file?. All the sites are in a text file (attached): wget -qO- http://www.genedx.com/test-catalog/available-tests/edar-gene-sequencing/ | cat > output.txt (4 Replies)
Discussion started by: cmccabe
4 Replies

10. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

LEARN ABOUT DEBIAN

atr_analysis

ATR_ANALYSIS(1p)						   User Commands						  ATR_ANALYSIS(1p)

NAME

       ATR_analysis - analyse a smart card ATR

SYNOPSIS

       ATR_analysis [ATRstring]

DESCRIPTION

       ATR_analysis is used to parse the ATR (Answer To Reset) sent by a smart card.

       The command also tries to find the card model using an ATR database stored in a text file smartcard_list.txt.

       The smartcard_list.txt file is searched in ./, /usr/local/pcsc/ and /usr/share/pcsc/ directories.

       Exemple:
	$ ATR_analysis '3B A7 00 40 18 80 65 A2 08 01 01 52'
	ATR: 3B A7 00 40 18 80 65 A2 08 01 01 52
	+ TS = 3B --> Direct Convention
	+ T0 = A7, Y(1): 1010, K: 7 (historical bytes)
	  TB(1) = 00 --> Programming Param P: 0, I: 0
	  TD(1) = 40 --> Y(i+1) = 0100, Protocol T = 0
	-----
	  TC(2) = 18 --> Work waiting time: 960 x 24 x (Fi/F)
	+ Historical bytes: 80 65 A2 08 01 01 52

	Possibly identified card:
	3B A7 00 40 18 80 65 A2 08 01 01 52
	       Gemplus GPK8000

BUGS

       Maybe many bugs since I am not a ISO 7816 expert.

FILES

       smartcard_list.txt

SEE ALSO

       pcscd(8), pcsc_scan(1)

AUTHOR

       Ludovic Rousseau <ludovic.rousseau@free.fr>

Version: 1.3							   October 2005 						  ATR_ANALYSIS(1p)
All times are GMT -4. The time now is 12:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy