Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Wget from multiple paths
Top Forums Shell Programming and Scripting Wget from multiple paths Post 302938091 by kumaran_5555 on Thursday 12th of March 2015 02:52:42 AM
Old 03-12-2015
wget can recursively pull file from a webpage - provided that webpage has links to other files. This is how recursion happens.

You can't enumerate from a home page to all of its subpages if you don't have any links in the home page. Logically there is no way for wget to find what all pages in that domain (brute force search is simply not practical).

If you have links to other pages, then you can use
Code:
--accept-regex urlregex

Code:
wget

to restrict what links are recursively pulled.

In your case, if you have one web page which provides links to say "path1, path2 ..." and each pathX provides further links, you can do what you want through
Code:
wget -r

 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

Multiple Paths to SAN with LVM in Linux?

I'm setting up a CentOS 5 server that will be connected to an HP EVA4000. My previous experience is with an HP VA7400 and HP-UX. In HP-UX I can add "alternate paths" to a volume group in order to have redundant paths to the SAN via dual fiber channel HBAs just by either adding them with... (3 Replies)
Discussion started by: deckard
3 Replies

2. Shell Programming and Scripting

Enabling a script to run in multiple paths

I have a script that i need to run from different paths. for example mypc/path1/path2/, mypc/path1/path2/path3/, and mypc/path1/path2/path3/path4 How do i set up that script so that it can execute in any of the above paths or how can i make it run on any path on my computer?? (5 Replies)
Discussion started by: dowell
5 Replies

3. Shell Programming and Scripting

find multiple paths

How do i find files in more than one directory? I searched through forums, but could not land into the right thread. I tried something like find dir1|dir2 -name file1 but it doesn't work. Please suggest. (5 Replies)
Discussion started by: krishmaths
5 Replies

4. Shell Programming and Scripting

KSH - Find paths of multiple files in CC (dir and sub-dir))

Dear Members, I have a list of xml files like abc.xml.table prq.xml.table ... .. . in a txt file. Now I have to search the file(s) in all directories and sub-directories and print the full path of file in a output txt file. Please help me with the script or command to do so. ... (11 Replies)
Discussion started by: Yoodit
11 Replies

5. Shell Programming and Scripting

Replace directory paths in multiple files at once

I need to update about 2400 files in a directory subtree, with a new directory path inside the files I need to change this occurence in all files: /d2/R12AB/VIS/apps/tech_st/10.1.2 with this: /u01/PROD/apps/apps_st/10.1.3 I know how to change single words using "find . -type f -print0 |... (6 Replies)
Discussion started by: wicus
6 Replies

6. Shell Programming and Scripting

Script to delete files older than x days and also taking an input for multiple paths

Hi , I am a newbie!!! I want to develop a script for deleting files older than x days from multiple paths. Now I could reach upto this piece of code which deletes files older than x days from a particular path. How do I enhance it to have an input from a .txt file or a .dat file? For eg:... (12 Replies)
Discussion started by: jhilmil
12 Replies

7. Shell Programming and Scripting

Storing multiple file paths in a variable

I am working on a script for Mac OS X that, among many other things, gets a list of all the installed Applications. I am pulling the list from the system_profiler command and formatting it using grep and awk. The problem is that I want to be able to use each result individually later in the script.... (3 Replies)
Discussion started by: cranfordio
3 Replies

8. Shell Programming and Scripting

Download multiple files uing wget

Need Assistance . Using wget how can i download multiple files from http site. Http doesnt has wild card (*) but FTP has it . Any ideas will be appreciative. wget --timeout=120 --append-output=output.txt --no-directories --cut-dirs=1 -np -m --accept=grib2 -r http://sample.com/... (4 Replies)
Discussion started by: ajayram_arya
4 Replies

9. Shell Programming and Scripting

Wget to download multiple source code

Can a modified command be used to download multiple source codes from specific sites and output each into a separate output file?. All the sites are in a text file (attached): wget -qO- http://www.genedx.com/test-catalog/available-tests/edar-gene-sequencing/ | cat > output.txt (4 Replies)
Discussion started by: cmccabe
4 Replies

10. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

LEARN ABOUT DEBIAN

grcompiler

grcompiler(1)															     grcompiler(1)

NAME

       grcompiler - Compile GDL into a TrueType Font

SYNOPSIS

       grcompiler [-d] [-D] [-g] [-n number] [-v number] [-w number] [-wall] [-q] gdl-file font-file [output-file] [output-fontname]

Description
       Grcompiler  compiles  a	Graphite Description Language description of the smarts for a font along with a TrueType Font to create a graphite
       font with graphite tables in it.

       -d     Generate XML debugger file.

       -D     Generate all debugger files.

       -g     Permit and ignore invalid glyph definitions.

       -nnumber
	      Set name table start location.

       -q     Quiet mode (no messages except on error).

       -vnumber
	      Set Silf table version number.

       -wnnnn Ignore warning with the given number. This option may be repeated.

       -wall  Display all warnings.

Environment
       grcompiler makes use of the following environment variables

       GDLPP  Specifies the command to execute to run the GDL preprocessor

Files
       gdlerr.txt
	      Warnings and errors are output to this file.

       dbg_classes.txt
	      Debug file that specifies the classes stored in the font in terms of their GDL source and containing glyph ids in hex.

       dbg_cmap.txt
	      Debug file giving Unicode to glyph mapping of all the glyphs in the font and vice versa.

       dbg_enginecode.txt
	      Debug file containing the action code and constraint code associated with each pass and rule in a pass.

       dbg_fsm.txt
	      Debug file that gives the finite state machines of each pass and the rules referenced. For each pass, a  mapping	is  given  between
	      glyph and column number.

       dbg_glyphattrs.txt
	      Debug file containing all the glyph attributes and their names, along with values specified for each glyph.

       dbg_parsetree.txt
	      Debug file giving the complete parse tree of the GDL source file.

       dbg_ruleprec.txt
	      Debug file that lists the rules in precedence order (order in which they will be tested) for each pass.

       $_temp.gdl
	      Debug file containing the output of gdlpp

Author
       Martin Hosken <mhosken@users.sourceforge.net>

							   $Date: 2012-05-29 11:36:50 $ 					     grcompiler(1)
All times are GMT -4. The time now is 02:41 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy