Sponsored Content
Full Discussion: wget pdf downloading problem
Top Forums UNIX for Dummies Questions & Answers wget pdf downloading problem Post 302485475 by fpmurphy on Wednesday 5th of January 2011 09:57:43 AM
Old 01-05-2011
There is no easy way to do what you want to do using wget. Looking at the source for that page would have shown you what is going on.

For example, consider the document entitled "Drought-tolerant plant growth promoting Bacillus ... ". The corresponding PDF file is "930332435.pdf" To retrieve that document you would have to parse this HTML code
Code:
<a target="_new" href="./ftinterface~db=all~content=a930332435~fulltext=713240930" title="Click to view the PDF fulltext"

to the extract the content tag, i.e. a930332435, and build a new URL which wget could then use to retrieve the document.

A good technique to prevent website scrapping!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Problem Downloading Red Hat

I've been having problems downloading Red Hat 7.2 from their FTP site. It downloads rather slowly(between 2-3k/sec, I'm on broadband) and after about 10 minutes stops downloading altogether. Am I doing something wrong? (2 Replies)
Discussion started by: Tradewynd
2 Replies

2. Linux

Problem downloading linux kernel 2.6 on to AT91RM9200

Hi, I builded the linux kernel 2.6 with the following tool chain binutils:2.16 gcc:3.4.4 glibc:2.3.5 kernel:2.6.10 and applied the corresponding patches to it.I got the kernel Image.I downloaded the Image on to the AT91RM9200 board.But when i am booting the image it is showing the... (1 Reply)
Discussion started by: yugandhar
1 Replies

3. Shell Programming and Scripting

Problem in Downloading one day old files from FTP site

HI, I'm downloading one day old files from ftp site. Below is my script ---------------------------- printf "open $HOST \n" > ftp.cmd printf "user $USER $PASSWD\n" >> ftp.cmd printf "bin\n" >> ftp.cmd #printf "cd /Models/\n" >> ftp.cmd printf "prompt\n" >> ftp.cmd printf "for oldfile... (4 Replies)
Discussion started by: shekhar_v4
4 Replies

4. Shell Programming and Scripting

downloading form content as pdf

Hi All, I have a requirement of dowloading the dynamic form content displayed in a webpage as a pdf file. The form content is not too complex but intermediate - it has textboxes, images, textarea, radiobuttons,dropdowns etc. Can anyone suggest how i can achieve this? Your... (0 Replies)
Discussion started by: DILEEP410
0 Replies

5. Solaris

HTTP error while downloading solaris patches using wget

Hello, I am getting a HTTP error while downloading solaris patches using wget. 'Downloading unsigned patch 113096-03. --2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40 Connecting to... (5 Replies)
Discussion started by: sunny_a_j
5 Replies

6. UNIX for Dummies Questions & Answers

Crontab Wget, downloading a file to a FTP

Hi there, I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours. I've used this command: wget -t inf http : / / www . somesite . com / webcam.jpg ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces... (24 Replies)
Discussion started by: zYx
24 Replies

7. Shell Programming and Scripting

Downloading with Wget

Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have... (2 Replies)
Discussion started by: keltonhalbert
2 Replies

8. Shell Programming and Scripting

Wget error while downloading from https website

Hi, I would like to download a file from a https website. I don't have the file name as it changes every day. I am using the following command: wget --no-check-certificate -r -np --user=ABC --password=DEF -O temp.txt https://<website/directory> I am getting followin error in my... (9 Replies)
Discussion started by: pinnacle
9 Replies

9. Shell Programming and Scripting

Download pdf's using wget convert to txt

wget -i genedx.txt The code above will download multiple pdf files from a site, but how can i download and convert these to .txt? I have attached the master list (genedx.txt - which contains the url and file names) as well as the two PDF's that are downloaded. I am trying to have those... (7 Replies)
Discussion started by: cmccabe
7 Replies

10. Shell Programming and Scripting

Wget for downloading a public file (stream) as mp4

I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the... (5 Replies)
Discussion started by: 1in10
5 Replies
NETSELECT-APT(1)					      General Commands Manual						  NETSELECT-APT(1)

NAME
netselect-apt - create sources.list for the fastest Debian mirrors SYNOPSIS
netselect-apt [OPTIONS] [stable|testing|unstable|experimental|release_codename|sid] DESCRIPTION
netselect-apt automatically creates a sources.list file for using with apt for the specified distribution by downloading the list of Debian mirrors using wget choosing the fastest servers using netselect, and testing the valid servers using curl (if available). The output file is written to OUTFILE. The list of fastest servers is determined by checking through netselect, which servers responder faster to ICMP queries. In order to deter- mine if the servers are valid a connectiong using the specificied protocol (HTTP or FTP) is done using curl. If netselect is not installed setuid, then netselect-apt needs to run as an administrator user (i.e. root). This is only required because the network probes done by netselect requires these permissions. No changes are done to the system. If -i INFILE is passed netselect-apt uses that rather than downloading another copy to a temporary file. The file will be downloaded from http://www.debian.org/mirror/mirrors_full OPTIONS
stable|testing|unstable|experimental|release_codename|sid Specify which distribution of Debian to use. By default stable is used. -a, --arch ARCH Use mirrors containing ARCH. By default the architecture of the current machine is used as reported by dpkg -s, --sources While generating OUTFILE include also deb-src lines to use with ``apt-get source'' to obtain Debian source packages. -i, --infile INFILE Use INFILE instead of downloading the mirror list to a temporary file. The file must be in the same format as mirrors_full. -o, --outfile OUTFILE Use OUTFILE instead of sources.list. -n, --nonfree Include also non-free section while generating OUTFILE. -f, --ftp Use FTP mirrors instead of HTTP and generate OUTFILE accordingly. -O OPTIONS The OPTIONS provided are added, verbatim, to netselect when it is run. Here you can provide a (quoted) list of options for netse- lect. -t, --tests hosts Make a short list with the number of hosts provided and use that list to test for mirror validity. By default 10 hosts are tested. -c, --country COUNTRY Only test the sites found under the country COUNTRY (the value can either be an ISO-3166 value or the full name of the language, in English). When this value is set the mirror list or the INFILE will be filtered and only the sites that are listed under the given country will be tested. Note that restricting the search might not give the best results, as the "fastest" mirror might not even be in the same country as the system the program is running in. ENVIRONMENT
WANT_SOURCES setting this to 1 is equivalent to --sources WANT_NONFREE setting this to 1 is equivalent to --nonfree LIMITATIONS
netselect-apt is unable to work with restricted environments in which network filtering is implemented as it relies on netselect being able to find a suitable mirror. To do this, the system where the script is run needs to have network visibility of the mirrors, as it will probe them using ICMP probes. netselect-apt is also unable to work in environments where HTTP or FTP network connections have to be done through a proxy host, as it relies on being able to test the validity of the remote mirrors doing direct network connections to them. netselect-apt will not check if the mirror it suggests as the "fastest" mirror is either valid or up-to-date. It is recommended that users that use this tool also validate that the mirrors suggested are official mirrors and are also current. SEE ALSO
netselect(1), wget(1), curl(1), apt(8), sources.list(5). For Debian GNU/Linux it is recommended that users review the official mirror list at http://www.debian.org/mirror/official as well as the mirror checker tool at http://mirror.debian.org/status.html (which provides information on the up-to-dateness status of mirrors) AUTHOR
Avery Pennarun <apenwarr@gmail.com> This manual page and program have been also enhanced by Filippo Giunchedi <filippo@esaurito.net> and Javier Fernandez-Sanguino <jfs@debian.org> DEBIAN
March 6, 2008 NETSELECT-APT(1)
All times are GMT -4. The time now is 07:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy