11-18-2012
RE: wget -i URLs.txt
Hi Corona688,
Thanks for your post. The membership site I belong to is resell-rights-weekly.com and I just login and click the links to download to my home computer. I want to bypass my home computer and copy the files for that week's downloads. server to server is much faster than me trying to DSL them down and back up to my server. The input file is necessary because new downloads are put on the site each week. I will then put the urls in URLs.txt before the wget, set up as a cron to run every Monday and bring the files over in a fraction of the time to copy. I had it working partially but could not remember the switches I set.
Here is my next try: -->> wget -i URLs.txt --post-data 'user=klondrie&password=XXXX' -o wgetlogfile.txt -c
What do you think? What would you change? This should be a piece of cake. I do not see a lot of security as I can login and click the links to download to my computer. Need them on my server though.
Any more help available?
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi all,
Iam trying to sort the contents of the file based on the position of the file.
Example:
$cat sample.txt
0101020060731 ## Header record
1c1 Berger Awc ANP20070201301 4000.50
1c2 Bose W G ANP20070201609 6000.70
1c2 Andy CK ANP20070201230 28000.00... (3 Replies)
Discussion started by: ganapati
3 Replies
2. UNIX for Advanced & Expert Users
Hi,
I've tried to download from ftp sites by wget but it failed and says "Service unavailable" but when I use sftp in binary mode and use "get" command it works perfectly. What's the problem?
BTW: I tried both passive and active mode in wget.
thnx for ur help (9 Replies)
Discussion started by: mjdousti
9 Replies
3. Shell Programming and Scripting
Hi,
I want to download some patches from SUN by using a script and I am using "wget" as the utillity for this.
The website for downloading has a "https:" in its name as below
https://sunsolve.sun.com/private-cgi/pdownload.pl?target=${line}&method=h
and on running wget as below
wget... (1 Reply)
Discussion started by: max29583
1 Replies
4. Shell Programming and Scripting
Hi,
I need to basically get a list of all the tarballs located at uri
I am currently doing a wget on urito get the index.html page
Now this index page contains the list of uris that I want to use in my bash script.
can someone please guide me ,.
I am new to Linux and shell scripting.
... (5 Replies)
Discussion started by: mnanavati
5 Replies
5. UNIX for Dummies Questions & Answers
Hi, I'm trying to install some libraries, when running the makefile I get an error from the "wget --no check certificate option". I had a look help and the option wasn't listed. Anyone know what I'm missing. (0 Replies)
Discussion started by: davcra
0 Replies
6. UNIX for Dummies Questions & Answers
I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory)
diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//'
Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies
7. Shell Programming and Scripting
Dear people, I got a problem with an scrip using wget to download pdf-files from an website which uses session-cookies.
Background: for university its quite nasty to look up weekly which new homeworks, papers etc. are available on the different sites of the universites chairs. So I wanted a... (1 Reply)
Discussion started by: jackomo
1 Replies
8. Shell Programming and Scripting
wget -i genedx.txt
The code above will download multiple pdf files from a site, but how can i download and convert these to .txt?
I have attached the master list (genedx.txt - which contains the url and file names)
as well as the two PDF's that are downloaded. I am trying to have those... (7 Replies)
Discussion started by: cmccabe
7 Replies
9. Proxy Server
I cannot download anything using wget in centos 6.5 and 7. But I can update yum etc.
# wget https://wordpress.org/latest.tar.gz
--2014-10-23 13:50:23-- https://wordpress.org/latest.tar.gz
Resolving wordpress.org... 66.155.40.249, 66.155.40.250
Connecting to wordpress.org|66.155.40.249|:443...... (3 Replies)
Discussion started by: nirosha
3 Replies
wdel(1) Internet Applications - FTP wdel(1)
NAME
wdel - A wget-like program for deleting files on ftp servers
SYNOPSIS
wdel [option]... [URL [file]...]...
DESCRIPTION
Wdel is a free utility that is able to delete files on an ftp server.
Wdel is non-interactive and background-capable. It can delete files or whole directories and is meant to be a robust client even for unsta-
ble connections and will therefore retry to delete a file, if the connection broke.
Wdel supports connections through proxies, allowing you to use it in an environment that can access the internet only via a proxy or to
provide anonymity by hiding your ip-address to the server. For SOCKSv5-proxies Wdel supports also listening mode, allowing you to use
port-mode ftp through a proxy (useful if the remote ftp is behind a firewall or a gateway).
URL-Input-Handling
URLs are recognized by the ftp://-prefix.
Wdel first reads the URLs from the command line, and associates every file with the last URL until a new URL is found (note that this is
different from wput). Then, it reads the --input-file (if any) and parses the URLs in the same way. So you can specify e.g. one URL and
read all filenames from a file.
If there are no explicit filenames given for a singe URL, then the last part of the URL is taken to be the file/directory to be deleted. If
there are filenames, then the last part of the URL is considered to be the directory where these files reside in. Slashes at the end of the
URL or the filename (in case it is a directory to be deleted) are ignored.
OPTIONS
There are no special options introduced by Wdel. All of Wput's options may be supplied, although some will not be of any effect because
they do not apply for the process of deleting files. Please see wput(1) for a description of the options.
SEE ALSO
Many options can be set in a wputrc file. For its documentation consult the sample file provided by Wput.
wput(1) describes all setable options. Not all apply for wdel too though.
AUTHOR
Wdel is written by Hagen Fritsch <fritsch+wdel-man@in.tum.de> and Jan Larres <jan@majutsushi.net>
Hagen Fritsch, Jan Larres 0.6.2 wdel(1)