Wget -i URLs.txt problem | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

Wget -i URLs.txt problem

UNIX for Dummies Questions & Answers


Tags
wget

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 11-18-2012
Keith londrie Keith londrie is offline
Registered User
 
Join Date: Nov 2012
Last Activity: 18 November 2012, 8:59 PM EST
Posts: 2
Thanks: 0
Thanked 0 Times in 0 Posts
Hammer & Screwdriver Wget -i URLs.txt problem

Hi Everyone,

I have a problem with wget using an input file of URLs. When I execute this -> wget -i URLs.txt I get the login.php pages transferred but not the files I have in the URLs.txt file. I need to use the input file because it will have new products to download each week. I want my VA to fill that file with the URLs that we need to transfer to my server. There must be some login necessary as I must login through http with a user name and password. Is that what I need on the command line or is there a cookie type problem,. See, I can login and download the files just fine but want them on my server for faster transfer to me and wget is not working yet.

Any help is appreciated.

Thanks,

Keith
Sponsored Links
    #2  
Old 11-18-2012
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 31 August 2014, 3:15 PM EDT
Location: Saskatchewan
Posts: 19,271
Thanks: 774
Thanked 3,237 Times in 3,035 Posts
How to login depends on the website. You probably can't plug the list of files in directly, given there's likely some POST things needed before, but the file can probably at least be used to generate arguments for wget in a script.
Sponsored Links
    #3  
Old 11-18-2012
Keith londrie Keith londrie is offline
Registered User
 
Join Date: Nov 2012
Last Activity: 18 November 2012, 8:59 PM EST
Posts: 2
Thanks: 0
Thanked 0 Times in 0 Posts
RE: wget -i URLs.txt

Hi Corona688,

Thanks for your post. The membership site I belong to is resell-rights-weekly.com and I just login and click the links to download to my home computer. I want to bypass my home computer and copy the files for that week's downloads. server to server is much faster than me trying to DSL them down and back up to my server. The input file is necessary because new downloads are put on the site each week. I will then put the urls in URLs.txt before the wget, set up as a cron to run every Monday and bring the files over in a fraction of the time to copy. I had it working partially but could not remember the switches I set.

Here is my next try: -->> wget -i URLs.txt --post-data 'user=klondrie&password=XXXX' -o wgetlogfile.txt -c

What do you think? What would you change? This should be a piece of cake. I do not see a lot of security as I can login and click the links to download to my computer. Need them on my server though.

Any more help available?
    #4  
Old 11-18-2012
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 31 August 2014, 3:15 PM EDT
Location: Saskatchewan
Posts: 19,271
Thanks: 774
Thanked 3,237 Times in 3,035 Posts
You can't just dump raw post data into a page that's not expecting it, it won't work. You need to look at the page and see what it's actually doing.

In this case, the login page is ( I think ) http://resell-rights-weekly.com/members/members.php, which expects post-data like you've been given.

To save the cookies, you need to tell wget to save the cookies(including session cookies), see man wget. So you could call wget once to login and save the cookies, and then call it again, loading the cookies to download the rest.
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Problem with wget and cookie jackomo Shell Programming and Scripting 1 06-26-2012 12:32 AM
find lines in file1.txt not found in file2.txt memory problem raptor25 UNIX for Dummies Questions & Answers 5 07-15-2011 11:38 PM
Extract urls from index.html downloaded using wget mnanavati Shell Programming and Scripting 5 10-13-2010 09:26 PM
Problem with wget max29583 Shell Programming and Scripting 1 09-10-2008 07:40 AM
Sorting problem "sort -k 16,29 sample.txt > output.txt" ganapati Shell Programming and Scripting 3 08-01-2006 05:55 AM



All times are GMT -4. The time now is 05:03 PM.