Unix/Linux Go Back    


Shell Programming and Scripting Unix shell scripting - KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and shell scripts and shell scripting languages here.

Help with using lynx/wget/curl when a link has an ampersand

Shell Programming and Scripting


Tags
curl, lynx, wget

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 01-29-2011
seagaia seagaia is offline
Registered User
 
Join Date: Jan 2011
Last Activity: 25 April 2011, 3:59 PM EDT
Posts: 3
Thanks: 1
Thanked 0 Times in 0 Posts
Help with using lynx/wget/curl when a link has an ampersand

Hi, for my own interest I want to scrape a lot of data off the Maple Story game rankings page.

The problem is, when I want to get the data at this page

maplestory(dot)nexon(dot)net/Rankings/OverallRanking.aspx?type=overall&s=&world=0&job=0&pageIndex=6

It gives me the data at this page

maplestory(dot)nexon(dot)net/Rankings/OverallRanking.aspx?type=overall

so I think it has to do with the ampersands. I've tried it with curl/wget/lynx --dump and none of them work. Maybe I'm just missing a command or using the wrong tool. Does anyone have advice?

Thanks.
Sponsored Links
    #2  
Old Unix and Linux 01-29-2011
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 26 August 2016, 5:27 PM EDT
Location: Saskatchewan
Posts: 21,430
Thanks: 967
Thanked 3,899 Times in 3,628 Posts
& means something special to the shell, yes. It means 'put this command in the background'.

so it's trying to run these commands simultaneously:
Code:
http://maplestory.nexon.net/Rankings/OverallRanking.aspx?type=overall
s=
world=0
job=0
pageIndex=6

Put the URL in quotes to make & be &.


Code:
wget "http://maplestory.nexon.net/Rankings/OverallRanking.aspx?type=overall&s=&world=0&job=0&pageIndex=6"

The Following User Says Thank You to Corona688 For This Useful Post:
seagaia (01-29-2011)
Sponsored Links
    #3  
Old Unix and Linux 01-29-2011
seagaia seagaia is offline
Registered User
 
Join Date: Jan 2011
Last Activity: 25 April 2011, 3:59 PM EDT
Posts: 3
Thanks: 1
Thanked 0 Times in 0 Posts
Thank you. I'm surprised I missed that. I may as well ask one more question -

In terms of grabbing webpages -

is there a faster way to grab webpages or is this mostly limited by the speed of the internet connection? I was playing around with wget, lynx --dump and curl and they're all at 0.5 seconds at the fastest. curl is the slowest.
    #4  
Old Unix and Linux 01-31-2011
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 26 August 2016, 5:27 PM EDT
Location: Saskatchewan
Posts: 21,430
Thanks: 967
Thanked 3,899 Times in 3,628 Posts
I suppose you're running a new instance of wget to download every single webpage? You don't have to run wget 900 times to download 900 pages, try:


Code:
cat <<EOF | wget -i -
http://url1.com/path/to/whatever
http://url1.com/path/to/whatever2
http://url1.com/path/to/whatever3
http://url1.com/path/to/whatever4
EOF

Much faster, especially when the URL's are from the same site which lets it reuse the connection.
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Help needed in Curl & Wget dineshbabu01 Shell Programming and Scripting 0 05-14-2010 12:08 AM
wget - force link conversion for all links? Allasso Shell Programming and Scripting 1 02-15-2009 11:52 AM
The Ampersand Legend986 Programming 2 10-10-2007 04:04 PM



All times are GMT -4. The time now is 05:48 PM.