Help with using lynx/wget/curl when a link has an ampersand | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Help with using lynx/wget/curl when a link has an ampersand

Shell Programming and Scripting


Tags
curl, lynx, wget

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 01-29-2011
seagaia seagaia is offline
Registered User
 
Join Date: Jan 2011
Last Activity: 25 April 2011, 3:59 PM EDT
Posts: 3
Thanks: 1
Thanked 0 Times in 0 Posts
Help with using lynx/wget/curl when a link has an ampersand

Hi, for my own interest I want to scrape a lot of data off the Maple Story game rankings page.

The problem is, when I want to get the data at this page

maplestory(dot)nexon(dot)net/Rankings/OverallRanking.aspx?type=overall&s=&world=0&job=0&pageIndex=6

It gives me the data at this page

maplestory(dot)nexon(dot)net/Rankings/OverallRanking.aspx?type=overall

so I think it has to do with the ampersands. I've tried it with curl/wget/lynx --dump and none of them work. Maybe I'm just missing a command or using the wrong tool. Does anyone have advice?

Thanks.
Sponsored Links
    #2  
Old 01-29-2011
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 16 September 2014, 5:59 PM EDT
Location: Saskatchewan
Posts: 19,361
Thanks: 787
Thanked 3,256 Times in 3,052 Posts
& means something special to the shell, yes. It means 'put this command in the background'.

so it's trying to run these commands simultaneously:
Code:
http://maplestory.nexon.net/Rankings/OverallRanking.aspx?type=overall
s=
world=0
job=0
pageIndex=6

Put the URL in quotes to make & be &.


Code:
wget "http://maplestory.nexon.net/Rankings/OverallRanking.aspx?type=overall&s=&world=0&job=0&pageIndex=6"

The Following User Says Thank You to Corona688 For This Useful Post:
seagaia (01-29-2011)
Sponsored Links
    #3  
Old 01-29-2011
seagaia seagaia is offline
Registered User
 
Join Date: Jan 2011
Last Activity: 25 April 2011, 3:59 PM EDT
Posts: 3
Thanks: 1
Thanked 0 Times in 0 Posts
Thank you. I'm surprised I missed that. I may as well ask one more question -

In terms of grabbing webpages -

is there a faster way to grab webpages or is this mostly limited by the speed of the internet connection? I was playing around with wget, lynx --dump and curl and they're all at 0.5 seconds at the fastest. curl is the slowest.
    #4  
Old 01-31-2011
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 16 September 2014, 5:59 PM EDT
Location: Saskatchewan
Posts: 19,361
Thanks: 787
Thanked 3,256 Times in 3,052 Posts
I suppose you're running a new instance of wget to download every single webpage? You don't have to run wget 900 times to download 900 pages, try:


Code:
cat <<EOF | wget -i -
http://url1.com/path/to/whatever
http://url1.com/path/to/whatever2
http://url1.com/path/to/whatever3
http://url1.com/path/to/whatever4
EOF

Much faster, especially when the URL's are from the same site which lets it reuse the connection.
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Help needed in Curl & Wget dineshbabu01 Shell Programming and Scripting 0 05-14-2010 12:08 AM
Using WGET or CURL to HTTP Post with Authentication krypton Shell Programming and Scripting 2 02-26-2010 11:51 AM
piping a url to wget or curl? ropers Shell Programming and Scripting 5 03-21-2009 12:35 AM
wget - force link conversion for all links? Allasso Shell Programming and Scripting 1 02-15-2009 11:52 AM
The Ampersand Legend986 Programming 2 10-10-2007 04:04 PM



All times are GMT -4. The time now is 01:31 AM.