Shell Programming and Scripting

Hi, for my own interest I want to scrape a lot of data off the Maple Story game rankings page.

The problem is, when I want to get the data at this page

maplestory(dot)nexon(dot)net/Rankings/OverallRanking.aspx?type=overall&s=&world=0&job=0&pageIndex=6

It gives me the data at this page

maplestory(dot)nexon(dot)net/Rankings/OverallRanking.aspx?type=overall

so I think it has to do with the ampersands. I've tried it with curl/wget/lynx --dump and none of them work. Maybe I'm just missing a command or using the wrong tool. Does anyone have advice?

Thanks.

& means something special to the shell, yes. It means 'put this command in the background'.

so it's trying to run these commands simultaneously:

Code:

http://maplestory.nexon.net/Rankings/OverallRanking.aspx?type=overall
s=
world=0
job=0
pageIndex=6

Put the URL in quotes to make & be &.

Code:

wget "http://maplestory.nexon.net/Rankings/OverallRanking.aspx?type=overall&s=&world=0&job=0&pageIndex=6"

Thank you. I'm surprised I missed that. I may as well ask one more question -

In terms of grabbing webpages -

is there a faster way to grab webpages or is this mostly limited by the speed of the internet connection? I was playing around with wget, lynx --dump and curl and they're all at 0.5 seconds at the fastest. curl is the slowest.

I suppose you're running a new instance of wget to download every single webpage? You don't have to run wget 900 times to download 900 pages, try:

Code:

cat <<EOF | wget -i -
http://url1.com/path/to/whatever
http://url1.com/path/to/whatever2
http://url1.com/path/to/whatever3
http://url1.com/path/to/whatever4
EOF

Much faster, especially when the URL's are from the same site which lets it reuse the connection.