UNIX for Advanced & Expert Users

There is an aspx page which, when accessed with certain parameters in the URL, downloads a file. I need to do this with a shell script. I tried using wget, but I get a response of 302 from the server, which redirects me to the default page and then downloads the default page html itself. I quickly tried curl which seemed to be doing the same thing. It works perfectly from browsers on either Linux or Windows. Originally, I had a problem with interpretation of ampersands in the URL. but I put quotation marks around the URL, so that isn't the problem now.

I cannot fix this on the server side, because it will be from a variety of servers which are all set differently and which aren't under our control.

Thanks in advance for any help you can give me.

Brandon

It may be blocking you because you're the wrong browser or have the wrong referer. Try

Code:

wget -U netscape --referer="http://site.com" "http://site.com/stuff"

Or it may actually be doing the redirection in the browsers too, but redirects to a page containing javascript junk which causes the file download.

Thanks, but I am not using a browser at all. I am trying to do this in a shell script, or, actually, at this stage, just by typing it in at the command line. What is a referrer?

If you'd read my post you'd have seen it contained instructions for using wget, and if you'd read wget's man page you would have found explanations for the options in it.

Exactly. So it tells servers that it's wget and gives them no referrer, with the result that many servers won't give it the time of day. Informs the web server what webpage they came from. Like if you loaded Google in a graphical browser, which loads among other things http://www.google.ca/intl/en_com/images/srpr/logo1w.png, the referer for that would be Google Load that same image from wget and it would have no referer.

Many providers refuse to let images and files be downloaded if there's no referer or a wrong referer, so you have to tell them you're actually coming from that page.

I see, so the server may be employing some type of security based on header information, which could be changed with the -U and --referrer options. I will give it a try Monday morning. Thanks.

---------- Post updated at 08:09 AM ---------- Previous update was at 08:07 AM ----------

I tried "-U Mozilla" and "-U Mozilla/5.0" and it made no difference. I tried using a referrer, to the extent that I understand the concept, and it also made no difference. As a referrer, I used the page which would be used to download the file interactively in a browser. Anyway, it still gave me a 302 redirect and downloaded the default page.

Did you try -U netscape like I suggested? And what did you use as the referer? All it is is what page the browser tells the server the request is coming from.

If user agents and referers don't help, there may be cookies involved.

I tried "-U nestcape," "-U Mozilla," and "-U Mozilla/5.0."

As the referer, I used the page that one would download it from manually. I also tried just the site address.

That is, I tried both of:

http://www.site.com/ThePage.aspx
http://www.site.com/

Since I cannot make wget work through the page from which people access this manually, I am resorting now to a completely different methodology, but it is less desirable.

Thanks for your input.