Wget error while downloading from https website


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Wget error while downloading from https website
# 1  
Old 08-16-2013
Wget error while downloading from https website

Hi,

I would like to download a file from a https website. I don't have the file name as it changes every day.

I am using the following command:

Code:
wget --no-check-certificate -r  -np --user=ABC --password=DEF -O temp.txt https://<website/directory>

I am getting followin error in my temp.txt file:
Quote:
Unable to load xml from message: Root element is missing.
I searched the entire forum and did google search but nothing helpful.

I plan to write shell script to automate this once I make this wget command to work.

Help is appreciated.
# 2  
Old 08-16-2013
I suspect wget is working correctly and the site is malfunctioning.

It's difficult to say why without knowing what site, but I suspect --user and --password aren't doing what you want. They will not input passwords into form-elements.
# 3  
Old 08-19-2013
Looks like website needs HTTPS POST Message.

I got following response from website administrator:

Quote:
That browser is setup to receive HTTPS POST messages, which is likely why you received the root element message.

If you were trying to send an XML message to that URL, please send me the XML so I can troubleshoot the message.
Help is appreciated with unix command to do this functionality.

I am using wget command not sure if this is the right command to use.
# 4  
Old 08-19-2013
From man wget:
Code:
       --post-data=string
       --post-file=file
           Use POST as the method for all HTTP requests and send the specified
           data in the request body.  --post-data sends string as data,
           whereas --post-file sends the contents of file.  Other than that,
           they work in exactly the same way. In particular, they both expect
           content of the form "key1=value1&key2=value2", with percent-encod-
           ing for special characters; the only difference is that one expects
           its content as a command-line paramter and the other accepts its
           content from a file.

The real question is, what data does this website expect in POST? There is no generic answer, that is up to their implementation, you'll have to look at its <form>s to see that.
# 5  
Old 08-21-2013
Looks like for the call to website I have to encode my password as follows:

first convert to uppercase,
then Unicode it in little-endian UTF 16,
Then SHA1 it then base64 encode it.

Unicode encoding = new UnicodeEncoding();
hashBytes = encoding.GetBytes(password.ToUpper().Trim());
SHA1 sha1 = new SHA1CryptoServiceProvider();
byte[] cryptPassword = sha1.ComputeHash(hashBytes);
String pwd= Convert.ToBase64String(cryptPassword);


Can we do all this in unix or will have to use some other technology. If all this can be done in unix.

Help is appreciated.
# 6  
Old 08-21-2013
Wow, they're really taking the long, long, long way around. Though I can kind of send the point of sending the hash, not the password -- except if anyone steals that hash instead, that's just as useful. In other words everyone that looks at the page without a browser sees right through it...

Last edited by Corona688; 08-21-2013 at 03:59 PM..
# 7  
Old 08-21-2013
Code:
STR=$(  printf asdf                     |
        tr 'a-z' 'A-Z'                  |
        iconv -f "UTF-8" -t "UTF-16LE"  |
        sha1sum                         |
        awk '{ for(N=1; N<=length($1); N+=2) printf("\\x" substr($1,N,2)); }'
        )

BASE64=$(printf "$STR" | openssl base64)

echo $BASE64

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Wget for downloading a public file (stream) as mp4

I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the... (5 Replies)
Discussion started by: 1in10
5 Replies

2. Shell Programming and Scripting

Downloading jpgs from a gallery type website

Can someone explain what this does step by step? I found this script on stackoverflow and want to customize it for personal use in downloading jpg images from a website. # get all pages curl 'http://domain.com/id/' -o '#1.html' # get all images grep -oh... (3 Replies)
Discussion started by: workisnotfun
3 Replies

3. AIX

wget https

Hello, can someone tell me where the certificate store on aix 6.1 is. If I want to use wget with a https site Iam getting an error that the issuer is untrusted. Thats right because its an self sign certificate. (3 Replies)
Discussion started by: ralphk
3 Replies

4. UNIX and Linux Applications

Problems with wget and https connection

Hi, I have wget for linux but I'm having trouble using it. I am trying to use it with a HTTPS connection to save cookies but it doesn't works. I do: wget --save-cookies=cookies.txt --post-data... (2 Replies)
Discussion started by: mierdatuti
2 Replies

5. Shell Programming and Scripting

Downloading with Wget

Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have... (2 Replies)
Discussion started by: keltonhalbert
2 Replies

6. UNIX for Dummies Questions & Answers

wget pdf downloading problem

Hi. I am trying to make a mirror of this free online journal: http://www.informaworld.com/smpp/title~content=t716100758~db=all Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Discussion started by: obo1234
5 Replies

7. UNIX for Dummies Questions & Answers

Crontab Wget, downloading a file to a FTP

Hi there, I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours. I've used this command: wget -t inf http : / / www . somesite . com / webcam.jpg ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces... (24 Replies)
Discussion started by: zYx
24 Replies

8. Windows & DOS: Issues & Discussions

Downloading a file from Website to a Windows Folder

Hi, Is it possible to download a file using Wget or some other command from a Windows machine? Say I want to download something from https server to C:\ABC\abc.xls Any ideas, Thanks. (4 Replies)
Discussion started by: dohko
4 Replies

9. Solaris

HTTP error while downloading solaris patches using wget

Hello, I am getting a HTTP error while downloading solaris patches using wget. 'Downloading unsigned patch 113096-03. --2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40 Connecting to... (5 Replies)
Discussion started by: sunny_a_j
5 Replies

10. Shell Programming and Scripting

Downloading info from website to database

Hi guys! I created a database using mysql in bash now i would like to download weather info from the data.(temp, date and time)...and just store this in the database to display after every 3 hours or so... i have tried to get the website using wget and now dont exactly now how to go from here... (0 Replies)
Discussion started by: vadharah
0 Replies
Login or Register to Ask a Question