If you use Perl to do it, it would require a simple while() loop and a call to LWP module to get the HTTP headers. It is likely more reliable as it does not need to parse the "messy" output of wget or any other similar command line tools.