Sponsored Content
Top Forums Shell Programming and Scripting For loop till the files downloaded Post 302852493 by RudiC on Wednesday 11th of September 2013 04:29:00 PM
Old 09-11-2013
I guess you've got 80 files in your filename.txt. Why don't you create a temp file, appending every file name that wget fails with, and, with a while loop around your above for loop, retry reading that ever decreasing temp file? Like
Code:
cp filename.txt tempfile
while tempfile not empty
  do
  while read FILE
    do wget $FILE || echo $FILE >>tempfile2
    done < tempfile
  cp tempfile2 tempfile
  done

 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

question regarding ftp. Files downloaded are of size Zero.

I need to download some files from a remote server using ftp. I have ftp'd into the site. I then do an mget * to retrieve all of the data files. Everything seems to proceed normally and I am given feedback that the files were downloaded. Now if I go into the DOS Shell or Windows explorer, it list... (5 Replies)
Discussion started by: ralphisnow
5 Replies

2. UNIX for Dummies Questions & Answers

Loop till you find a string in a fine <-- Need Help New to Unix Scripting

Guys - I am new to Unix scripting and am in need for a script that does the following. I have bits and pieces created and tested but i am just having a little difficult time getting it all together. - Loop through till it finds a string in a specific file. Any help is greatly appreciated. ... (1 Reply)
Discussion started by: mrehman
1 Replies

3. Shell Programming and Scripting

How to print lines till till a pattern is matched in loop

Dear All I have a file like this 112534554 446538656 444695656 225696966 226569744 228787874 113536566 443533535 222564552 115464656 225445345 225533234 I want to cut the file into different parts where the first two columns are '11' . The first two columns will be either... (3 Replies)
Discussion started by: anoopvraj
3 Replies

4. Shell Programming and Scripting

how to limit files downloaded by wget

I am trying to download a page and retrieve only wav and mp3 files via wget. the website is: Alarm Sounds | Free Sound Effects | Alarm Sound Clips | Sound Bites my command is : wget -rl 2 -e robots=off -A wav,mp3 http://soundbible.com/tags-alarm.html When not using the -A wav,mp3... (2 Replies)
Discussion started by: Narnie
2 Replies

5. Shell Programming and Scripting

Help with WGET and renaming downloaded files :(

Hi everybody, I would greatly appreciate some expertise in this matter. I am trying find an efficient way to batch download files from a website and rename each file with the url it originated from (from the CLI). (ie. Instead of xyz.zip, the output file would be http://www.abc.com/xyz.zip) A... (10 Replies)
Discussion started by: o0110o
10 Replies

6. Web Development

php files are downloaded

Hello, I have setup Cherokee web server and php 5.2 in Opensolaris zone. Problem is that all .php files are downloaded from web server and not served when I use IP address instead of DNS name in web brovser. Example: test.mydomain.com <-- php works 192.168.0.10/index.php <--... (3 Replies)
Discussion started by: kreno
3 Replies

7. Shell Programming and Scripting

BASH scripting - Preventing wget messed downloaded files

hello. How can I detect within script, that the downloaded file had not a correct size. linux:~ # wget --limit-rate=20k --ignore-length -O /Software_Downloaded/MULTIMEDIA_ADDON/skype-4.1.0.20-suse.i586.rpm ... (6 Replies)
Discussion started by: jcdole
6 Replies

8. Shell Programming and Scripting

Grep with loop till search is done

I need help to put a script where it runs the svn command grep'ing for the ticket# in the comments to see if the ticket was used in the latest commit. so on command line: ./test.sh ticket-1 ticket-2 ticket-3 It should be able to check if ticket-1 is used first and if not then check if... (2 Replies)
Discussion started by: iaav
2 Replies

9. Shell Programming and Scripting

While loop till length of line is great enough

I have the following code: # Get the line of stations_info.txt starting with "${xstation1} " and copy it to file temp.txt grep "^${xstation1} " stations_info.txt > temp.txt # Get lat and long of station nl=0 ... (2 Replies)
Discussion started by: claire.a
2 Replies

10. Shell Programming and Scripting

Deleting multiple files off an ftp server once they have been downloaded

Hello, I have a server that I have to ftp files off and they all start SGRD and are followed by 6 numbers. SGRD000001 SGRD000002 SGRD000003 The script I have will run every 10 mins to pick up files as new ones will be coming in all the time and what I want to do is delete the files I have... (7 Replies)
Discussion started by: sph90457
7 Replies
Directory::Scratch(3pm) 				User Contributed Perl Documentation				   Directory::Scratch(3pm)

NAME
Directory::Scratch - Easy-to-use self-cleaning scratch space. SYNOPSIS
When writing test suites for modules that operate on files, it's often inconvenient to correctly create a platform-independent temporary storage space, manipulate files inside it, then clean it up when the test exits. The inconvenience usually results in tests that don't work everwhere, or worse, no tests at all. This module aims to eliminate that problem by making it easy to do things right. Example: use Directory::Scratch; my $temp = Directory::Scratch->new(); my $dir = $temp->mkdir('foo/bar'); my @lines= qw(This is a file with lots of lines); my $file = $temp->touch('foo/bar/baz', @lines); my $fh = openfile($file); print {$fh} "Here is another line. "; close $fh; $temp->delete('foo/bar/baz'); undef $temp; # everything else is removed # Directory::Scratch objects stringify to base $temp->touch('foo'); ok(-e "$temp/foo"); # /tmp/xYz837/foo should exist EXPORT
The first argument to the module is optional, but if specified, it's interperted as the name of the OS whose file naming semantics you want to use with Directory::Scratch. For example, if you choose "Unix", then you can provide paths to Directory::Scratch in UNIX-form ('foo/bar/baz') on any platform. Unix is the default if you don't choose anything explicitly. If you want to use the local platform's flavor (not recommended), specify an empty import list: use Directory::Scratch ''; # use local path flavor Recognized platforms (from File::Spec): Mac UNIX Win32 VMS OS2 The names are case sensitive, since they simply specify which "File::Spec::" module to use when splitting the path. EXAMPLE use Directory::Scratch 'Win32'; my $tmp = Directory::Scratch->new(); $tmp->touch("foo\bar\baz"); # and so on METHODS
The file arguments to these methods are always relative to the temporary directory. If you specify "touch('/etc/passwd')", then a file called "/tmp/whatever/etc/passwd" will be created instead. This means that the program's PWD is ignored (for these methods), and that a leading "/" on the filename is meaningless (and will cause portability problems). Finally, whenever a filename or path is returned, it is a Path::Class object rather than a string containing the filename. Usually, this object will act just like the string, but to be extra-safe, call "$path->stringify" to ensure that you're really getting a string. (Some clever modules try to determine whether a variable is a filename or a filehandle; these modules usually guess wrong when confronted with a "Path::Class" object.) new Creates a new temporary directory (via File::Temp and its defaults). When the object returned by this method goes out of scope, the directory and its contents are removed. my $temp = Directory::Scratch->new; my $another = $temp->new(); # will be under $temp # some File::Temp arguments get passed through (may be less portable) my $temp = Directory::Scratch->new( DIR => '/var/tmp', # be specific about where your files go CLEANUP => 0, # turn off automatic cleanup TEMPLATE => 'ScratchDirXXXX', # specify a template for the dirname ); If "DIR", "CLEANUP", or "TEMPLATE" are omitted, reasonable defaults are selected. "CLEANUP" is on by default, and "DIR" is set to "File::Spec-"tmpdir>; child Creates a new "Directory::Scratch" directory inside the current "base", copying TEMPLATE and CLEANUP options from the current instance. Returns a "Directory::Scratch" object. base Returns the full path of the temporary directory, as a Path::Class object. platform([$platform]) Returns the name of the platform that the filenames are being interperted as (i.e., "Win32" means that this module expects paths like "fooar", whereas "UNIX" means it expects "/foo/bar"). If $platform is sepcified, the platform is changed to the passed value. (Overrides the platform specified at module "use" time, for this instance only, not every "Directory::Scratch" object.) touch($filename, [@lines]) Creates a file named $filename, optionally containing the elements of @lines separated by the output record separator "$". The Path::Class object representing the new file is returned if the operation is successful, an exception is thrown otherwise. create_tree(%tree) Creates a file for every key/value pair if the hash, using the key as the filename and the value as the contents. If the value is an arrayref, the array is used as the optional @lines argument to "touch". If the value is a reference to "undef", then a directory is created instead of a file. Example: %tree = ( 'foo' => 'this is foo', 'bar/baz' => 'this is baz inside bar', 'lines' => [qw|this file contains 5 lines|], 'dir' => undef, ); $tmp->create_tree(%tree); In this case, two directories are created, "dir" and "bar"; and three files are created, "foo", "baz" (inside "bar"), and "lines". "foo" and "baz" contain a single line, while "lines" contains 5 lines. openfile($filename) Opens $filename for writing and reading ("+>"), and returns the filehandle. If $filename already exists, it will be truncated. It's up to you to take care of flushing/closing. In list context, returns both the filehandle and the filename "($fh, $path)". mkdir($directory) Creates a directory (and its parents, if necessary) inside the temporary directory and returns its name. Any leading "/" on the directory name is ignored; all directories are created inside the "base". The full path of this directory is returned if the operation is successful, otherwise an exception is thrown. tempfile([$path]) Returns an empty filehandle + filename in $path. If $path is omitted, the base directory is assumed. See File::Temp::tempfile. my($fh,$name) = $scratch->tempfile; exists($file) Returns the file's real (system) path if $file exists, undefined otherwise. Example: my $path = $tmp->exists($file); if(defined $path){ say "Looks like you have a file at $path!"; open(my $fh, '>>', $path) or die $!; print {$fh} "add another line "; close $fh or die $!; } else { say "No file called $file." } stat($file) Stats $file. In list context, returns the list returned by the "stat" builtin. In scalar context, returns a "File::stat" object. read($file) Returns the contents of $file. In array context, returns a list of chompped lines. In scalar context, returns the raw octets of the file (with any trailing newline removed). If you wrote the file with $, set, you'll want to set $/ to $, when reading the file back in: local $, = '!'; $tmp->touch('foo', qw{foo bar baz}); # writes "foo!bar!baz!" to disk scalar $tmp->read('foo') # returns "foo!bar!baz!" $tmp->read('foo') # returns ("foo!bar!baz!") local $/ = '!'; $tmp->read('foo') # returns ("foo", "bar", "baz") write($file, @lines) Replaces the contents of file with @lines. Each line will be ended with a " ", or $, if it is defined. The file will be created if necessary. append($file, @lines) Appends @lines to $file, as per "write". randfile() Generates a file with random string data in it. If String::Random is available, it will be used to generate the file's data. Takes 0, 1, or 2 arguments - default size, max size, or size range. A max size of 0 will cause an exception to be thrown. Examples: my $file = $temp->randfile(); # size is between 1024 and 131072 my $file = $temp->randfile( 4192 ); # size is below 4129 my $file = $temp->randfile( 1000000, 4000000 ); link($from, $to) Symlinks a file in the temporary directory to another file in the temporary directory. Note: symlinks are not supported on Win32. Portable code must not use this method. (The method will "croak" if it won't work.) ls([$path]) Returns a list (in no particular order) of all files below $path. If $path is omitted, the root is assumed. Note that directories are not returned. If $path does not exist, an exception is thrown. delete($path) Deletes the named file or directory at $path. If the path is removed successfully, the method returns true. Otherwise, an exception is thrown. (Note: delete means "unlink" for a file and "rmdir" for a directory. "delete"-ing an unempty directory is an error.) chmod($octal_permissions, @files) Sets the permissions $octal_permissions on @files, returning the number of files successfully changed. Note that '0644' is "--w----r-T", not "-rw-r--r--". You need to pass in "oct('0644')" or a literal 0644 for this method to DWIM. The method is just a passthru to perl's built-in "chmod" function, so see "perldoc -f chmod" for full details. cleanup Forces an immediate cleanup of the current object's directory. See File::Path's rmtree(). It is not safe to use the object after this method is called. ENVIRONMENT
If the "PERL_DIRECTORYSCRATCH_CLEANUP" variable is set to 0, automatic cleanup will be suppressed. PATCHES
Commentary, patches, etc. are most welcome. If you send a patch, try patching the git version available from: <git://git.jrock.us/Directory-Scratch>. You can check out a copy by running: git clone git://git.jrock.us/Directory-Scratch Then you can use git to commit changes and then e-mail me a patch, or you can publish the repository and ask me to pull the changes. More information about git is available from <http://git.or.cz/> SEE ALSO
L<File::Temp> L<File::Path> L<File::Spec> L<Path::Class> BUGS
Please report any bugs or feature through the web interface at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Directory-Scratch>. ACKNOWLEDGEMENTS
Thanks to Al Tobey (TOBEYA) for some excellent patches, notably: "child" Random Files ("randfile") "tempfile" "openfile" COPYRIGHT &; LICENSE Copyright 2006 Jonathan Rockway, all rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.10.0 2008-06-09 Directory::Scratch(3pm)
All times are GMT -4. The time now is 07:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy