Sponsored Content
Top Forums Shell Programming and Scripting Text substitution & getting file name from url Post 302505614 by texttoolong on Thursday 17th of March 2011 12:22:41 PM
Old 03-17-2011
Text substitution & getting file name from url

hi, sorry if this seems trivial.

i have a file url.txt which consists of a list of urls (it was supposed to be my wget -i file). however, since the server from which i am trying to download uses redirect, wget dows not remeber the filename of ther original url will save to a file name which is unreadable. my solution is to use a script to read the urls line by line and to write a wget script command in to another text file todo.txt which i could then run as a script.

url.txt looks like
Code:
http://serverdoingstrangethings.com/file1.ext
http://serverdoingstrangethings.com/file2.ext

todo.txt should look like
Code:
wget http://serverdoingstrangethings.com/file1.ext -Ofile1.ext
wget http://serverdoingstrangethings.com/file2.ext -Ofile2.ext

(simplified, the actual wget-line will have a few more arguments)

after writing to the todo.txt the line in url.txt should be deleted.

i am a script newbie (i tried a few dos-batch and vba-tingies years ago) and am much to shy to post what I have written so for. my problem lies especialy with getting the file name from the url. could you help - i know i should be reading up on regular expressions and i certainly will, but could you point me in the right direction?

thank you so much,

thomas
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

script to open the specified url in a browser from a text file

Hi All, here i am struc (6 Replies)
Discussion started by: gsp
6 Replies

2. Shell Programming and Scripting

bash curl escape & in url

I'm running a curl command in bash, but the & in the middle causes the second half of the line to run in the background, here's what I'm trying to do: lat="37.451" lon="-122.18" url="http://ws.geonames.org/findNearestAddress?lat=$lat&lng=$lon" curl -s "$url" I tried escaping the & with \&,... (4 Replies)
Discussion started by: unclecameron
4 Replies

3. Shell Programming and Scripting

How to achieve UTF-8 encoding & URL escape in an xml file?

Is there any i can achieve entity escaping, URL escaping & UTF-8 encoded for the xml generated through shell script? #! /bin/bash echo "<path>" >> file.xml for x in `ls filename*` do echo -e "\t<dir>" >> file.xml echo -e "\t\t<file>$x</file>" >> file.xml... (0 Replies)
Discussion started by: vel4ever
0 Replies

4. Shell Programming and Scripting

How to find invalid URL in a text file using shell script?

How to find and remove invalid URLs in a text file using shell script? (1 Reply)
Discussion started by: vel4ever
1 Replies

5. Shell Programming and Scripting

Hit multiple URL from a text file and store result in other test file

Hi, I have a problem where i have to hit multiple URL that are stored in a text file (input.txt) and save their output in different text file (output.txt) somewhat like : cat input.txt http://192.168.21.20:8080/PPUPS/international?NUmber=917875446856... (3 Replies)
Discussion started by: mukulverma2408
3 Replies

6. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

7. Shell Programming and Scripting

Extracting the column containing URL from a text file

I have the file like this: Timestamp URL Text 1331635241000 http://example.com Peoples footage at www.test.com,http://example4.com 1331635231000 http://example1.net crack the nuts http://example6.com 1331635280000 http://example2.net ... (0 Replies)
Discussion started by: csim_mohan
0 Replies

8. Shell Programming and Scripting

Extracting the column containing URL from a text file

I have the file like this: Timestamp URL Text 1331635241000 http://example.com Peoples footage at www.test.com,http://example4.com 1331635231000 http://example1.net crack the nuts http://example6.com 1331635280000 http://example2.net ... (0 Replies)
Discussion started by: csim_mohan
0 Replies

9. Shell Programming and Scripting

Extracting the column containing URL from a text file

I have the file like this: Timestamp URL Text 1331635241000 http://example.com Peoples footage at www.test.com,http://example4.com 1331635231000 http://example1.net crack the nuts http://example6.com 1331635280000 http://example2.net ... (3 Replies)
Discussion started by: csim_mohan
3 Replies

10. Shell Programming and Scripting

Convert NAS URL to Physical Path & visa-versa

sometime the user gives me the Linux NAS path URL which is accessible usings windows explorer like below: This URL translates to the below physical path on Linux host Below is what I wish to achieve: 1. Detect if the path provided is NAS URL starting with "\" or a Physical Linux path... (7 Replies)
Discussion started by: mohtashims
7 Replies
uri_urn(3tcl)					    Tcl Uniform Resource Identifier Management					     uri_urn(3tcl)

__________________________________________________________________________________________________________________________________________________

NAME
uri_urn - URI utilities, URN scheme SYNOPSIS
package require Tcl 8.2 package require uri::urn ?1.1.2? uri::urn::quote url uri::urn::unquote url _________________________________________________________________ DESCRIPTION
This package provides two commands to quote and unquote the disallowed characters for url using the urn scheme, registers the scheme with the package uri, and provides internal helpers which will be automatically used by the commands uri::split and uri::join of package uri to handle urls using the urn scheme. COMMANDS
uri::urn::quote url This command quotes the characters disallowed by the urn scheme (per RFC 2141 sec2.2) in the url and returns the modified url as its result. uri::urn::unquote url This commands performs the reverse of ::uri::urn::quote. It takes an urn url, removes the quoting from all disallowed characters, and returns the modified urls as its result. BUGS, IDEAS, FEEDBACK This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category uri of the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have for either package and/or documentation. KEYWORDS
rfc 2141, uri, url, urn CATEGORY
Networking uri 1.1.2 uri_urn(3tcl)
All times are GMT -4. The time now is 08:18 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy