Sponsored Content
Top Forums Shell Programming and Scripting Downloading of dynamically generated URL using curl and sed Post 302607482 by Corona688 on Wednesday 14th of March 2012 02:21:02 PM
Old 03-14-2012
First thing I'd do is
Code:
wget -k "http://site.com/thing-containing-url.html" -O urlfile.html

the -k will cause it to convert a local link into http://...

Then you can

Code:
URL=`egrep -o 'http://[^?"]*key=[^"]*' urlfile.html`
echo "Got URL $URL"
wget "$URL"

 

10 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

songs downloading using curl

Hi , I need to download songs from website using shell script.I have written script for that. The code is #!/bin/ksh curl "http://www.songs.pk/$1.html" >tmp_page 2>tmp_error grep songid tmp_page|cut -d'"' -f2 > song_links while read link; do echo $link curl "$link" >tmp_song 2>tmp_err... (1 Reply)
Discussion started by: pandu25
1 Replies

2. Shell Programming and Scripting

bash curl escape & in url

I'm running a curl command in bash, but the & in the middle causes the second half of the line to run in the background, here's what I'm trying to do: lat="37.451" lon="-122.18" url="http://ws.geonames.org/findNearestAddress?lat=$lat&lng=$lon" curl -s "$url" I tried escaping the & with \&,... (4 Replies)
Discussion started by: unclecameron
4 Replies

3. Shell Programming and Scripting

Replace a pattern in a file with a generated number using sed or awk

my file has thousands of line but let me show what i want to achieve... here is one line from that file cat fileName.txt (2,'','user3002,user3003','USER_DATA_SINGLE',1,0,0,'BACKUP',2,NULL,0,450,NULL,NULL,'','2011-05-10... (13 Replies)
Discussion started by: vivek d r
13 Replies

4. UNIX for Dummies Questions & Answers

Launch a URL,validate username and password using wget or curl

Hi All, I want to launch "ex: http://gmail.com" from the cmd window and validate the credentials with username and password, is it possible? I have found something like this "wget --http-user=USER' --http-password=PASSWORD http://gmail.com" am new to this and unable to find a solution, i... (0 Replies)
Discussion started by: harsha85
0 Replies

5. UNIX for Dummies Questions & Answers

Read URL data from UNIX without wget,curl,lynx,w3m.

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts. My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled... (0 Replies)
Discussion started by: scott_cog
0 Replies

6. Shell Programming and Scripting

Read URL data from UNIX-CLI without Wget,CURL,w3m,LWP

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts.My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled it.... (12 Replies)
Discussion started by: scott_cog
12 Replies

7. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be... (1 Reply)
Discussion started by: Paul Walker
1 Replies

8. UNIX for Dummies Questions & Answers

When downloading a webpage by curl, what happen?

Hello, when im downloading an webpage from command line (CLI) by curl or wget the target website is loaded like i load it from browser? meaning target server connect to database and render data from mysql? Or only static content is downloaded? (2 Replies)
Discussion started by: postcd
2 Replies

9. Shell Programming and Scripting

Issue in Concatenation/Joining of lines in a dynamically generated file

Hi, I have a file containing many records delimited by pipe (|). Each record should contain 17 columnns/fields. there are some fields having fields less than 17.So i am extracting those records to a file using the below command awk 'BEGIN {FS="|"} NF !=17 {print}' feedfile.txt... (8 Replies)
Discussion started by: TomG
8 Replies

10. Shell Programming and Scripting

Using CURL command with special characters in URL

Hi. I 'm trying to hit a REST api and retrieve a JSON feed, and the URL has special characters in it. Something like: Example Domain The below curl command is failing curl -X GET https://www.example.com/?sample=name&id=1001 saying bad command at id=1001 I am going to use this as part... (3 Replies)
Discussion started by: kumarjt
3 Replies
CURLOPT_PATH_AS_IS(3)					     curl_easy_setopt options					     CURLOPT_PATH_AS_IS(3)

NAME
CURLOPT_PATH_AS_IS - do not handle dot dot sequences SYNOPSIS
#include <curl/curl.h> CURLcode curl_easy_setopt(CURL *handle, CURLOPT_PATH_AS_IS, long leaveit); DESCRIPTION
Set the long leaveit to 1, to explicitly tell libcurl to not alter the given path before passing it on to the server. This instructs libcurl to NOT squash sequences of "/../" or "/./" that may exist in the URL's path part and that is supposed to be removed according to RFC 3986 section 5.2.4. Some server implementations are known to (erroneously) require the dot dot sequences to remain in the path and some clients want to pass these on in order to try out server implementations. By default libcurl will merge such sequences before using the path. DEFAULT
0 PROTOCOLS
All EXAMPLE
CURL *curl = curl_easy_init(); if(curl) { curl_easy_setopt(curl, CURLOPT_URL, "http://example.com/../../etc/password"); curl_easy_setopt(curl, CURLOPT_PATH_AS_IS, 1L); curl_easy_perform(curl); } AVAILABILITY
Aded in 7.42.0 RETURN VALUE
Returns CURLE_OK if the option is supported, and CURLE_UNKNOWN_OPTION if not. SEE ALSO
CURLOPT_STDERR(3), CURLOPT_DEBUGFUNCTION(3), CURLOPT_URL(3), libcurl 7.54.0 February 14, 2016 CURLOPT_PATH_AS_IS(3)
All times are GMT -4. The time now is 02:44 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy