Sponsored Content
Top Forums Shell Programming and Scripting Extracting urls from curl output Post 302964615 by jozo95 on Saturday 16th of January 2016 01:21:27 PM
Old 01-16-2016
Quote:
Originally Posted by RavinderSingh13
Hello jozo95,

Could you please try following and let me know if this helps you.
Code:
awk '{match($0,/<a href=\"http.*><img/);A=substr($0,RSTART,RLENGTH-4);if(A){print A;A=""}}'  Input_file

Output will be as follows.
Code:
<a href="http://www.bth.se/web/utbildning.nsf/sidor/program?OpenDocument&expand=int">
<a href="https://www.antagning.se/se/triggerlogin?triggerloginurl=/se/mypages">
<a href="http://edu.bth.se/utbildning/utb_sok_resultat.asp?lang=sv&KtTermin=20161&PtStartTermin=20161&vy=hitta">
<a href="https://www.hogskoleprov.nu ">
<a href="http://www.bth.se/web/nyheter.nsf/sidor/8F5E44896F091A3AC1257E9F0045AC7D?OpenDocument">
<a href="http://edu.bth.se/utbildning/utb_sok_resultat.asp?lang=sv&KtTermin=20152&PtStartTermin=20152&vy=hitta&sortering=amne&sortering=installd&grupperingar=1">
<a href="http://www.bth.se/info/ophus.nsf/sidor/oppet-hus-pa-bth">
<a href="http://edu.bth.se/utbildning/utb_sok_resultat.asp?KtTermin=inne&PtStartTermin=inne&KtTyp=SOMM&lang=sv">

Thanks,
R. Singh
Doesnt work :/

I get output as this along the way :

Code:
g">Studievägledning</a></li><li><a href="http://www.bth.se/jobb">Lediga tjänster</a></li></ul></div><div class="footer-section set"><h5><br></h5><ul class="footer-list"><li><a href="http://www.bth.se/web/ombth.nsf/sidor/organisation">Organisation</a></li><li><a href="http://www.bth.se/bib">Bibliotek</a></li><li><a href="http://careergate.bth.se/">BTH Career Gate</a></li><li><a href="http://www.bth.se/for/Sakerhet.nsf/sidor/593cb6bf948640dac1257f1f00365b42?OpenDocument">I händelse av kris</a></li><li><a href=""></a></li><li><a href=""></a></li></ul> </div></span></div></div></div><div class="footer-info"><a class="footer-logo" href="http://www.bth.se">

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting fields from an output 8-)

I am getting a variable as x=2006/01/18 now I have to extract each field from it. Like x1=2006, x2=01 and x3=18. Any idea how? Thanks a lot for help. Thanks CSaha (6 Replies)
Discussion started by: csaha
6 Replies

2. Shell Programming and Scripting

let curl output to stdout AND save to a file

hello hackers. i have a curl process running as cgi directly pushing stdout to the client. but i want to additionally save that stream to a file at the same time. any directions madly welcome. thanks in advance (3 Replies)
Discussion started by: scarfake
3 Replies

3. Shell Programming and Scripting

Pattern matching extracting urls from rss, shell scripts

Hi all, how could i do ? I have a Rss file, i want to extract only the Urls (many) matching http://www.xxx.com/trailers/ from that file and copy into another file. like " <pubDate>Wed, 29 Apr 2009 00:00:00 PST</pubDate> <content:encoded><!Apple - Movie Trailers - The Hangover"><img... (3 Replies)
Discussion started by: BremboloIV
3 Replies

4. Shell Programming and Scripting

script to output curl result as html

hi, new to scripting and would like to know how can I have a script which will curl a few URLs and have the results such as the URLs being curled, dns lookup time, connection time, total time, etc save in a html format in a form of table with column and rows. thank you. (4 Replies)
Discussion started by: squidusr
4 Replies

5. Shell Programming and Scripting

Getting cURL to output verbose to a file

This is about to drive me crazy. What I want to do is simple: output ALL the verbose information from curl to a file I have read the manual, tried several options and searched this forum but no salvation... I'm using curl -k -Q "command" --user user:passwd --ftp-pasv --ftp-ssl -v... (1 Reply)
Discussion started by: caramandi
1 Replies

6. Shell Programming and Scripting

web service call: curl output to xsltproc input

I need to invoke a web service and extract what I need from the response using a combination of curl and xsltproc. However, any file-based parameters that must be supplied to both these programs must be from stdin and not actual files. At least with curl, it seems to think that I am supplying a... (3 Replies)
Discussion started by: webuser
3 Replies

7. Shell Programming and Scripting

ery weird wget/curl output - what should I do?

Hi, I'm trying to write a script to download RedHat's errata digest. It comes in a txt.gz format, and i can get it easily with firefox. HOWEVER: output is VERY strange when donwloading it in a script. It seems I'm getting a file of the same size - but partially text and partly binary! It... (5 Replies)
Discussion started by: jstilby
5 Replies

8. Shell Programming and Scripting

Encapsulating output of CURL and/or WGET

i use curl and wget quite often. i set up alarms on their output. for instance, i would run a "wget" on a url and then search for certain strings within the output given by the "wget". the problem is, i cant get the entire output or response of my wget/curl command to show up correctly in... (3 Replies)
Discussion started by: SkySmart
3 Replies

9. Shell Programming and Scripting

Filter output in curl

Hello guys, I'm writing a little script which sends me sms with my shell script via api of a sms provider. problem is I can't filter my curl output for this site: site url:... (1 Reply)
Discussion started by: genius90
1 Replies

10. Web Development

Filename output in curl

How can I get the name of the default output filename from curl using the argument -O? Using -o one can choose a filename. I want to get the name of the original file, but don't understand how to get it. curl -o filename http://www.website.com curl -O http://www.website.com The... (3 Replies)
Discussion started by: locoroco
3 Replies
CURLINFO_PROTOCOL(3)					     curl_easy_getinfo options					      CURLINFO_PROTOCOL(3)

NAME
CURLINFO_PROTOCOL - get the protocol used in the connection SYNOPSIS
#include <curl/curl.h> CURLcode curl_easy_getinfo(CURL *handle, CURLINFO_PROTOCOL, long *p); DESCRIPTION
Pass a pointer to a long to receive the version used in the last http connection. The returned value will be one of the CURLPROTO_* val- ues. PROTOCOLS
All EXAMPLE
CURL *curl = curl_easy_init(); if(curl) { CURLcode res; curl_easy_setopt(curl, CURLOPT_URL, "http://example.com"); res = curl_easy_perform(curl); if(res == CURLE_OK) { long protocol; curl_easy_getinfo(curl, CURLINFO_PROTOCOL, &protocol); } curl_easy_cleanup(curl); } AVAILABILITY
Added in 7.52.0 RETURN VALUE
Returns CURLE_OK if the option is supported, and CURLE_UNKNOWN_OPTION if not. SEE ALSO
CURLINFO_RESPONSE_CODE(3), curl_easy_getinfo(3), curl_easy_setopt(3), libcurl 7.54.0 November 23, 2016 CURLINFO_PROTOCOL(3)
All times are GMT -4. The time now is 09:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy