Sponsored Content
Top Forums Shell Programming and Scripting How to extract url from html page? Post 302463336 by kurumi on Sunday 17th of October 2010 02:42:07 AM
Old 10-17-2010
thanks scrutinizer for following up on this. you are almost there. Smilie
Code:
$ awk -F'>' '/[ \t]href=/{N=split($1,F,"\""); i=1; while(F[i++]!~/[ \t]href=/); print F[i],$NF}' RS='<' index.html |sort
/advanced_search?hl=en Advanced Search
http://mail.google.com/mail/?hl=en&tab=wm Gmail
http://maps.google.com/maps?hl=en&tab=wl Maps
http://news.google.com/nwshp?hl=en&tab=wn News
https://www.google.com/accounts/Login?hl=en&continue=http://209.85.132.104/ Sign in
http://video.google.com/?hl=en&tab=wv Videos
http://www.google.com/imghp?hl=en&tab=wi Images
http://www.google.com/intl/en/options/
http://www.google.com/ncr Go to Google.com
http://www.google.com/prdhp?hl=en&tab=wf Shopping
/intl/en/about.html About Google
/intl/en/ads/ Advertising&nbsp;Programs
/intl/en/privacy.html Privacy
/language_tools?hl=en Language Tools
/preferences?hl=en Settings
/services/ Business Solutions
/url?sa=p&pref=ig&pval=3&q=http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg iGoogle

$ ruby test.rb index.html |sort
-->/advanced_search?hl=en, Advanced Search
-->http://mail.google.com/mail/?hl=en&tab=wm, Gmail
-->http://maps.google.com/maps?hl=en&tab=wl, Maps
-->http://news.google.com/nwshp?hl=en&tab=wn, News
-->https://www.google.com/accounts/Login?hl=en&continue=http://209.85.132.104/, Sign in
-->http://video.google.com/?hl=en&tab=wv, Videos
-->http://www.google.com/imghp?hl=en&tab=wi, Images
-->http://www.google.com/intl/en/options/, more »
-->http://www.google.com/ncr, Go to Google.com
-->http://www.google.com/prdhp?hl=en&tab=wf, Shopping
-->/intl/en/about.html, About Google
-->/intl/en/ads/, Advertising*Programs
-->/intl/en/privacy.html, Privacy
-->/language_tools?hl=en, Language Tools
-->/preferences?hl=en, Settings
-->/services/, Business Solutions
-->/url?sa=p&pref=ig&pval=3&q=http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg, iGoogle

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to get the page size (of a url) using wget

Hi , I am trying to get page size of a url(e.g.,www.example.com) using wget command.Any thoughts what are the parameters i need to send with wget to get the size alone? Regards, Raj (1 Reply)
Discussion started by: rajbal
1 Replies

2. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies

3. Solaris

Accessing a HTML page

Hi All, In our unix server we have an apache web server running. I can access the default apache web page from my windows machine. Now, I want to create my own webpage. Therefore I created webpage at /export/home/myname/test.html file. Where do I need to place this file and what do I need... (0 Replies)
Discussion started by: pkm_oec
0 Replies

4. Web Development

findstr in html page

I am planning to create an html page that will count number of connected ports, challenge for me is how to put it in a page. Thanks! (1 Reply)
Discussion started by: webmunkey23
1 Replies

5. UNIX for Dummies Questions & Answers

Publishing HTML Page

Hi All, Thanks for reading. I am not sure if I am asking this in the correct group. But here it goes: There is a shell script which does some system checks and creates an html file called system_summary.html on my Red Hat machine say in /reports directory every hour. Now I want to view it... (1 Reply)
Discussion started by: deepakgang
1 Replies

6. Red Hat

Publishing HTML Page

Hi All, Thanks for reading. I am not sure if I am asking this in the correct group. But here it goes: There is a shell script which does some system checks and creates an html file called system_summary.html on my Red Hat machine say in /reports directory every hour. Now I want to view it... (6 Replies)
Discussion started by: deepakgang
6 Replies

7. Shell Programming and Scripting

Extracting anchor text and its URL from HTML files in BASH

Hi All, I have some HTML files and my requirement is to extract all the anchor text words from the HTML files along with their URLs and store the result in a separate text file separated by space. For example, <a href="/kid/stay_healthy/">Staying Healthy</a> which has /kid/stay_healthy/ as... (3 Replies)
Discussion started by: shoaibjameel123
3 Replies

8. Shell Programming and Scripting

URL/HTML encoding

Hey guys, looking for a way to encode a string into URL and HTML in a bash script that I'm making to encode strings in various different digests etc. Can't find anything on it anywhere else on the forums. Any help much appreciated, still very new to bash and programming etc. (4 Replies)
Discussion started by: 3therk1ll
4 Replies

9. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be... (1 Reply)
Discussion started by: Paul Walker
1 Replies

10. Post Here to Contact Site Administrators and Moderators

Page Not Found error while parsing url

Hi I just tried to post following link while answering, its not parsing properly, just try on your browser Tried to paste while answering : https://www.unix.com/302873559-post2.htmlNot operator is not coming with HTML/PHP tags so attaching file (2 Replies)
Discussion started by: Akshay Hegde
2 Replies
wml::std::lang(3)						     EN Tools							 wml::std::lang(3)

NAME
wml::std::lang - Multi-Lingual Support SYNOPSIS
#use wml::std::lang <lang:new id=xx [short]> <lang:area> (xx) ... (yy) ... </lang:area> <lang:set-wildcard ...> <lang:star: ...*..> <lang:star:href: index.*.html|index.html> <lang:star:slice: index.*.html> <lang:xx>...</lang:xx> <lang:xx: ...> <xx>...</xx> <xx: ...> <lang:current> <lang:list> DESCRIPTION
This include file provides high-level multi-lingual support via Slices. Its purpose is to define the slices ``"LANG_XX"'' according to the multi-lingual selection tags. The general intend of this slice-based approach is to use the defined slices in Pass 9 (Slice) via WMLs -o option. A typical shebang-line example for the use with a webserver's content negotiation feature is: #!wml -o (ALL-LANG_*)+LANG_EN:index.html.en -o (ALL-LANG_*)+LANG_DE:index.html.de Since WML 1.7.0, the "<lang:star:slice:>" tag is an alternative to this shebang-line. Before you can use a language, you have to define the corresponding tags via "<lang:new>". For instance when you want to use the languages english and german, use: <lang:new id=en> <lang:new id=de> Then the following tags are defined: <lang:en>...</lang:en> <lang:de>...</lang:de> <lang:en: ...> <lang:de: ...> i.e. for both languages a container tag and a simple tag is defined. The container tag is more readable while the simple tag is nicer for short variants. When the names "lang:xx" are still to large for you, you can use the "short" attribute to "<lang:new>" <lang:new id=en short> <lang:new id=de short> when then leads to the defintion of the shortcut variants: <en>...</en> <de>...</de> <en: ...> <de: ...> Additionally you always have the "<lang:area>"..."</lang:area>" container tag available which provides an alternative way of selecting the language in its body. It automatically surrounds the data between `"(xx)"' start tags with the corresponding "LANG_XX" slice. The following are equal: <lang:xx: Foo><lang:yy Bar> <lang:xx>Foo</lang:xx><lang:yy>Bar</lang:yy> <lang:area>(xx)Foo(yy)Bar</lang:area> Because these three lines internally get expanded to [LANG_XX:Foo:][LANG_YY:Bar:] [LANG_XX:Foo:][LANG_YY:Bar:] [LANG_XX:Foo:][LANG_YY:Bar:] There is one additional special tag: "<lang:star:>". This tag expands its attribute line like the "<lang:xx:>" tags but multiple times. Actually as much as defined languages exists ("<lang:new>"!). And in each expansion the asterisks (=stars) in the data gets replaced by the language identifier. Is is sometimes convenient to use another wildcard, e.g. when defining navigation bars. The "<lang:set-wildcard>" tag does the job. The attribute becomes the wildcard used in future substitutions. Without attribute, the default value is restored. You may specify any regular expression, and do not forget to escape special characters (the astersisk is in fact ``\*''). <lang:set-wildcard "%"> <lang:star: index.%.html> <lang:set-wildcard> There is a more specialized variant named "<lang:star:href:>" which is similar to "<lang:star:>" but treats its attribute value as a URL part and tries to check if it already exists. If it doesn't exist the tag expands the value without the star or an alternative value which can be appended with ``|alt-value''. The "<lang:star:slice:>" is another variant to help writing multi-lingual files quickly. It must come after all occurences of "<lang:new>" tags. <lang:star:slice: index.html.*> The `%BASE' form is recognized (see wml(1)) and an empty argument is equivalent to the string `"%BASE.*.html"'. But note that the use of this tag instead of the WML shebang line prevents WMk from doing its job, because WMk can not guess output filenames in this case. For complex multi-lingual documents, you may want to know in which language text is currently processed. This is achieved with <lang:current> which always returns current language (as defined in "<lang:new>" or an empty string when outside of any language portion. The macro <lang:list> prints the newline separated list of defined languages. EXAMPLE
The following is an example of a webpage "index.wml" with a multi-lingual header and hyperlink: #use wml::std::lang #use wml::std::href <lang:new id=en short> <lang:new id=de short> <lang:star:slice: index.html.*> <h1><en: Welcome><de: Willkommen></h1> <href name="The Hyperlink" url="<lang:star: index.*.html>"> <href name="The Hyperlink" url="<lang:star:href: index2.*.html|index2.html>"> When processed via $ wml index.wml The following two output files are generated (assuming that index2.html and only index2.de.html exists): index.html.en: <h1>Welcome</h1> <a href="index.en.html">The Hyperlink</a> <a href="index2.html">The Hyperlink</a> index.html.de: <h1>Willkommen</h1> <a href="index.de.html">The Hyperlink</a> <a href="index2.de.html">The Hyperlink</a> AUTHOR
Ralf S. Engelschall rse@engelschall.com www.engelschall.com Denis Barbier barbier@engelschall.com REQUIRES
Internal: P1, P2, P6, P9 External: -- SEE ALSO
slice(1) EN Tools 2014-04-16 wml::std::lang(3)
All times are GMT -4. The time now is 09:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy