How to extract url from html page?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to extract url from html page?
# 29  
Old 10-17-2010
Inner text?

You mean the text inside the URL used for display?
# 30  
Old 10-17-2010
yes.
Code:
<a href="http://www.blah.com" ....  > Inner text </a>

# 31  
Old 10-17-2010
This is easy with Perl regex. We run this type of PHP code using Perl regex daily.
# 32  
Old 10-17-2010
Quote:
Originally Posted by Neo
This is easy with Perl regex. We run this type of PHP code using Perl regex daily.
heh, Its easy when the regex is already done for us. Smilie
Anyway, if you happen to have it, please post it. I very much like to take a look. thanks in advance.
# 33  
Old 10-17-2010
Quote:
Originally Posted by kurumi
heh, Its easy when the regex is already done for us. Smilie
Anyway, if you happen to have it, please post it. I very much like to take a look. thanks in advance.
Hahaha... thanks kurumi, for helping me make a point during this thread.

Yes, regex can be complex, and I have found the online, web-based tools are not very good compared to the fat client ones, like Regex Buddy, that run natively on XP and Wine/Linux.

If you download Regex Buddy (not sure if there is a free version) you can easily find a number of canned complex regex that can extract all the info in URLs, since this is a common task.

Then, if you have some test cases, like cutting-and-pasting an HTML page, like the Google page used in this thread into the tool, you can easily debug, step-by-step.

So putting together a complex regex with the proper tool can be done in a matter of minutes.

I agree that regex can be complex, but just because it is complex, we should not say "it cannot be done only with regex", because I assure you it can.

... and I leave it up to you to do it. However, you may need a proper regex tool (not the simple online ones), and if you have a proper regex tool, you will save (you and your employer) a lot of time and money, because the price of a good regex tool (like Regex Buddy, under $40) can easily save a sys admin or programmer hours of work.

As I mentioned, it is cool to see a number of other solutions, but this is an easy regex problem, with a proper regex tool (complex, without a proper tool). Go get one Smilie
# 34  
Old 10-17-2010
Hi.

For URL extraction, I prefer lynx as noted in https://www.unix.com/shell-programmin...using-sed.html ... cheers, drl
This User Gave Thanks to drl For This Post:
# 35  
Old 10-17-2010
Haha, I tend to lean towards regex and string manipulation too if the problem statement is simple. Just that by chance, the requirement by OP requires the inner text as well, hence my solution using n HTML parser.
There are lots of sayings that goes "HTML is not regular, so its difficult to parse HTML" such as Coding Horror: Parsing Html The Cthulhu Way for example.
I tend to agree, partly at least, since i know that if my requirement is simple, a regex will do. If not, then use a parser.

Thanks for introducing Regex Buddy. Its good to know that it can help with constructing regex, debugging etc..I will take a look at it someday. Smilie

On my side, if there is anything too complicated (wrt HTML parsing), i would still go for a HTML parser, its free anyway Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Page Not Found error while parsing url

Hi I just tried to post following link while answering, its not parsing properly, just try on your browser Tried to paste while answering : https://www.unix.com/302873559-post2.htmlNot operator is not coming with HTML/PHP tags so attaching file (2 Replies)
Discussion started by: Akshay Hegde
2 Replies

2. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be... (1 Reply)
Discussion started by: Paul Walker
1 Replies

3. Shell Programming and Scripting

URL/HTML encoding

Hey guys, looking for a way to encode a string into URL and HTML in a bash script that I'm making to encode strings in various different digests etc. Can't find anything on it anywhere else on the forums. Any help much appreciated, still very new to bash and programming etc. (4 Replies)
Discussion started by: 3therk1ll
4 Replies

4. Shell Programming and Scripting

Extracting anchor text and its URL from HTML files in BASH

Hi All, I have some HTML files and my requirement is to extract all the anchor text words from the HTML files along with their URLs and store the result in a separate text file separated by space. For example, <a href="/kid/stay_healthy/">Staying Healthy</a> which has /kid/stay_healthy/ as... (3 Replies)
Discussion started by: shoaibjameel123
3 Replies

5. Red Hat

Publishing HTML Page

Hi All, Thanks for reading. I am not sure if I am asking this in the correct group. But here it goes: There is a shell script which does some system checks and creates an html file called system_summary.html on my Red Hat machine say in /reports directory every hour. Now I want to view it... (6 Replies)
Discussion started by: deepakgang
6 Replies

6. UNIX for Dummies Questions & Answers

Publishing HTML Page

Hi All, Thanks for reading. I am not sure if I am asking this in the correct group. But here it goes: There is a shell script which does some system checks and creates an html file called system_summary.html on my Red Hat machine say in /reports directory every hour. Now I want to view it... (1 Reply)
Discussion started by: deepakgang
1 Replies

7. Web Development

findstr in html page

I am planning to create an html page that will count number of connected ports, challenge for me is how to put it in a page. Thanks! (1 Reply)
Discussion started by: webmunkey23
1 Replies

8. Solaris

Accessing a HTML page

Hi All, In our unix server we have an apache web server running. I can access the default apache web page from my windows machine. Now, I want to create my own webpage. Therefore I created webpage at /export/home/myname/test.html file. Where do I need to place this file and what do I need... (0 Replies)
Discussion started by: pkm_oec
0 Replies

9. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies

10. Shell Programming and Scripting

How to get the page size (of a url) using wget

Hi , I am trying to get page size of a url(e.g.,www.example.com) using wget command.Any thoughts what are the parameters i need to send with wget to get the size alone? Regards, Raj (1 Reply)
Discussion started by: rajbal
1 Replies
Login or Register to Ask a Question