Shell Programming and Scripting

I used to use Regex Buddy (to create and test regex) for this. They had some stock regex that was quite good for extracting URLs from text. This is really a great tool but sadly only runs on Windows (and on Linux using Wine), as I recall. Using the tool, you create, test and debug complex regex. You can even optimize the regex for performance. Then, you cut-and-paste the regex into your code or application. I highly recommend this tool. I would be running it now, but sadly my XP machine died and I'm running OSX on the desktop and only Android on the go.

Nevertheless, its a good effort.

---------- Post updated at 03:02 AM ---------- Previous update was at 02:58 AM ----------

There are also many online regex sites for creating and testing regex as well. But that said, regex is really not the best tool to parse HTML, unless the requirement is really really really simple

Hi kurumi,

I thought this discussion was about extracting URLs from HTML, not parsing HTML.

There is a difference, you know, between a generic HTML parser, and simply extracting a URL.

URLs can easily be extracted with regex.

The added difficulty here in this case is that besides the url's also the descriptions had to be extracted which in themselves can contain tags with angular bracket, which I used as record separators. This became to complicated with the approach I had chosen, where I wanted to allow the tags to be spread out over multiple lines.. My approach would work fine in many situations, though.

Yes, I understand....

I have seen efficient regex that can easily extract entire URLs, even with tags and more complex, generalized URLs. I don't have them in front of me, so I can't back up my claims at the moment.

Hi Neo
I think extraction/parsing has no real big difference. We are still getting information out of something anyway.

Yup, urls indeed can be easily extracted (or is it? well.... ) . That is, if the requirement is only urls, nothing else. But not so for this particular question/thread since OP wanted to get the inner text as well. As demonstrated by Scrutinizer, its possible using gawk+regex, but there are still some corner cases left out. Anyway, i think OP (where ever he is) will find Scrutinizer's gawk code to be good enough for his purpose.

Different approach

Code:

sed 's|</a>|&\n|g' infile | sed -n '/<a /s|.*<a [^>]*href="\([^"]*\)[^>]*>\(.*\)</a>$|\1 \2|p'