Dear all.

I need to extract the title (text between <title> and </title>) of a set of HTML documents.
I've found a command that makes the work of extracting the text, but it does not always work.

It works with the next example:
However, it does not works with a real example:

The last command do not return anything.

I appreciate any comment or suggestion.

First off - dispose of the grep. One regex program is enough on every commandline and grep can do nothing which sed couldn't do too - and better so.

The reason is that sed works linewise - once a new line is read sed forgets (usually - we can overcome that) what it has done on the last line.

The following title would be extracted with your regex:

but the following would fail:

The reason is that sed would read the first line, notice that the search pattern (which specifies the opening AND the closing tag to be there) is not found and move on to the next line. On the next line the same is true so the output is null.

Fortunately there is a device to make sed less forgettable: the line-range.

When we write "s/x/y/" we imply that this rule is used on every line. Still, this is only the abbreviated form of a command, which would include a starting and an end line: "1,5 s/x/y/" would apply the rule only to lines 1-5. Try these with a test file to see the effect.

OK, using line numbers is a bit static, because usually we will not know on which line a certain rule has to be applied - at least not beforehand. But it is also possible to use additional regexes to define the first and the last line of the block where the rule will be applied:

Applying this to your problem, we could use "<title>" as the beginning and "</title>" of the block in question - it is legal to have only one line in a block - and apply your rule to the whole block instead of only one line:

This will print only the lines from the opening to the closing tag. Now we have to "trim" this to get a nice output.

There are three possible types of lines:

1. lines with a "<title>" in them. We want to delete everything up to "<title>" and display the rest

2. lines with a "</title>" in them. We want to keep everything up to "</title>" and dispose of the rest.

3. Lines in between. We want to keep them entirely.

Ok, lets do it - one more thing: it is possible to group commands in regex language like in any programming language. The curly braces "{}" are used to group several commands to a single one:

You might notice that there is no action for the type-3-lines, but in fact there is: its the "p" which prints all the resulting lines (or the parts which survived our trimming respectively) out. The "-n" makes sure no output is done save for explicitly ordered one.

I leave the task to concatenate the resulting lines to you as an exercise. If you still have troubles feel free to ask again.

I hope this helps.

bakunin

It definitely works.
Thank you very much bakunin for your excellent explanation, and for the fast reply.
I really appreciate your help