Standard POSIX BREs and EREs perform greedy matches. And, awk uses standard POSIX EREs. Greedy means that .* in the ERE <ref>.*</ref> matches the longest string of characters it can find that starts with <ref> and ends with </ref>. Creating an ERE that matches a string starting with a specific string and ending with another (longer than one character) string that doesn't contain the terminating (longer than one character) string is somewhere between hard and impossible depending on the terminating string. Shell parameter expansions provide ways to perform greedy expansions (${var##pattern} and ${var%%pattern}) and non-greedy expansions (${var#pattern} and ${var%pattern}). You may also be able to find something in gawk to tell it to use a non-greedy RE match.

You're right about my code missing references. I was just looking for <ref> when I should have also been looking for <ref name="string">. Changing my code to:

Code:

awk -F'<ref[^>]*>' '
{	for(i=2; i<=NF; i++)
		print substr($i, 1, match($i, "</ref>") - 1)
}' file.txt

will take care of that, but it does still depend on finding the opening and closing ref tags on the same line in your input files. Note that your <ref[^>]{0,1000}+> (which is one or more occurrences of zero to 1000 non-> characters between <ref and >) can be much more concisely written as <ref[^>]*> (which is zero or more occurrences of non-> characters between <ref and >).

The i=2 in the for loop should eliminate the blank lines problem.

If the above awk script doesn't work using gawk (which doesn't care much about line length limits), it must mean that some of your files do have the opening and closing ref tags on different lines. If that is your problem, we can try a shell script to do the parsing, but note that some of the references printed will contain <newline> characters in that case.

The grep commands that RudiC suggested also depend on the opening and closing ref tags being on the same line.