Shell Programming and Scripting

@polsun, let me make a suggestion, if I may. Since you are learning something new, I suggest that you learn perl instead. Perl can pretty much do what awk can (IMHO) and lots more. It is not difficult to learn, either.

Try O'Reilly's Learning Perl book, to get you started. There are also a lot of tutorials on the web. I am sure this is a better skill to have, just because you can do so much more with Perl.

Good luck!!

- GP

thanks for your advice. I think you are right. If i want to learn, why not learn the best.

Certainly you can do anything in Perl that you could in Awk. This is a feature of any turing-complete language.

But they're not the same at all. Perl is an extremely complex language with a bizarre combination of complex types and weak typing which leaves you always blindly groping trying to figure out whether you've been given an array, list, string, scalar, hash, a reference to any of the above, an array containing references, etc. awk has only two types, strings and maps, so much less confusion.

Perl also has the problem that perl doesn't play too well as a part of something else. It can do anything, sure, but the usual means to do this is to install a perl module to connect with libraries and such. Find a neat and tidy perl script to do something and you may be surprised just how much extra software, perl and other, you need to install to make this "small" script actually work. Python has the same problem. awk solves one class of problems and solves them well without trying to be everything and giving you library-shock.

Having learned Perl instead of awk, I wish I'd learned awk first -- much simpler syntax to solve some of the same problems and a better introduction to a lot of concepts that puzzled me in Perl.

In the end, which is best really depends on what you want to use it for. So learn both.

damn! now you confused me. I agree that AWK is simple. I once tried perl too and I agree with you that its bit more complex. But for some reason perl is more popular. People think AWK is obsolete, but I love it. May be you are right, I should learn both.

awk is specialized. It reads, processes, and writes delimited text but isn't good for much else. Its data structures are flexible but limited. Imagine trying to write a huge database application in it! There's nothing like structure.member. But as a connector between other things, or a small language for processing data, it's nice. I certainly don't think it's obsolete, just not the sort of thing you notice being used since it's not big applications.

Perl's data structures, though eye-twisting and hard to use, are complex enough for the needs of a large database application. And you can do just about anything with it by heaping on enough modules and using enough memory. It's not exactly what I'd call elegant, but it works.

Neither of the perl suggestions nor the ksh/egrep script handle this properly. They are all taking the second column and treating it as a regular expression. Should the text in that column contain any regexp metacharacters, they yield erroneous results. And if the text in the second column forms an invalid regular expression, they will error out.

Place a lone asterisk in the second column and you may see something similar to:

Code:

Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE / at substring.pl line 26, <> line 6.

Worse, a valid regular expression with metacharacters (.*) will silently return incorrect results. Perhaps your real data consists of nothing but alphanumerics, in which case the code provided should be adequate, but that wasn't made clear and your problem statement asks for fixed string matching.


The only sample data you've provided contradicts that statement; some of the lines have only one line.

Also, should the string appearing in the second column be searched for in column 1 of all rows or only rows that follow it? If only those that follow, does that include the current row? You should clarify, because you state above that "acg" only occurs once in column 1, and your desired output shows a 1 for "acg", but "acg" actually occurs twice in the first column.

All suggestions so far scan the entire column an in fact will return 2 where you say it should be 1.

Also, should the string occurring more than once in column1 of a single row be counted as one instance or multiple instances? One of the perl solutions and the ksh/egrep will count "acgacg" as one instance of "acg", while the remaining perl solution would count it as two instances.

Regards,
Alister

@alister - thanks for the detailed reply. Actually it was a typo from my side in the first post - "acg" should count twice. My original file has only alphabets in both columns and no other characters or metacharacters. So, the codes seem to be work fine.

Yes the string of 2nd column should be searched in all the rows of 1st column.

By 30000 lines I meant the maximum number of lines in any column. Yes the 2nd column has far fewer rows/lines than the first.