Shell Programming and Scripting

Hello,
I have a complex problem. I have a file in which words have been joined together:
Theboy ranslowly
I want to be able to correctly split the words using a lookup file in which all the words occur:
the
boy
ran
slowly
slow
put
child
ly
The lookup file which is meant for look up for splitting the words is huge and serves as a look up to correctly segment the input file which has “runon” words. The input file could also be very large.
It could also contain upto three to four words concatenated together.
I have 2 requirements:
1. Only the largest string should be used for splitting. Thus given that both slow and ly occur, I do not want the split to be :
the boy ran slow ly
But rather
the boy ran slowly.

2. In case a word is not found in the master list, all other largest strings should be spewed out
E.g. Assume that boy is not in the lookup file, I would still want the cut to be:
The boy ran slowly
i.e.” boy” is flagged as residue and tagged as such if possible.
I have tried to write a program which does this (both in Perl as well as in AWK, but it just fails and spews out incorrect forms, especially when I try to meet condition 1.
I am still a tyro at PERL and AWK since all my experience has been in C for the past 20 years and I am fascinated by AWK as well as PERL because of their speed and elegance.
Help would be most appreciated and gratefully acknowledged to help me learn a new skill. A commented code would be a great learning experience, if someone could have the patience to do that for me as well as for others like me who are learners,
Manythanks, (Many thanks)

GIMLEY

try:
**** first, make a sort on your lookup file based on length of word, largest to smalles, like:
slowly
child
slow
put
the
boy
ran
ly

then run it.

Code:

 awk 'NR==FNR{a[NR]=$1;b[$1]=1;x=NR}
NR>FNR{IGNORECASE=1;{for (j=1;j<=x;j++){for(i=1;i<=NF;i++) if(length($i)>length(a[j]) && !($i in b) && $i~a[j] && $i!=a[j])
{gsub(a[j]," "a[j]" ",$0)}}}}END{$1=toupper(substr($1,1,1))substr($1,2);print}' lookup raw
The boy ran slowly

Hello,
Many thanks for the prompt reply. It does work
But there are two issues:
A small glitch instead of handling the largest string:
slowly
it takes slow and ly and breaks up the catted sentence as :
the boy ran slow ly.

Residual data at thend is identified correctly but when the unknown word is in the middle, things seem to go wrong:

When I gave the string
theboyranthroughslowly
The output was:
the boy ran throughs low ly
Since low is not in the small lookup file, I am perplexed how it was generated.
Many thanks once more for the script and I hope these two bugs are soluble.

Gimley
This is precisely the problem, I have not been able to solve apart from the residue issue.
An add on to the awk script to handle this would be of great help.

Hi Gimley,

I have modified the code as the above,please try it,see how it is?

Best,

Y

Hi Yinyuemi,
Many thanks for the timely help. The residue problem seems to be sorted with the new code. However the largest string issue still remains.
I used the code which you had posted (reproduced below)

Code:

awk 'NR==FNR{a[NR]=$1;b[$1]=1;x=NR}
NR>FNR{IGNORECASE=1;{for (j=1;j<=x;j++){for(i=1;i<=NF;i++) if(length($i)>length(a[j]) && !($i in b) && $i~a[j] && $i!=a[j])
{gsub(a[j]," "a[j]" ",$0)}}}}END{$1=toupper(substr($1,1,1))substr($1,2);print}' lookup raw

And I still get
The boy ran through slow ly
for
theboyranthroughslowly

Sorry to hassle you, but the largest string split is vital for the dictionary work I am doing.
Many thanks once again and hoping to read you,
Best regards,
Gimley

It seems to work on my computer

Code:

cat lookup
slowly
child
slow
put
the 
boy 
ran
ly
 
cat raw
theboyranthroughslowly
 
awk 'NR==FNR{a[NR]=$1;b[$1]=1;x=NR}
NR>FNR{IGNORECASE=1;{for (j=1;j<=x;j++){for(i=1;i<=NF;i++) if(length($i)>length(a[j]) && !($i in b) && $i~a[j] && $i!=a[j])
{gsub(a[j]," "a[j]" ",$0)}}}}END{$1=toupper(substr($1,1,1))substr($1,2);print}' lookup raw
The boy ran through slowly

Sorry I did not see the sor routine and just blindly copied the code. I got the idea. it grabs the largest string first.
Two questions:
1. Any awk command to handle the largest to smallest sort.
2. I gave three sentences: it worked only on the first. How do I get the code to loop through the whole input file.
Sorry for such stupid questions but I am still learning awk programming,
Many thanks and excuses for not reading through your mail,

Best regards,

Gimley