All I see in the /usr/lib you provided, is an empty directory.

Sorry about the RPM command, I havent acutaly used that often - even while I was using RH based systems.

Code:

rpm --whatprovides libGLU -q smpeg

Now, I'm just wondering wether AIX still has yum.
Again, because you're using RPM, I cant get my mind of RH based systems - for which I expect either yum (old) or dnf (new) command to be available.

Regardless, lets see if any of these help - try on after one, and check wether it works in between.

Code:

rpm --verify libGLU
rpm --freshen libGLU

If it doesnt work 'by now'... - lets try to reinstall/overwrite

Code:

rpm -ivh libGLU

If AIX has 'yum' (the 'original' of the previously suggested 'dnf'), try this:

Code:

yum install libGLU

--> I cant get my mind off the idea that during initial installation of your packages, something must had been going sub-optimal.
Though, AFAIK rpm needs the packages (*.rpm) to be downloaded manualy, did you make sure to download the proper *ppc* variant of the libGLU rpm file?
Alltough, if that was the case, it *should not* had let you install it....

My 2 cents, hth - end of my 'knowledge'.