We were able to get it unmounted and have two possible theories why it wouldn't unmount.

1) The listener was still running. Even though its process was tied to another filesystem it may have been holding up something in memory preventing this to be unmounted.

2) There was a symbolic link in one of the other directories to the export directory. Since we unmounted the other directory first it couldn't follow back the link and unmount.

Either way it was strange that fuser did not pick it up.