if file name is only '!', you can remove it by " rm !"
but i feel its not the case.
its all files using "ls -b" it will print non printable characters also in ascii format,
use man ascii to see character belong to that value.
then use rm command to remove it "rm !(ctrl V + that key)
eg : # ls -lbA shows me
-rw-r--r-- 1 root other 0 Mar 7 14:55 !\177 ( ! + delet key)
rm !(CTRL-V) (press delete key)
will appear as
rm !^?
credit goes to some one else