Read this link first: Removing "Strange" files

You could try something like what they suggest, like this:
find . -name '*[^][a-zA-Z0-9#%~:,._+-]*' -print0 | xargs -0 rm

It'll find all files with those special characters and the print them, and the xargs command will remove them all. You probably want to try something like this without the xargs part first and just view the list of files to make sure you're not gonna delete anything you didn't want to!

You could also just backslash the special characters so they lose their special meaning, like:

ls | grep "filewith\"quotes"