I just tried that out of curiosity and now realize I was slightly wrong.

In most shells, test is a builtin which supports all operators the shell does.

Code:

$ test -N /etc/passwd
$ echo $?
0
$ whereis test
test: /usr/bin/test /usr/share/man/man1/test.1.bz2 /usr/share/man/man1p/test.1p.bz2
$ /usr/bin/test -N /etc/passwd
/usr/bin/test: extra argument '-N'
$

...so if your shell has it as a builtin you are set.

I definitely reccommend against injection rejection which leaves you wide open to things you don't know about and can react badly to valid things you still didn't expect. (What if a filename contains a literal backtick?) Better to not leave the door open in the first place and use something which doesn't allow for shell interpretation.