Do I understand that you removed 100,000 files or more? If so "rm *" would not work. A whole bunch of "rm pattern*" could do it. So could "ls | xargs rm". I would go with "rm -rf /directory" just as you did. It's the least work for me.
Which is easiest on the system? The difference will be minor, it's the 100,000 unlink() system calls that take forever. But "rm -rf /directory" is one process while the other techniques are many processes. That already gives it a edge. What's more, the other techniques are creating processes with long argument lists. The shell will fork() and exec() to create a process. exec() causes a new program to overlay the program that called exec(). But first, the envirinment and the arguments must be saved so that they can be passed to the program.
So you probably saved several seconds!
But seriously, I appaud your curiosity. An understanding of the internal operation of unix can often be very useful. But in this case, anyway yout it, you still had to do 100,000 unlinks.