Leaving aside the performance problems, there's a bigger fundamental difference between the two. There's things PCI can do and USB just can't. DMA as mentioned above is one of them.

This is because the PCI bus is an extension of the CPU's own hardwired bus, essentially. It has most of the same features as the CPU's own bus, and communication is extremely direct, to the point all hardware on PCI as well as system memory in general ends up inhabiting the same memory map. (More or less. It's gotten more complicated with each revision, but once configured, still handles itself in a manner like this.) Use one address and you're talking to system memory. Use another address and you're talking to video RAM. Use another address and you're talking to your network card's boot ROM. And such.

This turns out to be vital for things your computer absolutely needs to boot, like video. They can include little bits of machine code on the cards which give the BIOS a very basic ability to use them. The BIOS doesn't have to load a driver to see it, the code just kind of appears in memory where it needs to be thanks to the memory map. This is also why mac's require different kinds of video cards than PC's -- different kind of BIOS.

USB is extremely in-direct; it takes tons of messages back and forth to do anything, and wasn't designed as a CPU bus anyway. Things PCI does with one pulse would have to be emulated in software. Rigidly standard things like USB keyboards are handled in BIOS these days, USB media as well (though often badly). But you couldn't replace a PCI video card with a USB one and have your computer boot. The BIOS by nature is too dumb to load a driver, and sees no video as a result, and cannot boot.