But won't a slow card slow things down too? Once the buffer is full if the camera can't write the data to the card quickly enough things will slow down as you are then limited to the card write speed.

I haven't experienced this with my 7D but I certainly did on my 450D if I used a basic cheapo SD card rather than the Ultra 2 I had at the time.