Yes, that's the deciding factor.
It's rather like buying a Ferarri whose top speed is 300km/h, when the majority of your driving is done at 60km/h.
I've seen some of your shots taken at long lengths. You can certainly diffuse the background with a shorter length, and you don't need to be a long way away.
I've used my 300/2.8L IS for a few portraits, but in all honesty, when I've shot with it, portraiture hasn't been the objective; and when I do shoot portraits, it never comes along. The 85 gets used far more than any other lens for portraits, followed by the 35/1.4L and then the 135/2L.
I'd love to use the 135/2L more, but it requires more working distance, and typically I don't have that if I'm doing indoor shoots. I did use it for an outdoor portrait shoot at Berrima a few months ago, though, but I still found myself shooting tightly, as that tends to be my style.
I know of someone who uses a 200/2L IS for portraits, and the images look great, but you really do need a fair amount of distance if you want full-length shots.
Indeed -- a fair point, and I'm no different. Mind you, it's been several years since I bought a lens. :-)
I have seven lenses, of which two are zooms. I have overlap at the 35mm end of my 16-35/2.8L II (35/1.4L), and my 70-200/2.8L IS covers my 85/1.2L II, 135/2L and 180/3.5L Macro, although that lens doesn't get used for portraits. I never bothered with a 24-70/2.8L, as I just wouldn't find it useful for most of what I do.