The replay system is actually pretty sick with you being able to watch every match of every player, but the UI implementation has always been terrible.
I actually wouldn't be surprised if some of the CFN problems stem from the large amount of replay data.
I've thought about this, but given that replays and exp points/league points seem to make it to the server even after an initial failure to connect, I think there's separate services for matchmaking, replay storage and points
in a lobby I assume the spectators are seeing a streamed replay. so at least there's need to be a server immediately available for the replay in a lobby. the aggression meter may be based on replay data too. by the same token, replay data does not seem to need any post processing so the server just gives spectators the same data it received?
it is possible that SFV does a little more behind the scenes than SF4 did thus making it require more infrastructure, but from where I am sitting, it doesn't look an order of magnitude more complex than what SF4 did
I think the biggest issue is when the matchmaking servers do as a proxy between people who cannot get a connection, but this is so laggy as to be useless, and in my experience, really really rare. maybe it is there only to make sure that any two players can have a match together no matter what. that may make CFN really work more and put a strain on the servers, but again, it is so rare and undesirable that I cannot imagine it being a big source of problems