It's not really a choice between a matrix switch and a mixer... they don't serve the same purpose.
You want multiple outputs so that each output is piping out different audio. Output 1: Desktop speakers. Output 2: Headphones. Etc. The matrix functionality comes from the fact that you can send your SNES audio to your speakers OR your headphones OR both at once. Maybe SNES on speakers and Genesis on headphones OR reverse them. Whatever you want.
If you have a one-output switch, you want a mixer with multiple units so that you can connect your multiple devices and not have to keep unplugging/plugging each time you need to use one. Back to the SNES/Gen example: your SNES and Genesis are both connected to the single output (desktop speakers) via the mixer. If you have both consoles on, you will hear audio from both through the speakers unless you turn down the volume knob on the mixer for one of the systems.
Given all you've said, you are in fact looking for an audio matrix switch. I myself have a full video and audio matrix switch, but that's more because I have multiple monitors. Yes, it is the absolute best thing ever.