I usually read it as "someone who doesn't see gaming as his main hobby". Not as "someone who doesn't play games, ever."
That's probably what people mean by it most of the time I'd guess.
Curious about how many people here groaning over the term "gamer" have called people a "non gamer" on this forum before.