Am I wrong here? I know this was sort of a stream-of-consciousness post but I needed to get it out. Hoping I don't come off as an asshole.
I'm not sure there's a right/wrong here, its how you feel about the situation, and you're considerate enough to step back and think about it. I'm not sure you can make a rational decision though--there is no way to understand how your life will change, for good and bad. We have two children and it's amazing to watch their friendship evolve and change, and at the same time I'm sure we're in for pain as that transforms as the eldest becomes a teenager. The eldest is also old enough now to watch the younger when my wife and I want to do things, but honestly we rarely take advantage of this--we like doing things as a family. We had never considered having only one child, and in hindsight I'm glad we didn't stop at one, we very much enjoy the dynamic, the contrast in their personalities and development. We decided against more than two children for a few reasons:
1) talking to friends with three+ kids, we became aware of how the notion of personal time for the parent becomes impossible--there are few situations where one parent is excited about the prospect of being responsible for all three children, especially if he/she is the primary caregiver
2) we wanted to have a "reasonable" bound on the end of child-rearing given I was 30 and my wife 33 when we had our second child (my mother had me when she was 21!)
3) babies can be hard, and we weren't sure we wanted roll the dice again on getting a difficult baby sleep-cycle/etc., wise; you get that great routine going at 18months or so and disrupting that seemed scary. We got very lucky that both our kids were pretty easy, and even then there were times when you just want to put the baby down and walk into the ocean...
At the same time I'm perfectly cognizant that this is probably just rationalization for some "feeling" we had--maybe that both my wife and I have one sibling, so that's very familiar to us as a family dynamic. We could have afforded more children and maybe we should have and it would have been interesting and fun in some other new and exciting ways, and well feel like depressed empty-nesters when they head off on their adult adventures. But we won't know, and just have to wait for the cycle to come around again and get grand-children. And of course enjoy/taunt our friends who still have younger kids...