I know what you mean. My parents both came from families of four. But most of my cousins only have 1 sibling each, same with me.Ideally it would be awesome to have five or more kids. My parents had tons of siblings, and it really benefited both them and me. I'm a little disappointed that my hypothetical kids will just have one uncle on their dad's side.
Practically, I could never imagine having more than two, unless I was very wealthy.
I'd go with 2-3 for myself. 2 minimum.