As someone who has done a significant amount of astronomy outreach for over a decade, I can tell you that it's mostly "they just don't care". I'm always a little surprised at how many people aren't interested in looking through a telescope. That said, sometimes they just need a little bit of prodding and the right kind of exposure to get them interested.
One of my favourite examples of this was a time when I was helping out at an event in a campground. Our club (the
RASC) brought out a few people and their telescopes to show people nebulae, planets, galaxies, etc. As is very typical, everyone wants to bring their kids out. I find this to be partly awesome, but also partly disappointing, because it implies that this is "something for the kids". And while I do love the reactions I get from kids looking at celestial bodies for the first time ... they're kids. They're filled with wonder and excitement about everything. They see Saturn and say: "Oh, wow! That's so cool!" and then they'll see a train go by and say: "Oh, wow! That's so cool!"
Anyway, a father had his two kids out, both maybe around the age of 10-12. I was showing them the globular cluster
M13. The kids were amazed that they could resolve the little stars in the cluster so easily. They asked all sorts of great questions and then thanked me. Then the dad gave me a big thanks and they started to walk off. I was like: "... hey, did you want to have a look?" ... he kind of shrugged his shoulders and said: "Yeah, sure ... why not." ... he put his eye up to the eyepiece and let out a quiet, but audible
holy shit. Following this, he asked a bunch of questions and then went back to look through all the telescopes that only his kids had looked through up until that point.