I was wondering this myself the other day. The Olympics includes way more countries obviously so it should draw more people, but considering its one sport, the World Cup is an absolute monster of an event. City centres get packed out for for the big matches with hundreds of thousands of people on the streets watching the game in public places as its televised on big screens around the city.
The World Cup is definitely more exciting the watch for the sheer spectacle of it as it is far more focussed but the World Cup is pretty much always a packed out event, the past few Olympics actually look like they didn't draw huge numbers of spectators, Athens in particular was like a ghost town, I actually felt bad for them, I can't imagine how much they lost on that one.
I went to Sydney Olympics and there was definitely a lot of people around, saw the finals of the Soccer and some track and field. The closing ceremony was huge, everyone went down around The Rocks and Sydney Harbour to watch the fireworks. It was chaos. Nearly got crushed.