You got a point, judging by the Tower Bridge vehicles.Also think about volume not just height. To make a cuboid twice as tall would require 8 times the volume and 8 times the surface area. You might get away with larger bricks, but it'd be MASSIVE and cost way more than any Lego set ever made.
Tower bridge is closer to mini modular scale.