Yeah, that's the reason he can't control the Hulk. Basically his father abused him as a child and then killed his mother in front of him, which caused him to manifest a multiple personality disorder as a kid. But he was always exceptionally good at repressing it until the gamma bomb exploding happened and the superpowers latched onto the repressed Hulk side, which was essentially a furious four year old who'd just lost their mother. Which is basically why the Hulk is dumb and reacts to stuff by trashing the place, because it's four year old Banner's mind throwing the most epic temper tantrum ever.
Personally, I find it makes the character a lot more interesting, because Banner was always a ticking time bomb, even prior to Hulk abilities. Numerous writers have speculated that if he hadn't gone Hulk, he might have actually done more damage when he finally snapped.