Not only was Balon Greyjoy a dick, he was a goddamn moron. Attacking the North instead of the Westerlands went directly against his own self interests.
At that point in the saga, the Lannisters were losing the war, and it didn't look possible for them to come back. Half their forces had already been taken out by Robb, and King's Landing was looking to surely be taken by one of the Baratheon brothers. By allying with Robb and the North, the Greyjoy/Stark alliance would have almost assuredly taken out enough of the Lannisters that the Tyrells would not be so willing to enter into an alliance with the side that is clearly losing the war after Renly's death. With the Lannisters utterly defeated, the Greyjoys have a very good chance of being able to remain Kings of the Iron Islands and however much of the Westerlands they're able to carve out for themselves.
Instead, he decides to attack the North like a fucking dumbass, pretty much ensuring a Lannister victory over Robb. With the North defeated, Tywin would have no reason to allow Balon to continue in open rebellion as King of the Iron Islands and would just go put him down again. He had the choice of allying with the side that, if victorious, would allow him to be a king again, or the side that he damn well knows has no interest in letting him rule.
This decision is so unrealistically dumb that it just struck me as GRRM trying to shit on the Starks as much as possible without any founding in logic, just to add some "shock value."