Yeah, Dual Wielding is weird. Really, all it is for your damage increase (barring the actual +Stats not the damage of your Off-Hand) is that you get a 15% Attack Speed increase and more attacks per given timeframe (and, thus, more Spirit/Hatred/Fury). But since you're alternating attacks between the two, you MUST HAVE a good Off-Hand so time spent attacking with the Off-Hand isn't wasted as far as the character sheet damage over time is concerned.
Basically, Dual Wielding may give you +15% Attack Speed but it also adds your Off-Hands attack speed to the delay until your Main Hand swings. You either need that delay to deal about as much damage as your Main Hand or you need the delay to be infinitely small.
Let's say you have a 100 DPS Weapon that attacks once every second. Fairly straight forward, it attacks every second for 100 Damage.
Now, you add an Off-Hand that also attacks once every second. Due to the 15% Attack Speed bonus, you're now attacking once every .85 seconds for each of them. However, after your Main Hand swings, you wait .85s before your Off-Hand swings. Then you wait .85s before your Main Hand swings. So, really, your Main Hand doesn't swing a second time until after 1.7s, .7 seconds longer of a delay than if you were not Dual Wielding.
What this means is that you need that extra .7s delay to be "worth it." Generally, this means that your Off-Hand needs to deal 70% the damage to be equal to Main Hand only. In practice, this tends to not be the case, hence my suggestion of closer to 80%. Keep in mind though, you still hit more often so it still benefits in other ways besides raw damage over time from spamming your primary skill.