After listening to the CD a few more times, I've come to the conclusion that this album was simply overproduced.
I think the reason many have listed Exodus '04 as their favorite(it's mine too) is because there's not much going on in terms of vocal effects and what not. Just a nice beat with a slight touch of a Eastern sound, and letting Utada rip it with her amazing voice. I believe Timbaland produced that track, so that might explain it's high quality.
But other songs just add too much noise, for lack of a better term. (The Workout comes to mind.) At some points, I feel like they're trying to make her sound almost like a poppier version of Bjork, and it doesn't come off too well.
For example, the last track starts out just fine with a little acoustic guitar, but then has to go all electronica by the middle, when just the guitar and her voice could of carried that track alone.
Hopefully on the next album, they'll show a little restraint and let her voice carry the album, not lots of background noise.