That the technology in it detracts from the overall system budget quite significantly, relative to where they money could be spent. This makes it a gamble, as in order for the Wii U to be unique and justify the budget spent on the pad versus processing hardware developers need to use the pad in interesting, game enhancing and sellable ways. As he (and I) feels most developers, specifically third parties, will reserve pad use for gimmicks like maps, alternate cams, and other junk that isn't integral to the game design, the pad's value decreases. What we're left with is a system with a core controller that is largely missused outside of anybody who isn't Nintendo, and if the pad didn't exist at all additional budget could have been put towards further modernising the Wii U's hardware.
I feel quite similar because I'm not interested in the pad at all. I'd rather the pad be gone, and instead have the HD Wii with waggle wands that I've wanted for some time.