As more clear examples, I think the 3D0 was terrible hardware. I think the Neo Geo was terrible hardware. I think the Sega CD was terrible hardware. All of these systems were very powerful for their time.
In the past, the primary problem with high end hardware a la the 3D0 was that it was expensive to produce; this meant that the 3D0 itself was 400 dollars to the competition's 150 or 200. This automatically limited its appeal and potential. Today, the primary problem is not hardware cost, but development cost; as development costs continue to rise, your system has to be very popular to justify making a game at the high end of the cost spectrum.
You can still make PS2-esque games for the Vita, but those are extremely unlikely to be system sellers. "System sellers" tend to be things which take particular advantage of whatever the system's strong, unique selling points are, whether those strong points be graphics (e.g. Gran Turismo 1 on the PS1) or online (e.g. Halo and Call of Duty on 360) or controls (e.g. Wii Sports). Games which don't take advantage of the system they happen to be on can still be good games, but are very unlikely to be what we casually refer to as "system sellers."
The Vita's primary claim to fame, relative to its immediate competition, is its raw power. Making "system sellers" for the Vita is almost by definition an expensive endeavor, and that is a primary problem with the system.