Not a dedicated handheld that would require its own library of games; that's unsustainable these days. Even Nintendo can't do that and a separate home console, and they have arguably the least expensive AAA games in the business (in terms of production costs), and the cheapest hardware (in terms of specs and build quality, IMO).
But should this have been a PS4 Portable, that could do Remote Play of all PS5 games and native play of PS4 games on the go, and maybe native play of lower-scale PS5 games getting PS4 cross-gen releases anyway (basically lots of indies or annual sports releases, or certain GaaS titles)? Yes, definitely. A device with much more use-cases, and could have been priced higher to boot with likely similar or higher profit margins ($349 or even $399 for that type of portable would be very reasonable).
I get this device is for those who already have PS5s, for second-screen experience, but if those people are willing to pay $200 for "just" this, they would probably be willing to pay $399 for that plus something that could cloud stream PS5 games (including PS+ Premium titles), and natively play PS4 games. And, that would also be a device appealing to other people in the market, even those who don't have PS5 consoles. While still having good profit margins.
But that would've required more effort and, well, this thing doesn't seem like a lot of effort went into it at all, IMO.