Using Plex means sacrificing the ability to play music in the background. The app is also pretty slow compared to the PS4 Media Player and requires the use of a Plex server, whereas the DLNA standard works with a bunch of apps and network appliances. Personally I would rather just use the PS3 than use Plex on PS4.
I do wish Plex supported background music playback, but I suspect that's more on Sony since the radio apps don't do background music either. Plex does have builds for the server that run on network appliances, and there's a web client built-in that works with anything that supports HTML5 video and audio streaming. The Plex server also doubles as a DLNA server automatically, so all of my music in Plex is available to the PS4 Media Player.
Aside from this one fatal flaw, it does have a better interface than Plex, even as a pure media player. It is faster. DLNA allows you to browse both by metadata and folder structure; Plex only allows the former. And it has broader compatibility.
Now, if your only choice is between Plex and PS4 Media Player, I would pick Plex, because song order is just so essential. But if we are talking about all available solutions for doing in-home music streaming, I don't think Plex is particularly good. Whereas PS4 Media Player would be great if it just fixed this one issue.
The Plex app on PS4 does have a sort by folder option. Granted, it took them way too long to add it but it's there now. I'm not 100% sure it's available for music, but I use it for unscraped videos (Plex doesn't handle disc extras for TV shows well).
The Plex app on PS4 does have a lot of issues, but it's more flexible than you think.
The console can't find any of my FLAC music, only MP3. What am I missing here?
If you're using a DLNA server, it might still think the PS4 can't play FLAC and is ignoring the files. Seems unlikely though, since most of the servers that do that can transcode. Might depend on your settings. If it's on a USB drive I don't know.