The GPU is a 1024 or 2048 bit vector processor, depending on how you count it (RDNA/Wave32 or GCN/Wave64). Dreamcast was considered 128-bit in comparison (SIMD capabilities in the CPU).
So yeah, it makes sense to offload heavy FP tasks there...
The FPU is more useful on PCs, due to them having a discrete component structure (CPU <- PCIe -> GPU) which increases latency. If you want low-latency physics (that actually affect the gameplay) on a PC, GPGPU isn't a good fit.
Consoles have an APU, which means both the CPU and the GPU are in the same chip, so zero latency.
IMHO, there's a chance Sony decided to cut off legacy x86 instructions (such as x87/SSE, assuming PS4 games didn't use those) or maybe they optimized the FPU layout to be more dense (for example, I remember RX 5700 having tons of blank spaces in the silicon layout, which was heavily improved upon in RDNA2). I don't think we'll find out due to NDAs.