They built hardware to take advantage of many parallel GPU and GPGPU tasks. They created software technology to make writing such software easier.
The task people ended up creating stuff that while it leveraged GPUs, didn't parallelize nearly enough to make the architecture choices made for the Mac Pro good ones and the physical design apparently precludes doing anything else.
A good GPU-centric machine should have options for a
lot of GPU power, and it should get frequent refreshes since the GPU market moves so fast with substantial performance and feature improvements. To make those frequent refreshes possible, the system design should have enough slack that it doesn't have to be redone from the ground up every time the next gen of GPUs happens to have 30 watts higher TDP. User upgradability wouldn't have hurt either.
Apple failed all of that. Even at launch their top end GPU was weak (compare to GTX 780/Ti/Titan, all of which launched before the Mac Pro). They painted themselves into a corner with a design that surely made it quite difficult to update the machine, and then failed to offer even sporadic updates.
Dual slow GPU design is bad compared to if you can have the same performance in a single faster GPU. Harder to utilize, surely more expensive and limiting in terms of what models can be offered.
They also failed on software support side. Didn't even bother to make it possible to use both GPUs for rendering, which the exact same machine could do if booted into Windows. Typical for Apple to not give a shit about GPU driver functionality, but when you build a special GPU-centered machine, that might have been the time to make an exception.