And on what else? I'm using the same 980, the same operating system (win10), same ssd (500gb 850 evo), same PSU, same case, same cpu cooler etc.
And before you ask: My 3570k was overclocked to 4.3ghz, ran on a z77 board and had 16gb of ddr3 2400mhz ram. As you can see nothing was thermal throttling, voltages and power was also okay and I'm for sure not running something else in the background. Still I had to deal with many situations were JC3 went below 50fps.
Also as you mentioned the GPU load wasn't maxed and CPU load doesn't need to be at 99% on all threads to indicate that it's running at it's limit. A high CPU load (above 70%) and a GPU load below 90% is already an indication and a good clue.
To give you an example. I also did some testing with deactivated HTT on my new 6700k. CPU load went up to ~80%, fps went down and GPU load also went down to 89%.
For a long time I was under the impression that 'older' CPUs (especially overclocked i5s) are a perfect match for high end gpus. I was even planing on just upgrading my 980 in the future and to use my 3570k for one or two more years. Luckily I didn't. Before I upgraded my CPU I ran some games and took screenshots in areas that I remembered to run 'bad' on my PC. After upgrading I replayed those areas and to my own surprise fps were generally ~25fps higher in Witcher 3, GTA V, Batman AK and they even nearly doubled in JC3.