This is how you must look at it.
The CPU itself isn't transferring any data, it's looking for data and the time it takes to find that data can hurt gaming performance.
This is why CPUs with 3D v-cache performs better in games, it has more L3 Cache to store the data it needs and doesn't have to look for it in DDR memory as often.
The PS5's CPU only has 8MB of L3 Cache, so it has to access the GDDR6 memory more often than desktop CPUs and to make it worse, looking for data in the GDDR6 memory can have high latency penalties than DDR memory. Around 250ns or more if I'm not mistaken.
Sony at least doubling the CPU L3 Cache can increase fps in games.
I for one hope Sony take a look at HBM for the PS6 like they did with SSDs. HBM4 seems to attempt to dramatically reduce costs by removing the interposer in favor of 3D stacking.
SK Hynix on track to explore interposer-free HBM4 production
And the latency is after L4 Cache.