Right, I remember reading in a post a long time ago in the PCSX2 forums that some games will do some type of over-buffer causing a small slowdown. Sorry for the terminology, I can't remember exactly what was used, but it isn't solvable, least I don't think it is. As a quick test, you can try setting the EE Cyclerate to 2 or the middle setting, it may help with the weirdness.
If anything, I would try running in software mode with 3 threads enabled. To do this, go into the Video Plugin Settings, change the renderer to Direct3D 11 (software) and then make sure you have set the Software Mode Settings (located at the bottom) to 3 extra rendering threads.
If software mode doesn't cause this weird slowdown, it's a hardware type issue like I mentioned above.