AI stuff (including scaling and framegen stuff like DLSS, or to generate your own porn pics or videos using genAI locally) and RT are very VRAM dependent, and in general Nvidia is better for anything AI related and some AI stuff is even Nvidia exclusive.
So I'd choose keep the 8GB and 12GB cards out of the question and I'd go for Nvidia. Now, which Nvidia?
Some AI stuff (like DLSS) are dependent on the architecture/generation of the GPU (if it's a 3000, 4000, 5000 card), so the higher the better. If possible, a 4000 or 5000 card (second hand to reduce prices), not only because of the current stuf, but also considering new DLSS and AI stuff to be released during 2026-2028.
These cards will be bottlenecked because both the PCIe 3 and the CPU, but will be the case of any new powerful GPU you'd put there. But well, you should notice a big jump compared to the current 1660 Super.
What I'd do would be:
- Get now a second hand 16GB VRAM GPU as good as possible (I have no idea about prices, but I'd say the best one you find with your budget between 4060, 5060, 4070 or 5070) and as fast as possible, before the price increase
- Then somewhere later (depending on your budget, maybe 2027/2028) to get the full potential of the GPU and to remove CPU, PCIe and RAM bottlenecks (maybe 2027 or 2028, or earlier if you can) get new motherboard + CPU + RAM (new Zen 4 and Zen 5 CPUs use a new socket and require DDR5 RAM, so can't reuse in a new board your current CPU or DDR4 memory). If price of this goes to high, a shorter term alternative would be to get the max you can put in your current board: something like the maximum CPU you can put there (as of now I'd say AMD Ryzen 7 5800X3D) and 16GB more of DDR4 3000
Regarding stutters, it's something even people with full high end rigs have in PC (in many cases it's game/engine depending), but obviously having lower end stuff bottlenecking doesn't help. Some protip for the rig you already have: use the SSD for "C:" and the game you are playing and virtual memory, leave the HDD for the other things like emulator roms, savedata, docs, non-gaming apps etc. Make also sure you aren't using virtual memory in the HDD, that also causes performance issues in games.
HDD is an important bottleneck nowadays and SSD work way better. Format and reinstall Windows (if possible one of these less bloated versions) if you haven't done it for many years if the SSD is too full, there's a lot of crap left there from installed and uninstalled stuff, windows backups stuff, no longer needed temp files and so on.