Navi 21 will have a 128MB Infinity Cache. Meanwhile AMD isn't speaking about other GPUs, but those will presumably include smaller caches as fast caches eat up a lot of die space. On which note, doing some quick paper napkin math and assuming AMD is using standard 6T SRAM, Navi 21's Infinity Cache would be at least 6 billion transistors in size, which is a significant number of transistors even on TSMC's 7nm process (for reference, the entirety of Navi 10 is 10.3B transistors). In practice I suspect AMD has some optimizations in place to minimize the number of transistors used and space occupied, but regardless, the amount of die space they have to be devoting to the Infinity Cache is significant. So this is a major architectural trade-off for the company.