Oh wow, I didn't see this earlier.
https://www.reddit.com/r/civ/comments/5947kp/civ_vi_mechanics_how_district_production_cost/
Basically district costs don't increase once you have the number of the type of district of an average player in the current game. You get a max of 25% cost reduction depending on how many fewer districts of that type you have in your civ compared to the average player.
The main cost increase driver is instead the number of civics or techs that you have completed. That kind of means that if you want to have an empire that has a ton of districts, it would be in your best interest to slow down both your science and culture rate.
As a side note you can lock in cheaper districts by putting them down immediately and only finishing them afterwards. That's actually a pretty good idea for planning districts as well.
Time to do a district heavy game. I also realized I have forgotten about the general district adjacency bonuses for most districts in my games so far. That explains why I always thought the adjacency bonuses civic cards were horribly weak.