You have to reach a certain "weight" of reports.
The weighting of reports is somewhat complicated, but it has to do with:
1. The types of offense for which you are being reported. (Racism for instance tends to leapfrog the rest of the weighting system and go straight to tribunal.)
2. The frequency with which you are reported vs. not reported. So, basically playing more games may make you more likely to get reported, but as long as the overall percentage of games in which you are reported stays low, the weighting does too.
3. How many people reported you for a single incident. If only one person in the game reported you, the weighting is much lower than if all nine of the other players did.
4. The "trust" the system has in the people reporting you. Basically, players who constantly file frivolous reports have the weight of their reports de-valued as a result, whereas players that very rarely report people and only do so in instances where they've done something imminently punishable pack a lot of weight behind the reports they do file.
So, if you're Bronze 0 MMR and playing with nothing but trolls, you can get reported every game and never have Riot even contact you to say, "Hey dog, get right with God." If you're a brand-new Challenger and manage to piss off the top ladder players, most of whom haven't reported someone in their last thousand games, well... good luck.