His character Walker got one of, if not the, the most brutal deaths of the series. Also, right before he died, half his face was chemically burned badly, so they couldn't even pull a "oh, it was another guy with a Walker mask on who for some reason was acting exactly like Walker for…..reasons", oh god, I'm having that bullshit Halloween Resurrection retcon nightmares from saying that, bleh, anyway, then Walker's skull gets impaled by a hook and he is pulled into a helicopter that explodes and then plummets a massive drop.
Suffice to say, only way Cavill was returning after that was in a flashback. Jeremy Renner's characte from 4 and 5 returning would have been cool.