As it is, i think it's explained enough without recurring to exposition of it, you have the context that jackson it's safe and they manage to deal with a couple hunters that attack without problem now. You get the back story that they patrol the surroundings and that they even have favorite routes, being their favorite route, that exact route.
You also get Joel having hobbies and a house now. You see kids growing up without worries inside Jackson so people like Joel, Ellie or Thommy are probably the ones most exposed to this kinda of trouble but even then, they seem pretty chill about it.
I think all of this makes sense, people can't live in survival mode 24/7 and in the situation they find Abby, they don't have much choice. (which is bad luck, but common, it makes as much sense as it should, we have logical explanation as to how Abby tracks them and how they even get in this proble, seasonal hordes, favorite route, etc)
Remember this, Tommy presents themselves, without last name before arriving to the cabin to Abby, Tommy obviously does it triying to inspire trust in Abby.
Second point is that Joel isn't even fond of going to the cabin but Tommy convinces him that it's either that or death by infected.
Abby it is conflicted for a moment or 2 before going all John Wick on Joel too, but she quickly pulls her shit together and comes up with the plan of taking them to the cabin.
And in the moment that Tommy presents himself and Joel to the group, not even the group knew who they were, so the whole situation isn't even something to be suspect about, but since Abby has other plans everything shifts in a second.
I also think that it often gets ignored, how efficient the flashbacks are in context. Without the story starting as it does you wouldn't be missing Joel and even then, every time the game thinks that you might be forgetting about Joel, bam! flashback so you don't dare forget about him. The game kinda thrives in making you feel loss during Ellies half i think. I kinda went off track, but since we were on the topic...