Honestly, the worst thing about Garrisons is the lack of any real 'payoff' without having a huge gated wall behind it, be it in material gathering that can't really be sped up (or even grinded out, as rewards for garrison resources start getting scarce after the 1st 'x' event of the day/week)
Combine that with nothing to really 'do' in the follower management, and it makes for a sad, boring time.
Granted, I came back in about a month ago, so I missed most of the content dearth. But it's still kind of noticeable, imo.
edit: Also, I still don't get why reg/heroic/mythic raids give zero valor - Yet LFR does?
I get they want people to populate LFR, but give them more or something.