http://www.wowhead.com/garrison-calc#G2qCIQMBI0E10BReFiIBcaGAgBkoOz0D8dQ0AFEq
This would be my suggestion then, with the building levels not factoring in, and is pretty much my Garrison minus the small profession buildings. I'll go through the basics/must haves.
Small - Tailor/Enchanter is a no brainer if you're already a Tailor/Enchanter. Work Orders will help you out a lot and you'll get tons of use out of it. I like the Salvage Yard a lot because it gives lots of small goodies, though it can be a bit random. If you like, you can replace the Salvage with a 3rd Profession building or a Storehouse.
Medium - The Tavern is a pretty solid choice for people like us, who just enjoy casually playing the game and exploring the flavor of the world. The Tavern is perfect for this because it gives you quests from a lot of the popular NPCs you may remember from the 1-60 zones as well as some old favorites and some newcomers. You get pretty decent rewards from the actual quests, and some great Toys and items to play with. Then you have the Lumber Mill, and you either want to have that or a Trading Post to get your Garrison Resources flowing.
Large - War Mill/Bunker here is close to mandatory I think, if you want to engage with the end-game Follower game. Being one of the only sources of getting follower enhancements, it is absolutely needed to advance through the 630+ ilvls for your followers. The second building is a bit of a preference/convenience thing, as I don't fancy the Spirit Lodge/Mage Tower at all, and if you aren't really PvPing, the Workshop isn't that great. I would consider the Stables for the convenience factor, but in the end the Barracks wins out for me just for the +5 followers and the fact that Patrol missions are almost a must have if you got bad rolls on your followers and need to level them up.
At the end of the day though, Garrisons are fairly flexible, so long as you don't want to be at the cutting edge of 'progression' and don't mind waiting a week or so to catch up to where you would be if you made a different choice. Nothing you choose will ultimately affect you in the long run, with the ultimate price being some gold and time. I think.