FD is just an overlay on the PFD, the pilot can use the FD as a guide to maintain the course generated by the FMC / Nav aids during manual flight.
There are a few instances where pilots will manually fly during cruise, that being any mechanical / electrical failure, at least until they are sure it won't affect things, many carriers have a blanket rule about manual flight during certain icing conditions, which could be attributed to some accidents where icing and subsequent stalling has been present, there is also a certain level of turbulence in which the pilots will return to manual flight.
You've also got to appreciate the flight logic system of the Airbus system, manual flight isn't truly manual flight, because there are always a number of logic systems inbuilt in the flight control system (A-Floor etc) that will override the pilot inputs if the flight system thinks things are going wrong.