5 Seconds after you complete your transaction: the money leaves your account
5 secs to 2 days after: The bank/credit card company is holding onto the money while they check all the info, make sure its all legit, go through federally mandated safety measures, etc.
2 to 3 days after: The merchant actually receives the money.
Now, just apply that in reverse for refunds. 5 seconds after the transaction, the money has left the merchant's account, then the bank does a day or two of processing, and then, 3 days later, you finally get the money.
It only LOOKS like its done in 5 seconds for payments, and takes 3 days for refunds, because you're only seeing your end of things. Youre not seeing the middle part of the merchant's part.