Unbeknownst to Xander, he had false visions of a future implanted in him by a demon who disguised himself as a futuristic elderly version of Xander who traveled back to his wedding day to warn him.
The demon showed Xander a future where he was miserable in his oncoming relationship: a man who resented his marriage and even children as he grew more and more frail with age.
This reminded Xander too much of his alcoholic father, who was already making a fool of himself at the wedding reception by drunkenly belittling both his wife and future in-laws. Xander feared becoming his old man.
Even after realizing these futuristic visions were falsified, Xander was ultimately so shaken by the possibility of who he could become that he felt as though he didn't deserve Anya as a bride.
Xander left Anya to walk down the aisle alone, in tears and wounded, bleeding from the fight against the demon they had just defeated. The entire church in shambles after a family sized brawl. Yet it was Xander who fled, possessing the largest scar of all that day.