I came to the sad realization that Simon left Marcy when she was a child, because the first time we ever saw Marcy, her shirt & bear were faded (Someone pointed it out that she was probably so hungry that she ate the color red from her things) and in the episode Simon and Marcy, her shirt and bear were still red.
When I pointed this out to my boyfriend, he suggested something even sadder, "What if Ice King told her the same exact thing, 'I'm hurting you because I love,'" when he chose to leave in order to protect her from himself. =(