His guilt wasn't that he couldn't solve it, it was that he didn't speak up.
He even said that he thought it was weird how the teacher also raised her hand. He also said how he felt overwhelmed by the classes overwhelming direction. And he even recounts how he never had anything to forget, that it was always there.
He didn't know who did it at the beginning of the episode. He had a hunch but Ougi had to make him work out all the details. That was the whole point of the subconscious line. He hadn't forgotten anything but he had also avoided connecting the dots.