I haven't experienced it myself, but I know of at least two people who have been in the same boat you are in, and as I say, it's not an area that will vanish by itself.
In the meantime, you can work out your pectoral muscles, which will make a visible difference to that area of your body. Bench presses and push-ups are two easy ways to target that area of your body. But beware, even if you work this area really hard, it's still more than possible that you will end up looking like you have firm man boobs (yeuch!) as opposed to lean muscle. Quite often you can see this amongst heavy weight lifters and wrestlers...
Based on the description you have posted here, I'm afraid to say that it sounds like you'll need to visit the Doctor. As I understand it, having surgery on that part of the body is actually really common amongst men, so I wouldn't feel discouraged by it.
Hopefully I haven't scared you off or anything. You've done brilliantly to lose the weight you have done, it's just that unfortunately, sometimes the human body doesn't want to reward you in the way that you deserve
.