I don't have a problem with saying no to random guys at the subway/bus, but my problem takes place in environments such as work and school (lol, okay not really school), and these are also guys who I see regularly and get along with. So once it comes to a point where we're pretty friendly they take it a step further to ask for my number, which I'm not so comfortable giving but at the same time sort of feel bad if I say "no."
Ugh, why do I care so much :< That's the real problem.
Best advice you could have given yourself lol. Then again depending on the area and who you are I guess it is something to a person should care about.