You shouldn't have to listen to people question your judgment especially when only you know what's right for your life. No one knows you like you know yourself. When you say "no" the answers no. When you say "yes" then let it be a yes. No explaining.
The first murder was committed with a rock. Shall we ban all rocks? After all, there are a lot more rocks in the world than guns. Let's deal with the real problem: evil in the hearts of men.