You shouldn't have to listen to people question your judgment especially when only you know what's right for your life. No one knows you like you know yourself. When you say "no" the answers no. When you say "yes" then let it be a yes. No explaining.