You shouldn't have to listen to people question your judgment especially when only you know what's right for your life. No one knows you like you know yourself. When you say "no" the answers no. When you say "yes" then let it be a yes. No explaining.
I did not happen to learn this a long time ago, only a few months ago. Give it some thought. Turns out to be true. They are not entitled and they will think what they want anyway.