You shouldn't have to listen to people question your judgment especially when only you know what's right for your life. No one knows you like you know yourself. When you say "no" the answers no. When you say "yes" then let it be a yes. No explaining.
Ask yourself this when someone says you can't do it. What's holding you back? Exactly. Nothing but them and you, but only because you are letting them.
Dear Self. You know what they say about perfectionism...don't you? Why you don't have to be perfect, and in fact it's necessary that you don't try to be.