You shouldn't have to listen to people question your judgment especially when only you know what's right for your life. No one knows you like you know yourself. When you say "no" the answers no. When you say "yes" then let it be a yes. No explaining.
"Holding onto anger is like drinking poison and expecting the other person to die." That's why we could care less about you. I'm happy to see we still get under your skin though. We haven't thought about you in almost a decade but we're flattered you