Should I care what anybody else thinks?

Wow. Don't ever let anyone influence who you date. These people that make fun of you are morons. It says a lot more about them than it does about you. Get back together with this guy and don't worry about what these other idiots think. I certainly hope these people are not your 'friends'. If they are, get new friends. If they insult you, be the bigger person and ignore it. They're just losers.