I can understand you not wanting to do an activity where people are going to make fun of you purely for doing it, even though it is a harmless activity. Bringing up weight/size is, to me, the sad, tired fall back of someone who has nothing constructively critical to say.

With regards to the jealousy, though, there is always going to be someone better, in almost any activity. I'm able to ignore that and still have fun, but if that makes the activity no longer fun for you, by all means, pursue a different passion. :)