Will I break up with him?

I am with this guy who I really love, and it is my first time to be in a relationship, and I agree, love is very confusing and hard. I sit beside him in homeroom which is good so we can talk a lot. But during recess, It's like we are not together. His ex girlfriend keeps flirting and it's like they are the ones who are together and he spends more time with her than me. I wanted to break up with him a week ago but I can't because I love him and I don't want to lose him but I'm getting hurt. I really don't know what to do. He kept asking me what's wrong because he said I look so sad and different because I ain't like that before winter break. I don't know what to answer and I just don't know what to do... Please help me...