How do I become more aggressive?

I find it incredibly difficult to even get angry if anyone even punches me, or verbally insults me… After the incident I think "I wish I hit him back now!" but I never do it while the situation is happening.

I'll give you some information about me. I go to the gym 5-6 days a week weight lifting. I've been bodybuilding for several years now and I'm 15 years old, 16 in a few months. I'm not being big-headed, stuck up or anything, but I lift heavier than some grown men in my gym with my normal sets (not maxing out) so that tells me I'm stronger than the average 15 year old AND definitely more muscular.

I even trained boxing for months, but when someone is kicking off with me, ready to fight, I can't push myself to it. I hate being pushed around for merely walking down the street and I hate being attacked without me reacting in any way possible. I've tried being assertive but the 'attackers' lose their fuse and start punching me anyway.
I don't know the problem, I know I can win them in a fight but I don't have the guts to do it at the time. I even avoid people who stare at me, even when they are smaller than me. What's my problem and how do I fix it?

PS. I don't want people to tell me to be a man and walk away. It doesn't work in my area. If you're confronted you WILL get in a fight, what I'm asking for is to get the courage to defend myself. Otherwise I'll end up getting seriously injured.

I've had one fight in my life with a bully. I won.
Obviously no one read what I wrote. I did say I'VE TRIED TO BE ASSERTIVE. The last resourt is to defend myself, all I fucking want is to stand up for myself. You guys seem to think I'll be searching for fights. It's fucking self defence.

BTW, I aint jet li, so I don't intend to take classes to dodge attacks.