Why is wrong to hit women, but seems perfectly fine the other way around?

If you attack someone it's bad regardless of gender. If I attacked a man I'd expect him to hit me back I would deserve it. If it's self defense slug away but to be the aggressor is just barbaric.