Sure but depending on time and place its akin to a man punching a woman closed fist vs the other way around...on average. I'm sure some woman could knock out some men.
Every argument I've seen about this question is between people who obviously don't share a definition of "racism". Personally, I come down firmly on the side of "It depends what you mean by racist". I kind of understand why people have semantic turf wars, but I still hate them.