It's easier to let people write you off and then operate under normal moral compunctions than it is to constantly be guarding yourself against appearances while also trying to be moral and effective
Put another way, when people say " but that's anti-[ingroup]" they are basically playing the same gambit that the judge who wagered his own genitals for removal did in Player of Games, and if your job involves serving any purpose at all you are obligated to just win regardless.