Look, you can have trans women in your washrooms or trans men. For a cis man to fake being a trans woman to enter a restroom he'd have to put on a dress. For a cis man to fake being a trans man he'd do nothing, bc we're indistinguishable from cis men to the human eye.
If you're uncomfortable with men in your restrooms then you don't want to force trans men in your restrooms based on birth sex. If you push on, we'll take you up on it and be there, looking like cis men, making it easier for cis men to enter women's restrooms too.