Women are entitled to associate only with each other if they so choose. There are men-only groups and spaces. Recognising that human beings are sexually dimorphic is to remain within the realms of reality. Why must women accommodate everyone?
The history of feminism would demonstrate that women do not all agree with one another about many things. There may be some women who think that there is a natural solidarity on the basis of sex, but not many. Wanting women-only spaces does not require a fantasy about love etc.