If the only reason we have sex-segregated facilities is the protection of women from men (as seems mostly to be the case), then perhaps we should concentrate instead on correcting or proscribing the permitted behaviors of men instead?

Seems logical to me.