Most animal species display dimorphic sexual behaviors and male-biased aggressiveness. Current models have focused on the male-specific product from the () gene, which controls male courtship and male-specific aggression patterns in fruit flies, and describe a male-specific mechanism underlying sexually dimorphic behaviors. Here we show that the () gene, which expresses male-specific Dsx and female-specific Dsx transcription factors, functions in the nervous system to control both male and female sexual and aggressive behaviors. We find that Dsx is not only required in central brain neurons for male and female sexual behaviors, but also functions in approximately eight pairs of male-specific neurons to promote male aggressiveness and approximately two pairs of female-specific neurons to inhibit female aggressiveness. Dsx knockdown females fight more frequently, even with males. Our findings reveal crucial roles of , which is broadly conserved from worms to humans, in a small number of neurons in both sexes to establish dimorphic sexual and aggressive behaviors.