New genes (or lineage-specific genes) can facilitate functional innovations. MALE STERILITY 5 (MS5) in Brassica napus is a fertility-related new gene, which has two wild-type alleles (BnMS5 and BnMS5 ) and two mutant alleles (BnMS5 and BnMS5 ) that could induce male sterility. Here, we studied the history and functional evolution of MS5 homologs in plants by phylogenetic analysis and molecular genetic experiments. We identified 727 MS5 homologs and found that they define a Brassicaceae-specific gene family that has expanded partly via multiple tandem gene duplications and also probably transpositions. The MS5 in B.?napus is inherited from a basic diploid ancestor of B.?rapa. Molecular genetic experiments indicate that BnMS5 and BnMS5 are functionally distinct in B.?napus and that BnMS5 can inhibit BnMS5 in B.?napus in a dosage-dependent manner. The BnMS5 protein can move in coordination with meiotic telomeres and interact with the nuclear envelope protein SUN1, with a possible crucial role in meiotic chromosome behavior. In summary, BnMS5 belongs to a Brassicaceae-specific new gene family, and has gained a novel function that is essential for male fertility in B.?napus through neofunctionalization that has likely occurred since the origin of B.?rapa.