Several previous studies have shown that some morphologically distinctive, small genera of vascular plants that are endemic to the Qinghai-Tibetan Plateau and adjacent Hengduan Mountains appear to have unexpected and complex phylogenetic relationships with their putative sisters, which are typically more widespread and more species rich. In particular, the endemic genera may form one or more poorly resolved paraphyletic clades within the sister group despite distinctive morphology. Plausible explanations for this evolutionary and biogeographic pattern include extreme habitat specialization and hybridization. One genus consistent with this pattern is Nomocharis Franchet. Nomocharis comprises 7–15 species bearing showy-flowers that are endemic to the H-D Mountains. Nomocharis has long been treated as sister to Lilium L., which is comprised of more than 120 species distributed throughout the temperate Northern Hemisphere. Although Nomocharis appears morphologically distinctive, recent molecular studies have shown that it is nested within Lilium, from which is exhibits very little sequence divergence. In this study, we have used a dated molecular phylogenetic framework to gain insight into the timing of morphological and ecological divergence in Lilium-Nomocharis and to preliminarily explore possible hybridization events. We accomplished our objectives using dated phylogenies reconstructed from nuclear internal transcribed spacers (ITS) and six chloroplast markers.

Our phylogenetic reconstruction revealed several Lilium species nested within a clade of Nomocharis, which evolved ca. 12 million years ago and is itself nested within the rest of Lilium. Flat/open and horizon oriented flowers are ancestral in Nomocharis. Species of Lilium nested within Nomocharis diverged from Nomocharis ca. 6.5 million years ago. These Lilium evolved recurved and campanifolium flowers as well as the nodding habit by at least 3.5 million years ago. Nomocharis and the nested Lilium species had relatively low elevation ancestors (<1000 m) and underwent diversification into new, higher elevational habitats 3.5 and 5.5 million years ago, respectively. Our phylogeny reveals signatures of hybridization including incongruence between the plastid and nuclear gene trees, geographic clustering of the maternal (i.e., plastid) lineages, and divergence ages of the nuclear gene trees consistent with speciation and secondary contact, respectively.

The timing of speciation and ecological and morphological evolutionary events in Nomocharis are temporally consistent with uplift in the Qinghai-Tibetan Plateau and of the Hengduan Mountains 7 and 3–4 million years ago, respectively. Thus, we speculate that the mountain building may have provided new habitats that led to specialization of morphological and ecological features in Nomocharis and the nested Lilium along ecological gradients. Additionally, we suspect that the mountain building may have led to secondary contact events that enabled hybridization in Lilium-Nomocharis. Both the habitat specialization and hybridization have probably played a role in generating the striking morphological differences between Lilium and Nomocharis.

The online version of this article (doi:10.1186/s12862-015-0405-2) contains supplementary material, which is available to authorized users.