Relationships between the ancestors of modern humans and other archaic populations such as Neanderthals and Denisovans were likely more complex than previously thought, involving interbreeding within and outside Africa, according to a new estimator developed by geneticists. Findings were reported at the American Society of Human Genetics (ASHG) 2016 Annual Meeting in Vancouver, B.C.

In recent years, genetics has led to the revision of many assumptions about archaic populations, explained Ryan J. Bohlender, PhD, a postdoctoral researcher at the University of Texas MD Anderson Cancer Center and first author on the research. For example, the 2010 release of the Neanderthal genome led to the discovery that Neanderthals and the ancestors of modern Europeans interbred. A few years later, scientists discovered the existence of Denisovans, a population known of only through genetics, through a fossilized sample of DNA.

"My colleagues and I set out to find out what we might share with these ancient populations and how our histories interacted," Dr. Bohlender said.

They developed an estimation tool to model these interactions based on parameters such as current estimates of population size and dates when populations separated - how long ago they stopped interbreeding - and look for inconsistencies with information known from genetic studies about the overlap between the modern human genome and those of ancient populations.

Compared to previous estimators, this one made increased use of genetic data to cut down on statistical bias. The researchers then allowed estimates of population size and separation dates to vary in a series of simulations, in order to find out if adjusting these parameters better fit the genetic data.

"Using this process, we found that the population in Africa was likely about 50 percent larger than previously thought. We also found that an archaic-modern human separation date of 440,000 years ago was the best fit, suggesting that Neanderthals diverged from our lineage 100,000 years more recently than we thought," Dr.

Bohlender said. "We got the same separation date using data from multiple modern human populations, which is a good sign."

In addition, their results suggest that throughout Eurasia, ancient populations interbred less than previously believed, and that - contrary to previous findings - the level of mixing with Neanderthals did not differ significantly between Europe and East Asia.

The findings bring up many new questions, including to what extent the new estimator can be trusted, why it produces results that differ from prevailing estimates, and how to reconcile these differences.

"Overall, our findings confirm the human family tree is more complicated than we think it is," Dr.

Bohlender said. "For example, other archaic populations are likely to have existed, like the Denisovans, who we didn't know about except through genetics." They plan to try out simulations with multiple other populations, to see if this adds some clarity to the results.

Dr. Bohlender also believes that more detailed studies of African populations may shed some light.

"Africans have been underrepresented in genetics research - they're not as well studied as European and Asian populations, yet they are more diverse genetically than any other group," he said.