We are solid state chemists and this is our confession. We scientists tend to think of ourselves as rational and objective, and we focus on detailed scientific questions. But we’re not as rational as we like to think we are.  Science is about decisions (what experiments do we run, what do we notice—much less record and analyze, what do we do when an experiment fails) and the decisions that we make are subject to all of the biases and heuristics encountered in other areas of life.  

Humans have evolved fantastically useful decision heuristics to find freshwater, decide which mushrooms to eat (or not eat), and where might we find the best hunting grounds.  Modern day equivalents might involve knowing that well-known chains will always supply passable coffee or tacos that will fill our stomachs, but won’t excite our palates.  Following the crowd can be a useful decision heuristic, especially when the payoff is uncertain and the risks are high.  On the other hand, getting “stuck in a rut” prevents exploring the full range of options and finding better opportunities.  It means eating the same burrito and drinking the same espresso again and again and again. These are fairly well-known in the social sciences, but surely we chemists do not suffer from these habitual patterns…or do we? 

While attempting to create generative machine learning models for materials synthesis, we found that our training data was clustered strongly around certain “recipes” of ingredients and reaction conditions. We’ve been studying organically templated metal oxides for many years now, during which we’ve searched through the Cambridge Structural Database (CSD) countless time.  It seemed to be that a small number of organic amines were found quite a lot, so we put a few undergraduate students on the task of extracting amine identities from all the organically template metal oxides (or metal borates) to actually quantify the number of structures in which a given amine is observed.  Reported compounds, which are of course publically accessible, are a proxy for efforts expended on specific reagents, but what about bias in reaction condition? Such data are not generally available. So we turned to our Dark Reactions Project,  in which we’ve created a database of all our reactions for the past 15 years (assembled by a team of undergraduate students who entered data from old laboratory notebooks).Both the CSD and unpublished laboratory notebook data indicate that a typical chemist tends to “follow the crowd” by selecting reagents and reaction conditions similar to past experiences.  We had a team of undergraduate students perform new randomly generated experiments, and found that there was no “good” reason for this strategy, as the unpopular choices are statistically just as likely to result in successful reactions as the popular ones.

The human behavior we found might just be a curiosity, if it weren’t for the recent attempts to use machine learning models to predict and plan syntheses...and these models typically rely upon anthropogenic experimental data. We found that machine learning models trained on anthropogenic data were less capable of predicting novel reactions than those trained on a smaller set of randomly generated reactions.  Furthermore, the machine learning models tended to inherit the biases inherent in the anthropogenic dataset. Avoiding bias in the underlying data result in stronger model.

 Traditional linear march experimental campaigns only really facilitate human understanding, while algorithm-based modeling approaches can extract more value from data that has less intentionally-imposed structure.  These need not be anything sophisticated, like statistical design-of-experiments or active learning algorithms. We found that even randomly selected reactions are better than using biased anthropogenic data. Maybe the advantage of machine learning is not any “super human” predictive capability, but that it’s consistent and is less likely to get stuck in the same cognitive biases that humans fall prey to.  If we want to realize that potential, then it is important to make sure that the training data doesn’t persuade algorithms to make the same mistakes as humans.

To read the full article, please click https://www.nature.com/articles/s41586-019-1540-5