A February 2017 Workshop on Robustness, Reliability, and Reproducibility in Scientific Research was sponsored by the US National Science Foundation (NSF). The workshop was part of NSF’s response to growing concerns in Congress triggered by increasing media coverage of an apparent lack of reproducibility among findings, especially in clinical sciences. The participants, spanning diverse scientific subfields, were charged to assess the extent of any problems of reliability and reproducibility, and to formulate next steps toward solutions.
The breadth of expertise among participants afforded opportunities to learn, and explore prejudices, about each other’s subfields, while probing the nature of reproducibility problems in sub-disciplines at different maturity levels. Much of the time was spent in small groups to discuss aspects of the broader issue. One such group focused on the role of theory in guiding experimental research and identifying non-robust results. A prominent experimentalist in soft condensed matter physics broke the ice by opining that theory was largely irrelevant to his research. He was following his nose to discover breakthrough phenomena or new materials with potentially important applications. In that approach, what might appear as irreproducibility among experiments by different groups often just reflects variability arising from not-yet-recognized dependences on some experimental conditions. Attempts to resolve these discrepancies promote understanding of the conditions and dependences.
I pointed out that in nuclear and particle physics we were at a quite different stage, with well established, if not always complete, theories and an extensive experimental database to build upon. Experiments to confirm or find flaws in those theories have the potential to unveil profound new physics. But experimenters who produce results that are real outliers from theoretical predictions or previous measurements have “a lot of ‘splainin to do” before the broader community will take the new result seriously. A prominent irreproducible result in these fields occurred in September 2011 when the OPERA Collaboration reported apparent evidence that neutrinos travel faster than light. That result would have violated the fundamental postulate of Special Relativity and seemed inconsistent with the earlier observation that neutrinos from supernova SN1987A were detected on Earth at nearly the same moment as the light from that stellar explosion. Widespread skepticism was validated when the OPERA experimenters later traced their spurious result to a faulty fiber optic cable in their setup. Identifying inadvertent mistakes is part of the self-correction embedded in the scientific method.
This example prompted a provocative question from my sparring partner:
“Why is anyone still interested in doing nuclear or particle physics? The answers already seem known. The Higgs boson discovery completed your Standard Model. Opportunities to discover real breakthrough phenomena seem slim. You folks spend inordinate amounts of time, energy and money redoing experiments that have already been done many times. Aren’t you beating dead horses?”
In response I explained that ours is a subtle Universe. The fact that there is any matter in it at all seems to rely on some mechanism that disrupted an exact early balance between quarks and antiquarks in the infant Universe by a part per billion. This mechanism violated fundamental symmetry principles in ways that our currently accepted theories cannot yet explain. The residual matter is able to produce the energy and the elements essential for life only because the down quark is just enough more massive than its up quark partner, a mass difference parametrized but not explained by current theories. The agglomeration of matter into galaxies and galaxy clusters seems to have relied on attractors comprising dark matter, whose existence we infer from indirect observation of its gravitational influences, but whose nature we do not understand. The Universe’s longevity and accelerating expansion appear to be assured by an unknown source of dark energy that would have been utterly negligible in comparison with radiation and matter in the infant Universe, and that is mind-bogglingly smaller in magnitude than theoretical expectations.
The horses of which he spoke aren’t dead. We’re using our shovels not to beat them, but to keep digging so they don’t get buried alive. Jim Cronin and Val Fitch won a Nobel Prize for discovering that so-called CP symmetry is broken at the 0.2% level in kaon decays, when a previous experiment had placed only an upper limit of 0.3% on that violation. It paid to keep digging. But this violation, now incorporated in the Standard Model, is still wholly inadequate to explain the matter-antimatter asymmetry in the Universe. So nuclear and particle physicists keep digging to unveil other sources of CP violation, for example, via tiny differences in behavior between neutrinos and anti-neutrinos or a tiny spatial separation between positive and negative electric charge inside the neutron, or in the virtual particle-antiparticle cloud surrounding an electron. Clever experimentalists, undaunted by 60 years of null results, continue to find innovative ways to improve the sensitivity of previous experiments by orders of magnitude.
Previous experiments have failed to observe even a single proton decaying, demonstrating that the proton’s lifetime is at least 23 orders of magnitude longer than the age of the Universe. But it’s still worth searching with improved sensitivity for this “forbidden” decay because theories predicting the unification of strong, weak and electromagnetic forces in the earliest instants of the Universe require protons to be not perfectly stable. Similarly quixotic searches under way for an exceedingly rare, exotic decay mode of some nuclei have the potential to establish that neutrinos are their own antiparticles, adding plausibility to speculative suggestions that as yet undiscovered very heavy neutrinos played essential roles in creating the Universe’s matter-antimatter asymmetry. Particle physicists are exploring many avenues to reveal the microscopic nature of dark matter and additional features of dark energy. We keep digging in order to understand the subtle mechanisms by which our Universe evolved.
Science progresses with a wide spectrum of practitioners, from intrepid explorers who go only where there are no maps, to meticulous craftspeople who design exquisite experiments searching for tiny loose threads that, once pulled, may unravel large portions of what we think we already know.
Featured image credit: ‘The boy lost in the woods’ by Guillaume Jaillet. Public Domain via Unsplash.
You certainly ask some interesting questions for physics.