Evaluating the Supplementary Materials

Research

Most of us read the main text of a paper as the definitive record of an experiment. It might be more useful to think of the main body as a curated narrative. Authors have a legitimate need to streamline their findings to make a clear point, especially given journal page limits and the need for a cohesive story. If every failed trial or minor variance were included in the primary text, the core discovery might be buried in noise. However, that curation can mask the actual behavior of the system. If the goal is to understand the robustness of a finding, the supplementary materials are where the real story usually lives. This is where the raw variance and the less convenient data reside. When reviewing a new study, try skipping the results section and going straight to the supplementary data. Look specifically for three things: first, the raw scatter plots before the data was smoothed or filtered. Second, the lists of excluded outliers and the specific criteria used to remove them. Third, the control experiments that are mentioned only in passing in the main text but detailed in the supplement. Suppose a paper claims a highly efficient new catalyst with a clean 95 percent yield. The main text will likely show the best three runs to demonstrate the potential. But the supplement might reveal that the first ten attempts yielded only 20 percent, or that the reaction only worked when the ambient humidity was within a very narrow range. That doesn't necessarily mean the result is fraudulent; it simply tells you how fragile the effect actually is.