One of the first quotes I share with my students is one that (as far as I know) is attributed to Carl Sagan…
“Science is a way of thinking much more than it is a body of knowledge.”
I think at first, the students shrug and say, “Ok, neat quote, C-P.” But I really want them to internalize this idea. Mull it over. Reorganize their understanding of what they (think they) know about science and the world they live in. Really appreciating what this means can be transformative for a student. Often, they have developed a sense that “knowing science” is simply having at their disposal a set of facts and statements about the relationships between those facts. It is difficult to convince them that “knowing science” is more about understanding how those facts came to be, to what degree we can accept or work with those facts, and how it is that science moves forward. It is not about simply gathering “more facts”. It is about how we can best maintain a balance between being skeptical and accepting ideas about the world that are put before us.
With that thought in mind, I thought I would share a recent rumination by Dan Simons on a study that was recently published in Nature (article here and a more accessible description of the study here). Simons provides a textbook example of how a scientist thinks… The study in question examined how playing a video game might affect the visual attention and multitasking capabilities of an older adult. In short, the study found evidence that the game play increased certain cognitive functions of the older adults (aged 60 to 85). The response that Simons has is what he calls a “HIBAR” (“Had I Been A Reviewer” – referring to the peer review process that scientific publication undergo before being published). Some of his concerns are methodological (e.g. tasks completed by the control conditions), some are statistical (e.g. lack of measures of variability for differences scores), some are theoretical (e.g. why the video game affected some functioning but not others). Altogether, they illustrate what productive skepticism looks like. He is not simply dismissing the study or blindly embracing it. He is trying to better understand what the results show (and don’t show) about the topic, and he is trying to work through how we should think about those results. It would be silly to think that a single study answers all the potential questions about something as complex as attention, training, and aging. However, if we take a single study, like the one in question, and then carefully assess it, we see a way ahead. That is science.