required reading for research methods students…

Subtitle: The Challenge of Thinking Scientifically

A study is completed. It is published. It gets cited a whole lot. Obviously, it is a done deal – science has once again illuminated some dark corner of our world… Or maybe not. I came across a post by Dan Simons that was intended to focus on the relatively simple issue of why the replication of a particular experiment might fail or not. Click on the link – read the post. It is a well articulated explanation of how one might think about results from a replication – reasons it might confirm the original experiment, and reasons it might not (btw – the post responds to a recent squabble about a published replication failure – you can read about that particular brouhaha here). Understanding how you can interpret a replication is important.

Now, read the comments. The post is good, but the comments for this particular post provide a wonderful glimpse of how psychological scientists think about a range of issues:

• Replication, of course.
• Appropriate scientific discourse.
• The challenges associated with getting published.
• Pressures on researchers.
• Effect size.
• The file-drawer effect (see this site for one approach to dealing with this issue).
• P-value fishing.
• Statistical power.

I think I’ll just use the comment section from the post as the outline for my Research Methods class next semester. Or maybe have the students critically respond to it as a part of their final exam…

autism in context…

One of the first topics we cover in Research Methods is that science does not occur within a vacuum – and this is especially true for psychological science. Recently, there has been a flurry of news stories about how certain mental health disorders are diagnosed, with a lot of debate focused on the diagnostic criteria for autism. This is occurring now because a revision of the Diagnostic and Statistical Manual (DSM), THE reference for psychiatrists and mental health professionals to diagnose mental health disorders, has been underway for years. The DSM is currently in its fourth (although revised) edition, and the fifth edition promises to make some important changes. (An overview of the revision is available here.) The changes are intended to increase the reliability and validity of the diagnoses that are made – essentially, the goal is to avoid false positives (diagnosing someone with a disorder who is actually healthy) while not excluding individuals from a diagnosis who do have a legitimate need for clinical intervention. It seems as though this should be a fairly straight-forward, empirically grounded process (the ideal), but there are a lot of other pressures playing into it – you can read a good summary of some of those issues here: Redefining Autism For DSM-V. The diagnostic criteria for autism have been arguably “loose” in past editions of the DSM, reflecting a very healthy debate about exactly what autism is.

In recent years, these has been an increase in the diagnosis of autism (MSNBC has a series of stories related to the increase – always worthwhile to see how the popular press approaches a subject, and they don’t disappoint with the title “Autism: The Hidden Epidemic?”). This increase has been accompanied by a debate – is the increase due to an increased prevalence of the disorder or does it reflect that the diagnostic criteria are too broad, resulting in lots of false positives? Depending on how you answer that question, you will see the proposed revision to the DSM in a very different light. For those that see an increase in prevalence – the revision may exclude many people who would benefit from clinical intervention. For those that see an increase in false positives – the revision represents a better diagnostic tool that will increase the reliability and validity of the identification of individuals with autism.

I am not a clinical psychologist. I am not an expert in autism. I am however interested in this debate because (a) it is important that we get diagnoses of mental health disorders right and (b) it nicely illustrates how the science of autism diagnosis is contextualized. This revision will have very real impact on families – possibly how they view the behaviors of a child that does not play the way other children on the playground do or what access they have to clinical or education services to assist them in working with that child. This revision will have a very real impact on healthcare groups – facilities and organizations may see funding disappear if the number of individuals with autism shrinks because of tighter diagnostic guidelines. The revision will impact researchers, teachers, and many others. I point this out to my students because sometimes when we are working our way through topics like “construct validity” or “false positives” in class, it may all seem rather academic and they might not see the importance of what they are learning.

creativity… going it alone or with the herd?

So, this week begins a new semester, and with it a certain resolution to more frequently add to the information I share through this little portal. Hopefully, I can maintain that focus and make a this a useful stopping point for my students (and others) who have some interest in various matters of a psychological nature. This semester, I am teaching a seminar (“Creativity & Cognition”) and a section of Research Methods. This may flavor the topics I choose to pick up on for this blog.

Case in point – with impeccable timing, the New York Times ran an opinion piece this past week looking at creativity and collaboration (The Rise of the New Groupthink). Susan Cain argues that the shift towards an emphasis on collaborative work (as evidenced by brainstorming meetings, shared work spaces, etc.) in the classroom and corporate office has been counter-productive because it stymies the endeavors of people who feel more comfortable and productive working on their own. Her new book “Quiet: The Power of Introverts in a World that Can’t Stop Talking” would seem to further develop this idea (although I haven’t read it and don’t see much of an opportunity to do so arising anytime soon). I just wanted to point out a few things in the opinion piece that caught my eye.

I think Cain’s closing idea is right on. In the second to last paragraph, Cain says, “… we need to move beyond the New Groupthink and embrace a nuanced approach to creativity and learning”. Creativity is complex – once you begin considering “why” and “how” it occurs, you realize that it involves the interaction of various components, none of which is sufficient alone to explain “why” or “how” creativity occurred. This means that there are no easy answers to either the “why” or the “how”, and most current theories of creativity reflect that idea. What I see as being the most complete theories of creativity engage with the notion of convergence in some way – the attributes and experiences of the individual interact with the current state of the domain that person is working within and the resulting ideas get picked up and transmitted by others if they are sufficiently novel and appropriate to the issues at hand. Cain’s comment in the closing reflects that complexity, but the main point of her article (that being forced into these group collaborations gets in the way of the creative individual) actually works against recognizing that complexity. She emphasizes the experience of the creative/introverted individual (e.g. Steve Wozniak of Apple), working alone, tirelessly endeavoring to create the breakthrough that will change things forever… (what in her last sentence, she appears to call “the real work”). But this ignores that fact that all of the hard work of the individual is for nothing if it is not situated properly within the domain (e.g. having the opportunity to start Apple with Steve Jobs who seemed to provide a vision for Wozniak’s tireless brilliance) and subsequently picked up by the others involved in developing home computers (from the computer “Homebrewers” she mentions to the consumers who provide a market for the yet to developed pcs). Her deference to the creative individual is reflected in the mention of Gregory Feist (who focuses on studying creativity as arising from a cluster of personality traits – a clear focus on the individual) at the start of her piece. More curious is her mention of Mihaly Csikszentmihalyi at the same point – Csikzentmihalyi has proposed a “systems view” of creativity that takes the notion of convergence mentioned above to heart. In one chapter, Csikzentmihalyi (1999) states very clearly his dissatisfaction with a focus on the individual as of critical importance in the creative endeavor:

“What the present chapter seeks to accomplish, however, is to point out that creativity cannot be recognized except as it operates within a system of cultural rules, and it cannot bring forth anything new unless it can enlist the support of peers. If these conclusions are are accepted, then it follows that the occurrence of creativity is not simply a function of how many gifted individuals there are, but also how accessible the various symbolic systems are and how responsive the social system is to novel ideas.” (p. 333)

Csikzentmihalyi is proposing that we distort our understanding of creativity by putting too much emphasis on the role of the individual, something that Cain appears to be doing in her proposal. And because she identifies the “creative” individual as being of a certain type (introverted), she fails to recognize the potential interactions between the person and the situation that are truly critical in understanding creativity. It could be that bustling and rich environments keep introverts from their creative potential, but it may benefit others. With a quick search, I found several studies that found positive correlations between extroversion and creative potential (Borod, Grossman, & Eisenman, 1971; Khosravani & Guilani, 2008; King, Mackee Walker, & Broyles, 1996), and these other creative types might just flourish in that environment.

If classrooms and workplaces are becoming as chaotic and socially focused as Cain proposes, it may not be an ideal environment for some people to create within. However, it may be the right kind of environment for others, and moving back to everyone sitting in neat rows and individual offices is hardly going to be the solution. There is a lot of interesting and meaningful research being done to better understand creativity (and much of it is focused specifically on the classroom and workplace environment), and I think we are better able to determine how different environments affect different kinds of individuals and their ability to create than Cain’s opinion piece suggests.

open source science…

This one is for all my Research Methods students (past and present). We spend quite a bit of time considering what the scientific process is and how it works, but I have to admit, in an attempt to “keep it simple”, I don’t always engage with some of the controversies that surround how science really works – as in what happens when you take that idealized notion of the scientific process and implement it in the “real world” where economic, social, personal, and professional pressures all work to distort the ideal. A new article in the New York Times does some of the heavy lifting for us, so we can take a look at what keeps scientists up at night…

a brief blog post about someone else’s blog post…

There is a blog titled “Research Digest: Blogging on brain and behaviour” that is affiliated with the British Psychological Society (but you already knew that based on the “funny” spelling of behavior in the title of the blog). Anyhow, they do a much better job than I at keeping up with interesting psychological findings and providing snappy, yet pithy, commentaries on the various topics they post about.

I wanted to point folks over to their page for two reasons. First, this has been an absolutely overwhelming semester in terms of time spent shaping the young minds of my students, so I have been severely hobbled in my efforts to make this a worthwhile stopover during your internet wanderings. Second, I saw they had this fun little collection of posts – comments from “famous” psychologists that expressed how psychology had impacted their lives (in less than 200 words). Reading through the various submissions is quick and fun. You get an insight on love from Robert Sternberg. Howard Gardner chimes in about creating an effective research team (hint – don’t look for mini-mes). Susan Fiske notes that a little status envy is ok as long as it is tempered with a sense of cooperation. There are quite a few of these vignettes, each providing a unique insight into the thoughts of a psychologist (even the one who said psychology was essentially useless in his day-to-day life).

steppin’ out to learn about human behavior…

Fields for Psychology – Association for Psychological Science.

This is one of those easy posts – all I essentially have to say is “go read the article linked above”. The article is written by Doug Medin, so you know it will be a quality read.

Just a bit of background though. I want to direct my Research Methods students to this article to emphasize some of the points I’ve made in class about the relative strengths and weaknesses of various research methodologies used in psychological research. I also wanted to get my Cognitive students thinking about the impact of relying nearly exclusively on laboratory experimentation in the study of cognition. Dr. Medin does a great job articulating the issues and presenting the challenges for psychological researchers. Enjoy.

psychological science and the who-dunit?

Even though it was a year ago that we (the Denison “we”) had a wonderful visit with Elizabeth Loftus, I wanted to make sure folks were still thinking about some of the issues she had discussed. You can click here for an interesting article in the on-line magazine Slate that discusses Dr. Loftus’s work, as well as some of their own inquiry into various false memory phenomena.

Once you have revisited some of the basic issues, I invite you to read the current New York Times “Room for Debate”. They have comments from five experts (including Dr. Loftus) about the use of eyewitness testimony in jury trials. This is a great chance to see psychological science making important contributions to difficult discussions. This is definitely one of those situations where there are no “easy answers”, so it is especially important that the science that can inform the debate. From what I could tell, all five experts sided with the general conclusions of a recent ruling handed down by the New Jersey Supreme Court – the ruling outlines a series of procedures for handling challenges to eyewitness evidence in criminal trials, and in a large part the decision is based firmly on conclusions drawn from scientific study of eyewitness identification. Yeah.  I’m not sure this will completely fix the occurrence of wrongful convictions based on faulty eyewitness testimony. From what I can see, it looks as though the next issue will be with the juries themselves… People have a tendency to hold onto their naive theories about the world pretty strongly, so simply requiring the judge to inform the jury about the potential for eyewitnesses to be mistaken may not be enough for some members of the jury to shift how they weigh that evidence. Still, it is refreshing to see this kind of shift occurring in how the courts think about eye witness testimony.

the “myth” of learning styles…

I meant to post on this many moons ago when the topic first came to my attention by way of a review in Psychological Science in the Public Interest. Now, with school back in session, it seems timely again, especially because NPR had a brief piece on the topic this morning… You can check out the webpage and listen to the segment here. If the short bit here piques your interest, you can read a full review of the research that they are discussing (Pashler, McDaniel, Rohrer, & Bjork (2009) – pdf available, scroll down).

The take home message is fairly straightforward. In education, there has been a rather pervasive, and seemingly intuitive, notion that different people learn best through different forms of presentation of material. I taught high school for several years and remember the focus among my colleagues on engaging all learners, whether they are auditory or visual or kinesthetic learners, because the learning style of the child made a big difference. It came up in my teaching training, but primarily during the courses dealing with pedagogy (as opposed to those that dealt with educational psychology for example). There is a huge industry offering teacher training and curricular materials to engage with the variety of learning styles. BUT… despite the intuitiveness and pervasiveness of the idea of learning styles, there has been incredibly little empirical validation of the ideas this approach. Simply, it sounds good, but so far it doesn’t seem to be true… Caveats, of course. The Pashler et al. (2009) study does impose a rigorous bar for what should be counted as evidence for the learning styles hypothesis, so even though there has been some study of the idea, the research was not methodologically sound enough to insure the validity of the findings. Also, basic research can be difficult to equate with what happens in more applied settings (like a classroom). Finally, the conclusion that there is no evidence for the notion is not the same as “the idea is totally wrong”. It could be wrong (and given that some of the most rigorous studies found no effect of learning styles indicates that might be the final conclusion), but it can also be taken as a call to action – more research needs to be done on this topic to specify what are the benefits and what sort of gains do they provide before we spend time and money trying to accommodate different learning styles.

Final note – there is good evidence that motivated learners learn better. So, if students are motivated by different instructional approaches, we should use those different approaches. And, encountering information through different modalities (visual, auditory, tactile) can enrich the learning experience for everyone. From my own experience, the best teachers recognize that varying the ways that students engage with the material is a sound practice. So, setting aside the idea of learning styles doesn’t mean that we should embrace any singular approach to learning.

Side note – students in Cognitive Psychology this semester, please note that the NPR segment includes some comments by Daniel Willingham, the esteemed author of your textbook.

supremely interesting violent issues and no respect for the data…

I’m sure you’ve heard about the recent Supreme Court ruling that struck down a California law that sought to ban the sale of violent video games to minors. As a parent, someone who played the first iterations of the first person shooter games (Doom and Quake were my favorites), and a psychologist, I find the issue interesting on several levels. Most of all though I guess I am interested in how the science played a role in this court case. Unfortunately, I have not had gobs of extra time to follow along with the arguments presented to the justices (a selection can be found here) or to scrutinize the final lengthy decision (accessible here – be warned, it is over 90 pages…). However, from my causal following of the case and an initial perusal of the final decision, an interesting issue arises.

On the one hand, there has been a notable consensus that interactive, violent video games do have a negative psychological impact on children:

• Synopsis from a news site
• FAQ from Craig Anderson, a leading researcher on the issue
• Recent meta-analysis of the research

On the other hand, the court did not feel that the science behind these claims was sufficient to support the law passed in California:

“The State’s evidence is not compelling.  California relies primarily on the research of Dr. Craig Anderson and a few other research psychologists whose studies purport to show a connection between exposure to violent video games and harmful effects on children.  These studies have been rejected by every court to consider them, and with good reason: They do not prove that violent videogames cause minors to act aggressively (which would at least be a beginning). Instead, “[n]early all of the research is based on correlation, not evidence of causation, and most of the studies suffer from significant, admitted flaws in methodology.” Video Software Dealers Assn. 556 F. 3d, at 964. They show at best some correlation between exposure to violent entertainment and minuscule real-world effects, such as children’s feeling more aggressive or making louder noises in the few minutes after playing a violent game than after playing a nonviolent game.” – quoted from p. 12-13 of Brown vs. the Entertainment Merchants Assn.

As a parent, I do think that kids have abundant and potentially detrimental exposure to images and ideas that they are simply not prepared to deal with.

As a former game player (now I am reduced to Angry Birds and WordTwist), I recognize that I had fun and no real harm came of my slaying of many beasts.

Oh yeah, I also happen to be a very big supporter of the first amendment.

But… As a psychologist, I have mixed feelings. I was very happy to see that the justices at least understood the correlation-does-not-equal-causation issue. But, I thought that there was not a full appreciation for what the pertinent research offered. The meta-analyses and survey work done was correlational – there is a connection between the time kids spend playing violent video games and real-world behaviors, e.g. school disruption, fighting, etc. They are right that this in no way proves that the video games cause those behaviors. However, a body of evidence has also accumulated from within the lab that does support a causal link. Actual experimentation has shown that when all else is held constant and the participants are randomly assigned to experimental conditions, playing a violent video game effects behaviors. Ok – the behaviors are not extreme (although the justices missed my favorite study methodology – the hot sauce paradigm), but they do show a consistent, elevated antisocial tendency among participants that play violent video games compared to those that play non-violent ones. For a whole host of (hopefully obvious) reasons, researchers can not see whether playing violent video games leads to actual violence. We have to look at a range of behaviors and see whether the interconnected patterns of behavior support the conclusion that violent video games have a negative effect. Although there are still many questions to address, and we need to remain appropriately skeptical and open to the possibility that violent video games are indeed harmless, I thought the final decision missed the importance of the empirical evidence available. The justices live in a world of legalese and case precedent. I’ll never fully understand the reasoning they employ. In this situation, I thought that they did not give science its due. They seemed to misinterpret the skepticism and caution necessary for careful scientific inquiry as a counter to the actual data. Whether putting more weight on the science would have changed the final decision – much of the discussion was appropriately focused on first amendment issues – I don’t know, but at least they could have shown more respect for data. Respect the data, man, that is what it boils down to…

of historical import with implications for today…

This brief post points to a very disheartening story, but one that all students of psychology should be familiar with. National Public Radio has a story about a man who was given a transorbital lobotomy at the age of 12. Known as the “ice-pick lobotomy”, the procedure was a particularly crude attempt at behavioral neurosurgery that was used widely during the 1950s and 1960s. Basically, a sharp instrument was inserted into the eye socket and was then agitated so that it tore away at the neural tissue (the brain) located in that area (a more complete description of the procedure can be found at the Discovery Health site). The practitioners had only a rough understanding of the brain structures that they were destroying, and little control over what disruption the procedure caused. Indeed, a later study found that the outcomes were roughly 1/3 positive, 1/3 fair, and 1/3 poor – the variability a result of a lack of control in what the procedure actually accomplished. The procedure would damage parts of the prefrontal cortex – a brain region that is structurally complex in that it connects to numerous other brain regions and is behaviorally complex in that it hosts neural processing associated with a wide range of cognitive functions (e.g. attention, problem solving, integration, emotional processing, comprehension, etc.). However, just what connections were disrupted and the degree of damage varied from patient to patient.

The subject of the NPR story, Howard Dully, has written a book about his experiences. The short version is that he was given the lobotomy essentially because he behaved like a preteen boy (moody, didn’t always listen, messy) and his family (especially his stepmother) did not want to deal with those behaviors. Howard was able to function after the procedure (which meant his outcome was not “poor” like some who had the procedure), but he was institutionalized when it did not “cure” him of the behaviors that had led to the lobotomy. He has struggled with the consequences of the damage to his prefrontal cortex throughout his life.

The lobotomy is no longer used as it once was. It was replaced largely by the introduction of psychotropic drugs in the 1960s. Still, the issues associated with the use of the lobotomy are very much alive today. When we attempt to treat people for mental illness, when we try to change behaviors, there are no quick fixes, and we need to proceed carefully. I worry at times that we have traded the ice-pick for the pill bottle, and that despite the amazing advances we have made in understanding the neural circuitry that underlies our behaviors, we are still at risk of creating new Howard Dullys. I am especially worried about this in light of the challenges our nation faces with economic instability, increases in poverty, veterans returning from war, and a focus on limiting health care costs. The treatment of mental illness has been tied to medical practices (and insurance) which has improved access for some, but it has also warped how we think about how we treat people and how they get that treatment. There was an interesting and informative special issue of The American Prospect that looked into several related issues. There is no way to go back to 1960 and return Howard Dully to his original state, but we can do all that we are able to avoid those kinds of mistakes in the future.