This week’s reading for PIG-IE is
Kandler et al (2012). Left or Right? Sources of political orientation: The roles of genetic factors, cultural transmission, assortative mating, and personality. Journal of Personality and Social Psychology, 102, 633-645.
Enjoy and leave a comment if you like.
It seems a critical mass of papers, blogs, articles, and comments has hit the field of social and personality psychology in the last few months. So much so, that I thought it would be constructive to catalogue everything that has emerged from Lehrer, to Bem, to Stapel, to Bargh, to Simonsohn, and on to Francis (borrowing freely from Sanjay Srivastava’s hard work). I would like this post to be a resource of sorts for the whole sordid affair, which can be used for methods courses taught in the not too distant future. So, if I’ve missed anything important please send the link or paper my way.
The denouement: Lehrer & Bem
The story probably starts with Lehrer’s piece on the decline effect, which was followed closely by the news of Bem’s article on the existence of ESP. Both pieces caused consternation in psychology in general and in social psychology in particular. Bem’s article resulted in a veritable storm of criticism. One of the most thorough critiques of Bem’s study can be found on the Skeptical Inquirer blog. One of the most even-handed reactions can be found on the citation needed blog by Tal Yarkoni.
Just recently, Ritchie et al (2012) published their failure to replicate Bem’s research. The paper contains a nice back and forth between the authors and Bem in the comment section.
Jumping the shark: Stapel
If anyone possessed any modicum of ESP, it would have been nice if they would have told us ahead of time about the Stapel affair. Fortuitously, it came right on the heels of Bem. I think, no I hope that the juxtaposition of these two events is fortuitous because the combination has created a motivation for some methodological navel gazing which might make our science a little better. I think if either one of these events happened in isolation, it would have been easy to blithely forget about them.
For those following the Stapel affair the committee in charge of evaluating his research has posted a preliminary report identifying an initial list of his fraudulant studies. For another perspective on Stapel and how it might affect our reputation, see the recent post on the Gene Expression blog at Discover magazine. The comment section is especially illuminating or depressing, depending on your perspective. For a relatively complete compilation of Stapel-related information see Sanjay Srivastava’s post on his blog site, the Hardest Science
Throwing salt in the wound: Bargh & Doyen
Just when things seemed to be settling down a bit, a new kerfuffle erupted over on PloS-ONE and Psychology Today. Doyen et al. (2012) reported a failure to replicate a classic study by John Bargh. The findings were highlighted in Ed Yong’s blog at Discover magazine. Bargh responded with an unsolicited review of Doyen and Yong’s missives that included a harsh take on the PloS-ONE model of publishing. A number of people, including our own Dan Simons, have described Bargh’s post as a prime example of how not to respond to your critics. To his credit Ed Yong responded in a most polite way. And, finally Bargh posted some additional thoughts that did little to assuage his critics. Again, read the comments. They are more edifying that the original posts. For a comprehensive run down on the whole affair, see Cedar Reiner’s post.
Another criticism, another dust up: Simonsohn & Schwarz
Simultaneous to the Bargh brouhaha, Simonsohn and Schwarz got into a hissy on the SPSP blog site. Simonsohn along with Simmons & Nelson are co-authors on the now infamous Psych Science paper on false positives in psychology. Apparently, Schwarz took umbrage with some of the points made in the symposium and expressed his views most passionately at the recent SPSP meeting. This resulted in a rather uncomfortable exchange on the SPSP discussion site. The gist of Schwarz’s comments appear to be that Simonsohn and colleagues have overstated the problem. I hope that one conclusion drawn from the litany of issues posted here is that Simmons et al, might have underestimated things.
No-one expects the Spanish Inquisition: Greg Francis
In what appears to be a fit of moral indignation, Greg Francis from Purdue University has produced a string of articles that apply Ioannide’s p-hacking analysis to a variety of experimental studies. One paper calls into question Balcetis and Dunning’s research. Balcetis and Dunning respond in the comment section–be sure to read their response as it is well crafted. The second paper takes on Bem’s ESP work and Meissner & Brigham’s work on verbal overshadowing. One has to wonder what would happen if the Ioannide’s analytical technique would be used systematically across all of our journal articles.
Psychologist heal thyself
Along with the Simmons et al paper numerous new papers are emerging that analyze the issues being raised in these forums. There is the highly controversial John et al (in press) paper which attempts to identify the base rate of Questionable Research Practices. Lebel & Peters use the Bem article as a case study for identifying deficiencies in modal research practice, for example, an overemphasis on conceptual rather than close replications. Fanelli has written about the rise of positive results across time and how positive results correlate with the perceived status of the field of scientific inquiry. Ledgerwood & Sherman fault the rise of the short report (especially cutesy, press happy nuggets) while Burtamini & Munafo show that short report format is correlated with publication bias. The editors of Psychological Science see no problem with their format. These papers clearly point to the fact that the status quo in our field–and other fields for that matter–is unacceptable.
Replication: The road to redemption?
Beyond identifying the problematic research and the problematic research methods we employ, some individuals have started to propose solutions to this mess. Yours truly proposed the Journal of Reproducible Results in (put your field here: Personality Psychology, Social Psychology, Neuroscience, Medicine, Genetics, etc.). Sanjay Srivastava has written about how replications could be handled by journals and whether journals should be groundbreaking or definitive. Roediger wrote an eloquent paean to the varieties of replication that we can and should employ and reward in our field. Pashler and friends have attempted to popularize www.psychfiledrawer.org as a site to record your attempts to replicate published research. Purveyors of the site have nominated the top 20 articles that should be replicated. Similarly, Brian Nosek has started the Open Science Consortium dedicated to replicating psychological experiments published in our leading journals. All of these efforts highlight the need to value the cornerstone of science–direct replication. It may not be creative, but it is necessary.
Where do we go from here?
The goal of this post is simple–collect together in one place the majority of methodological issues that have been raised in the past two years in social/personality psychology. From my vantage point, the gestalt of the information is straightforward–we have a problem. Moreover, we have enough of a problem that we should do something about it. My initial inclination was to end by saying that all of these issues deserve discussion. To be honest, I think they deserve action.
Addendum I: The crows come home to roost
So much for things settling down over the summer. Close on the heels of updated reports on the Stapel affair, we have another report of “fraud” at Erasmus University. Once again, an experimental social psychologist, Dirk Smeesters, has gotten in hot water and has been forced to resign. This time, he apparently did not fake data outright, but did what he believes many other colleagues do–resort to questionable research practices in order to compile a seemingly convincing set of statistically significant findings. Here’s the quote from the report:
He “repeatedly indicates that the culture in his field and his department is such that he does not feel personally responsible, and is convinced that in the area of marketing and (to a lesser extent) social psychology, many consciously leave out data to reach significance without saying so.”
The new affair is documented nicely here at Retraction Watch.
The interesting, and troubling aspect of this new case is that Smeesters equates his actions with Standard Operation Procedure (SOP) in social psychology. As Ed Yong notes, if these methods are SOP in psychology this may be the “first flake of the avalanche” of individuals having their QRPs rooted out by relatively simple post hoc analyses of their research reports.
Brent W. Roberts
Inspired by Rich Lucas’s recent analysis of the type 1 error rates underlying Daryl Bem’s approach to running his ESP research, we decided to write a little missive ourselves. Please comment.
Psychological Science has a Serious Problem: The Bem Fallout
Sometime in the near future Bem’s article showing evidence for the existence of ESP will be published in the Journal of Personality and Social Psychology, the flagship journal of our field. As could be expected an uproar has occurred over this eventuality. To our distress, the hue and cry has focused mostly on the topic itself and not the reasons for why an article like this is possible. Our thesis is simple. This article was a fait accompli because it is the result of the standard operating procedures of psychological science. Those operating procedures are the problem. The problem is not the peer review process per se or the reality (or lack thereof) of ESP.
The problem we have is one of evolutionary sociology. In the words of one of our esteemed colleagues, a publication is only worthy if “it shows something” (i.e., costly signaling). In no uncertain terms “showing something” means showing that something is statistically significant. Thus to be successful in our field we must publish a string of articles that reveal statistically significant results, even if the point of the article (e.g., ESP) is fanciful. We believe that this has led to practices that are widely accepted that undermine our ability to create a foundation of reliable scientific findings.
From this standpoint, the publication of Bem’s article is an indictment of our widely held practices. Therefore, the action editor and reviewers are not at fault. We are. As long as we implicitly or explicitly condone these practices, we should consider much of our science no different than Bem’s article. Moreover, our current concern is if we fail to address the implications of Bem’s article constructively, the effect will be to further marginalize our field at a time when many important institutions, such as the National Institutes of Health, already question the usefulness of our scientific contributions.
To this end, we have done two things. First, below, we diagnose the problem by documenting the practices that we believe are the basis to the publication of an article like Bem’s. We consider the list only provisional. Please add to it or modify it to your liking. Second, we have proposed a new journal that will be built on evaluation practices that will help to overcome our current inertia to change our standard practices. Preferably, our current journals would change their policies to endorse these practices. We believe that if a journal is created that uses these practices and/or an existing journal adopts these practices it will, with time, become the most important outlet for our science, trumping even JPSP.
Problematic Practices in Psychological Science
1. Null Hypothesis Significance Testing (NHST): We compare our results to the null hypothesis despite the fact that we are almost never testing something that has not been tested before. We should be comparing our results to previous results and testing whether they differ from what has been found before, not from the null.
2. Not valuing the Null Hypothesis: Our explicit goal is to produce “knowledge”. Our system for creating knowledge is showing that something is “statistically significant”. This creates a situation where we do not value null results, which are intrinsically necessary for creating knowledge because null results show us where our ideas and/or constructs do not work. Without the figure-ground patterns of null and confirmatory findings we have a cacophony of findings that add up to nothing.
3. Data churning: Running repeated experiments until you get a hit; surfing the correlation matrix in search of a finding.
4. Not replicating.
5. Not reporting your lack of replication.
6. Peeking: Checking data as it is being collected and discovering “significant” effects along the way.
7. HARKing: Hypothesizing After the Results are Known.
8. Data Topiary: The process of pruning insignificant findings or findings that contradict hypotheses followed closely by changing ones hypotheses (see HARKing).
9. Outcome fragmentation grenade: Collect so many outcomes that something is bound to hit.
10. Betting against the house: Running underpowered studies, which means you have a 50:50 chance of finding an effect even if it exists.
A Modest Proposal: The Journal of Reproducible Results in Social and Personality Psychology
Feeling guilty about not making it to Pig-E? Maybe you shouldn’t worry about it. According to Rick Robin’s latest JPSP article (he’s replaced Ed Diener as the monthly JPSP columnist), feeling guilty may not be so bad. On the other hand, don’t feel ashamed. We don’t care that much, and it might be bad for you. If you want to learn more, come to the impromptu Pig-E meeting this Monday, Dec. 13
Remember that “little” paper by Almlund, Duckworth, Heckman, & Kautz that we read at the beginning of the semester–roughly 90 pages or so? It has been revised. It is now 248 pages. The authors are taking comments:
Here’s a paper by Peter Molenaar presenting a model of personality based on the explicit examination of within-person variation rather than between-person variation. We’ll discuss this Monday, 11/15
small behavioral ecologist’s perspective on personality
.63. That is the correlation between a set of “exploratory” behaviors measured over ten occasions in one social group with a set of “exploratory” behaviors measured in over ten occasions in a separate social group. Seems reasonable, until you find out that the correlation is between exploratory behavior of stickle back fish, not people. Sound interesting? Come to the PIG-E meeting Monday at 2 and find out more from Alison Bell’s lab.
pigee 