Okay, so this week our discussion was less conceptual and more in line with a “post publication review.”
Deflationary expectations
We love qualitative and idiographic approaches–at least in the abstract. But, in the breach we often find ourselves less than satisfied. Ironically, “scientific” work on qualitative material seems to end up reducing the material to a small number of rating scales–often fewer than 5. Maybe we should give up trying to shoe-horn qualitative material into the standard model? I remember Ken Craik (RIP) describing architectural journals where a heavily quantitative study would be followed by a simple case study of a beautiful new home. We study people. People are intrinsically interesting. Why can’t we do the same thing in our journals? This would provide a place for qualitative material and help to inspire new insights into human functioning.
Conceptual myopia
Tomkins work is wonderful and under appreciated, but it is not the only conceptual frame that could be used to organize script-like, idiographic material. That is to say, readers in our group questioned whether there was really any alternative hypothesis in this work. Pitting McAdams, or Singer, or Cervone against Tomkins might have led to a more dynamic set of hypotheses.
Methods, methods everywhere
1. There were some clarity issues with study 1 that made us less than confident in the findings. First, the nature of the prompts in study 1 were vague. Were the participants fed their own scripts? If so, the findings are not surprising or interesting. Second, the response latencies were “z-scored”. Some interpreted this as centering within person. Others interpreted this to literally mean the scores were z-scored, which is a between person transformation and would do nothing meaningful to the data. Regardless, knowing how things worked in raw and transformed data structures would be preferable.
2. The lack of independence within the Big Five is starting to bug us (study 2). We get the same thing in many different types of samples. Correlations above .5 are too high. Either we clean up our measures or walk back the myth that there are 5 independent dimensions.
3. 50 participants in each study? Really? Based on what justification? A power analysis? The low power causes problems across the board. There appears to be a bit of a desire to argue that the structure of scripts in this study can generalize or is comprehensive. We really have to stop generalizing from small groups of college students to anything. Moreover, a good number of the most interesting correlations in study 2 are ignored because there was too little power to detect them (conscientiousness and trauma-fear, for example). We understand that conducting qualitative research can be time-consuming, but that is no excuse for running underpowered studies.
pigee 
Okay, so this week our discussion was less conceptual and more in line with a “post publication review.”
Deflationary expectations
We love qualitative and idiographic approaches–at least in the abstract. But, in the breach we often find ourselves less than satisfied. Ironically, “scientific” work on qualitative material seems to end up reducing the material to a small number of rating scales–often fewer than 5. Maybe we should give up trying to shoe-horn qualitative material into the standard model? I remember Ken Craik (RIP) describing architectural journals where a heavily quantitative study would be followed by a simple case study of a beautiful new home. We study people. People are intrinsically interesting. Why can’t we do the same thing in our journals? This would provide a place for qualitative material and help to inspire new insights into human functioning.
Conceptual myopia
Tomkins work is wonderful and under appreciated, but it is not the only conceptual frame that could be used to organize script-like, idiographic material. That is to say, readers in our group questioned whether there was really any alternative hypothesis in this work. Pitting McAdams, or Singer, or Cervone against Tomkins might have led to a more dynamic set of hypotheses.
Methods, methods everywhere
1. There were some clarity issues with study 1 that made us less than confident in the findings. First, the nature of the prompts in study 1 were vague. Were the participants fed their own scripts? If so, the findings are not surprising or interesting. Second, the response latencies were “z-scored”. Some interpreted this as centering within person. Others interpreted this to literally mean the scores were z-scored, which is a between person transformation and would do nothing meaningful to the data. Regardless, knowing how things worked in raw and transformed data structures would be preferable.
2. The lack of independence within the Big Five is starting to bug us (study 2). We get the same thing in many different types of samples. Correlations above .5 are too high. Either we clean up our measures or walk back the myth that there are 5 independent dimensions.
3. 50 participants in each study? Really? Based on what justification? A power analysis? The low power causes problems across the board. There appears to be a bit of a desire to argue that the structure of scripts in this study can generalize or is comprehensive. We really have to stop generalizing from small groups of college students to anything. Moreover, a good number of the most interesting correlations in study 2 are ignored because there was too little power to detect them (conscientiousness and trauma-fear, for example). We understand that conducting qualitative research can be time-consuming, but that is no excuse for running underpowered studies.