Intercoder Reliability
I think we need to establish intercoder reliability for our content analysis. How we do it will depend on how we divide up the content analysis workload. Two potential ways of doing this are: each of us analyzing one-third of the ads, or each of us analyzing all the ads for one-third of the parameters.
I don't have the number of ads in front of me, but I think it's around 30. If there are 30 ads, we could each analyze 10 ads. Then each of us could re-analyze one ad from each of the other two folks' sections. (So we each would analyze 12
ads.) The three ads we all have analyzed in common would serve as our intercoder reliability sample. The rough Venn diagram illustrates this method.
The alternative method would involve splitting up the parameters instead of the ads. I'm not sure how many parameters we will wind up with, but the same idea would apply. We each do our section and then a small sample of the others' sections. That sample winds up being the intercoder reliability sample.
I think either method has pros and cons. In the first method, we don't analyze that many ads each, but we probably have to make several passes through each ad to make sure we don't miss anything. With the second method, we each have to analyze all the ads, but only for one-third of the code-sheet parameters, so we might have to make fewer passes through each ad.
Thoughts?
I don't have the number of ads in front of me, but I think it's around 30. If there are 30 ads, we could each analyze 10 ads. Then each of us could re-analyze one ad from each of the other two folks' sections. (So we each would analyze 12
ads.) The three ads we all have analyzed in common would serve as our intercoder reliability sample. The rough Venn diagram illustrates this method.The alternative method would involve splitting up the parameters instead of the ads. I'm not sure how many parameters we will wind up with, but the same idea would apply. We each do our section and then a small sample of the others' sections. That sample winds up being the intercoder reliability sample.
I think either method has pros and cons. In the first method, we don't analyze that many ads each, but we probably have to make several passes through each ad to make sure we don't miss anything. With the second method, we each have to analyze all the ads, but only for one-third of the code-sheet parameters, so we might have to make fewer passes through each ad.
Thoughts?
posted by Barry P. Smith | 9:55 AM
4 Comments:
Pete, by "parameters" I just mean individual items that we're coding. Although we may only have 30 or so ads (or "cases"), we have many individual items for each case. We'll wind up with several hundred individual coding item judgments. If we each code two cases in addition to our own share, we should wind up with enough of a sample for intercoder reliability. If I'm not mistaken, intercoder reliability is usually coded with about 10 percent of the total sample.
Another way of helping intercoder reliability is to use another small sample that we would all code and then run reliability stats on. We code use one or two of the party or candidate ads for this.
This analysis wouldn't be reported, but would allow us to "calibrate" ourselves before we begin coding the real sample.
Once we get our samping done, we can use Cohen's Kappa or Krippendorf's Alpha to calculate our intercoder reliability. We can also use percent agreement, but that doesn't seem to stand alone as "good enough" any more.
You just knew the quantitative guy would throw in an intercoder model. ;o) Somehow it does seem more like a real study if you have charts and graphs, though. Maybe we should come up with a model for this study. Maybe that would impress folks. It could be the "Smith3 model of videostyle communication ideology in political advertising."
Post a Comment
<< Home