Survey Methods Musings

In my last post, I talked about the costs of collecting interviewer observations. These observations have to reduce total survey error (likely nonresponse error) in order to justify this cost. The original definition of paradata was that it was a by-product of the data collection process. Computers stored information on keystrokes at basically no extra cost. Call records were necessary to manage the data collection process, but they were also found to be useful for developing nonresponse adjustments and other purposes. At some point, interviewer observations became paradata. I think many of these observations started out as information that interviewers recorded for their own purposes. For example, listers would record if there were any barriers to entering the segment (e.g. gated communities or locked buildings) so that interviewers would know about that before traveling to the segment. These could be thought of as no cost. But we have added a lot of observations that the interv...

At the recent Joint Statistical Meetings I was part of an interesting discussion on paradata and nonresponse. At one point, someone reported that their survey had reduced the number of observations being recorded by interviewers. They said the observations were costly in a double sense. First, it takes interviewer time to complete them. Second, it diverts attention from the task of gathering data from persons willing to respond to the survey. I have to say that we certainly haven't done a very good job of determining the cost of these interviewer observations. First, we could look at keystroke files to estimate the costs. This is likely to be an incomplete picture as there are times when observations are entered later (e.g. after the interviewer returns home). Second, we could examine the question of whether these observations reduce the effectiveness of interviewers in other errors. This would require experiments of some sort. Once these costs are understood, then we can place...