Survey Methods Musings

We've been using "Sorry I missed you" (SIMY) cards for many years in face-to-face surveys. We don't know that they work, but we keep using them anyway. I suspect that these cards are useful sometimes, not useful other times, and possibly harmful in some situations. We haven't really collected data on the use of these cards, but interviewers do usually say something in their call notes about the use of SIMY. I've been working with data based on these notes. I'm trying to identify cases where the SIMY is useful and where it may be harmful. We should be running some experiments with these cards in the near future. As with many of the experiments we've been running in face-to-face surveys, we have a double burden of proof. First, will interviewers respond to recommendations delivered via our computerized sample management system. Second, if they follow the recommendation, does it help. Hopefully, we'll have some evidence on one or both of these...

I'll be presenting results from the adaptive call scheduling experiment on Monday, November 2nd at the FCSM Research Conference. The results were promising, at least for the calls governed by the experimental protocol. The following table summarizes the results: The next step is to extend the experimental protocol to the calls that were not involved with the experiment (mainly refusal conversion calls), and to attempt this with a face-to-face survey.

Many surveys work to a sample size/response rate requirement. The contract specified the target response rate. The survey organization works hard to meet that target. In this context, the logical thing for the survey organization to do is to focus on interviewing the easiest cases to interview. The underlying assumption of this approach is that a higher response rate leads to a lower risk of bias. Theoretically, this need not be true. Empirically, there have been a number of recent studies where this is not true (see Groves and Peytcheva, POQ 2008). So what are we supposed to do? The search is on for alternative indicators. Bob Groves convened a meeting in Ann Arbor to discuss the issue two years ago. The result was a short list of indicators that might be used to evaluate the quality of survey data (see the October 2007 issue of Survey Practice : http://www.surveypractice.org/ ). Now these new measures are starting to appear! Barry Schouten, Fannie Cobben, and Jelke Bethlehem ha...

AAPOR defines response rates that include an adjustment factor for cases that have unknown eligibility at the end of the survey. They call the factor "e". Typically, people use the eligibility rate from the part of the sample where this variable (eligible=yes/no) is observed. This estimate is sometimes called the CASRO estimate of e. But in a telephone survey, this estimate of "e" is likely to be biased upwards for the unknown part of the sample. Many of the cases that are never contacted are not households. They are simply numbers that will ring when dialed, but are not assigned to a household. These cases are never involved in estimates of "e". A paper in POQ (Brick and Montaquila, 2002) described an alternative method of estimating e. They use a survival model. This lowers estimates of e relative to the CASRO method. But it's still upwardly biased since many of the noncontacts could never be contacted. I like the survival method since it's clos...

I had a useful conversation with project managers about the call scheduling experiment for a face-to-face survey. My proposal for an experiment had been to randomize at the line level. That way, interviewers would have both experimental and control cases in their sample. The project managers felt that this might lead to inefficient trips. In other words, interviewers might follow the recommendation and then ignore cases without a recommendation or go to cases that are very far apart in distance while not visiting cases that are closer but do not have a recommendation. The experiment is certainly more clean if the randomization occurs at the line level and not the interviewer level, but I certainly wouldn't want to create inefficiencies. One goal is to improve efficiency (another goal is to increase our ability to target cases). I thought training interviewers to use the recommendation as one piece of information while planning their trips. But maybe that wouldn't work. I'll...