Saturday, October 31, 2009

Sorry I missed you...

We've been using "Sorry I missed you" (SIMY) cards for many years in face-to-face surveys. We don't know that they work, but we keep using them anyway. I suspect that these cards are useful sometimes, not useful other times, and possibly harmful in some situations.

We haven't really collected data on the use of these cards, but interviewers do usually say something in their call notes about the use of SIMY. I've been working with data based on these notes. I'm trying to identify cases where the SIMY is useful and where it may be harmful.

We should be running some experiments with these cards in the near future. As with many of the experiments we've been running in face-to-face surveys, we have a double burden of proof.
  • First, will interviewers respond to recommendations delivered via our computerized sample management system.
  • Second, if they follow the recommendation, does it help.

Hopefully, we'll have some evidence on one or both of these questions soon.

Thursday, October 29, 2009

Presenting Results: Adaptive Design for Telephone Surveys

I'll be presenting results from the adaptive call scheduling experiment on Monday, November 2nd at the FCSM Research Conference. The results were promising, at least for the calls governed by the experimental protocol. The following table summarizes the results:


The next step is to extend the experimental protocol to the calls that were not involved with the experiment (mainly refusal conversion calls), and to attempt this with a face-to-face survey.

Thursday, October 22, 2009

New Measures for Monitoring the Quality of Survey Data

Many surveys work to a sample size/response rate requirement. The contract specified the target response rate. The survey organization works hard to meet that target. In this context, the logical thing for the survey organization to do is to focus on interviewing the easiest cases to interview.

The underlying assumption of this approach is that a higher response rate leads to a lower risk of bias. Theoretically, this need not be true. Empirically, there have been a number of recent studies where this is not true (see Groves and Peytcheva, POQ 2008). So what are we supposed to do?

The search is on for alternative indicators. Bob Groves convened a meeting in Ann Arbor to discuss the issue two years ago. The result was a short list of indicators that might be used to evaluate the quality of survey data (see the October 2007 issue of Survey Practice: http://www.surveypractice.org/).

Now these new measures are starting to appear! Barry Schouten, Fannie Cobben, and Jelke Bethlehem have an article in the June 2009 issue of Survey Methodology. They develop an indicator they call the "R-Indicator." They describe this as an "indicator... for the similarity between the response to a survey and the sample or the population under investigation."

I've been working on using the fraction of missing information, a concept from methods for missing data, as an indicator of the risk of nonresponse bias. I have an article accepted for publication that should be appearing soon.

Monday, October 12, 2009

How can we estimate "e"?

AAPOR defines response rates that include an adjustment factor for cases that have unknown eligibility at the end of the survey. They call the factor "e". Typically, people use the eligibility rate from the part of the sample where this variable (eligible=yes/no) is observed. This estimate is sometimes called the CASRO estimate of e.

But in a telephone survey, this estimate of "e" is likely to be biased upwards for the unknown part of the sample. Many of the cases that are never contacted are not households. They are simply numbers that will ring when dialed, but are not assigned to a household. These cases are never involved in estimates of "e".

A paper in POQ (Brick and Montaquila, 2002) described an alternative method of estimating e. They use a survival model. This lowers estimates of e relative to the CASRO method. But it's still upwardly biased since many of the noncontacts could never be contacted.

I like the survival method since it's closer to reality. But, for other reasons, I started imputing eligibility. I like this approach as it develops a nice range of estimates. And it allows great flexibility. It's very easy to include covariates in the model. It's not as easy to include covariates in the survival model.

Wednesday, October 7, 2009

Operationalizing Experimental Design

I had a useful conversation with project managers about the call scheduling experiment for a face-to-face survey. My proposal for an experiment had been to randomize at the line level. That way, interviewers would have both experimental and control cases in their sample. The project managers felt that this might lead to inefficient trips. In other words, interviewers might follow the recommendation and then ignore cases without a recommendation or go to cases that are very far apart in distance while not visiting cases that are closer but do not have a recommendation.

The experiment is certainly more clean if the randomization occurs at the line level and not the interviewer level, but I certainly wouldn't want to create inefficiencies. One goal is to improve efficiency (another goal is to increase our ability to target cases). I thought training interviewers to use the recommendation as one piece of information while planning their trips. But maybe that wouldn't work. I'll keep thinking about this one...

Followers