Friday, March 23, 2012

Sequence in a Mixed Mode Survey, II

There was another interesting thing that happened in our mixed mode that varied the sequences of modes. As a reminder, the survey was a screening survey. It identified households with an eligible person. An interviewer later returned to eligible households to complete a 'main' interview.

We found that eligible households in locked buildings that had received the FtF-Mail treatment on the screening interview responded at higher rates to the main interview. Better even than Mail-FtF. The results were significant, even when accounting for the clustering in the sample. It may be that the FtF-Mail sequence displays our earnestness to the respondent more clearly. It would be nice to replicate these results.

Friday, March 16, 2012

Sequence in a Mixed Mode Survey

We recently tried to implement a mixed mode approach to a large screening survey that is usually done face-to-face (FtF). We wanted to be sure that modes with lower response rates didn't "contaminate" the sample. If not, there may be cost savings in using a mailed version of the screening survey.

We varied the sequence of the mixed mode approach to see if that had any impact. We did FtF-Mail, Mail-FtF, and FtF.

We also monitored response rates to the main interview, which is conducted only with eligible persons. The response rates to the screening survey were very similar across the three arms of the trial. But it turns out that one mode combination did better on the main interview response rate with cases in locked buildings. That mode combination was FtF-Mail. This might be fluke, but definitely worth exploring.

Friday, March 9, 2012

The Long Perspective on Call Scheduling

I recently went back and read papers written during the early days of the development of CATI. I found this very interesting quote from an article by J. Merrill Shanks from 1983:

“Among the procedures that are supported by (or related to) CATI systems, none has proved more difficult to discuss than the algorithms or options available for management of interviewers’ time and the scheduling or assignment of actual calls to specific interviewers. Most observers agree that computer-assisted systems can yield improvements in the efficiency or productivity of interviewer labor by scheduling the calls required to contact respondents in a particular household across an appropriately designed search pattern, and by keeping track of the ‘match’ between staff availability and the schedule of calls to be made” (Shanks, 1983, p. 133).
It seems like today people would agree that this is a "difficult to discuss" problem. I'm not sure that there is a sense that there are large gains out there to be had.

It's not as if there is a lot of published research in this area. My fear is that most of the experience in this area is undocumented. That different organizations have had a long evolution of gradually tuning their procedures. Each step might represent a minor improvement. Over the long haul, this might lead to a significantly better system. But no one person has enough of a view of the process to really see that. Hmm.

Friday, March 2, 2012

Interviewer Variability

In face-to-face surveys, interviewers play a very important role. They largely determine when they work, at which times they call cases, and how to address the concerns of sampled persons. Several studies have looked at the variability that interviewers have in achieving contact and cooperation. Durrant and Steele (2009) provide a particularly good example of this.

It is also the case that interviewers have only a partial view of the data being collected. They cannot detect imbalances that may occur at higher levels of aggregation.

For these reasons, it seems like controlling this variability is a useful goal. This may be done through improved training (as suggested by Groves and McGonagle, 2001), or by providing specific recommendations for actions to interviewers.

We have had some success in this area. In NSFG Cycle 7, we ran a series of 16 experiments that asked interviewers to prioritize a set of specified cases over other cases in their workload. The results were positive in that in each experiment the cases that were prioritized received more calls than those that were not.

On the other hand, in a separate experiment, we asked interviewers to attempt contact with households at a specified time. In this randomized experiment, did not follow the recommendation.

It seems that the question of why some recommendation/requests will be followed, while others will not is an important one. If these methods are to be employed successfully, we need to know when and how they work.