Friday, August 21, 2015

Balancing response... without simply retreating

I've seen several studies that examine whether "balancing response" with respect to a set of covariates available on the frame can lead to reductions in nonresponse bias. Most of the studies indicate that more balanced response is associated with less nonresponse bias.

However, there is a strategy for balancing response that worries me a bit -- reducing the response rates of the groups that have the highest response rates and, thereby, reducing the overall response rate.

Why does this worry me? Several reasons. First, when does this work? We have some studies that show reductions in bias. The studies that show increases in bias might be suppressed due to publication bias. So, how are we supposed to know when it works and when it doesn't?

Second, it's easy to reduce response rates. It's harder to raise them. What's worse, once we reduce response rates, how do we ever get back the skills required for obtaining higher response rates.

Maybe we are simply living in a period of retreat. But I'd like to think we can take some (small?) steps forward. For example, Luiten and Schouten increased balance of response without reducing response rate and for only a small (2.6%) increase in budget. They don't attach sampling error to the cost estimates, but I'm guessing that in the next iteration of the survey, the costs might have been equal. That's at least a step forward.

Friday, August 14, 2015

Timing of the Mode Switch

I just got back from JSM where I presented the results of an experiment that varied the timing of the mode switch in a web-telephone survey. I'm not going to talk about the results of the experiment in this post, just the premise.

The concern that motivated the experiment had to do with the possibility that longer delays before switching modes could have adverse effects on response rates. This could happen for several reasons.
  • If there is pre-notification, then the effect of the prenote on response to the second mode might be reduced with longer delays before switching. 
  • If the first mode is annoying in some way, it can diminish the effectiveness of the second mode.
The latter case is particularly interesting to me. It points to the ways that different treatment sequences can have different levels of effectiveness. We saw an impact like this in an experiment we did of two sequences of modes for a screening survey. The two sequences functioned about the same in terms of response to the screening survey. But among the eligibles, one of the screening sequences led to higher response rates on a follow-up, in-depth interview.  Lynn  found a similar interaction in a panel survey, where earlier modes were related to response at later waves.

I'm interested in these sorts of interactive effects between treatments as it makes it seem that we should be thinking about the sequence of treatments. That is, the full context of each component of the sequence.