The title of this post is a paraphrase of a saying coined by Tolstoi. "Happy families are all alike; every unhappy family is unhappy in its own way." I'm stealing the concept to think about survey respondents.
To simplify discussion, I'll focus on two extremes. Some people are easy respondents. No matter what we do, no matter how poorly conceived, they will respond. Other people are difficult respondents. I would argue that these latter respondents are heterogenous with respect to the impact of different survey designs on them. That is, they might be more likely to respond under one design relative to another. Further, the most effective design will vary from person to person within this difficult group.
It sounds simple enough, but we don't often carry this idea into practice. For example, we often estimate a single response propensity, label a subset with low estimated propensities as difficult, and then give them all some extra thing (often more money).
I suspect we would often do better finding several explanations for why the difficult cases are difficult, address each of these with a potentially different treatment, and then assign those treatments to the cases for which they are most useful. We could measure useful in a number of ways including increases in response rates, or some optimization of a sample balance measure for a fixed budget. In any event, taking that into progress implies some extra steps. But I think it may be worth it.
To simplify discussion, I'll focus on two extremes. Some people are easy respondents. No matter what we do, no matter how poorly conceived, they will respond. Other people are difficult respondents. I would argue that these latter respondents are heterogenous with respect to the impact of different survey designs on them. That is, they might be more likely to respond under one design relative to another. Further, the most effective design will vary from person to person within this difficult group.
It sounds simple enough, but we don't often carry this idea into practice. For example, we often estimate a single response propensity, label a subset with low estimated propensities as difficult, and then give them all some extra thing (often more money).
I suspect we would often do better finding several explanations for why the difficult cases are difficult, address each of these with a potentially different treatment, and then assign those treatments to the cases for which they are most useful. We could measure useful in a number of ways including increases in response rates, or some optimization of a sample balance measure for a fixed budget. In any event, taking that into progress implies some extra steps. But I think it may be worth it.
Comments
Post a Comment