Survey Methods Musings

I have read a couple of recent studies that compared early to late responders and concluded that late responders did not add anything to estimates. I have a couple of concerns. The first concern is with this approach. A simulation of this sort may not lead to the same results if you actually implement the truncated design. If interviewers know they are aiming for a lower response rate, then they may recruit differently. So, at a lower response rate, you may end up with a different set of respondents than this type of simulation would indicate. My second concern is that it is always easy to conclude that a lower response rate yields the same result. But you could imagine a long series of these steps edging up to lower and lower response rates. None of the steps changes estimates, but cumulatively they might. I have this feeling that we might need to look at studies like this in a new way. Not as an indication that it is OK to lower response rates, but as a challenge to redesign ...

My last post was about balancing response. I expressed the view that lowering response rates for subgroups to that of the lowest responding group might not be beneficial. But I left open the question of why we might benefit from balancing on covariates that we have and can use in adjustment. At AAPOR, Barry Schouten presented some results of an empirical examination of this question. Look here for a paper he has written on this question. I have some thoughts that are more theoretical or heuristic on this question. I start from the assumption that we want to improve response rates for low-responding groups. While true that we can adjust for these response rate differences, we can at least empirically verify this by improving response for some groups. Does going from a 40% to a 60% response rate for some subgroup change estimates for that group? Particularly when that movement in response rates results from a change in design, we can partially verify our assumptions that nonresponder...

I have been back from the AAPOR conference for a few days. I saw several presentations that had me thinking about the question of balancing response. By "balancing response," I mean actively trying to equalize response rates across subgroups. I can define the subgroups using data that are complete (i.e. on the sampling frame or paradata available for responders and nonresponders). I think there probably are situations where balancing response might be a bad thing. For instance, if I'm trying to balance response across two groups, persons 18-44 and 45+, and I have a 20% response rate among 18-44 year olds and a 70% response rate among 45+ persons, I might "balance response" by stopping data collection for 45+ persons when I get a 20% data collection. It's always easy to lower response rates. It might even be less expensive to do so. But I think such a strategy avoids the basic problem. How might I optimize the data collection to reduce the risk of nonresp...

I'm reading a book, Kluge , about the working of the human mind. The author takes an evolutionary perspective to explain the odd ways in which the brain functions. Newer functions were grafted onto older functions. The whole thing doesn't work very smoothly for certain situations, particularly modern social life. In one example, he cites experimental evidence (I believe, using vignettes) that says people will drive across town to save $25 on a $100 purchase, but won't drive across town to save $25 on a $1,000 purchase. It's the same savings, but different relative amounts. I tend to think that the decision to participate in surveys is not very important to anyone but the methodologist. And that's why it seems so random to us -- for example, our models predicting whether anyone will respond are so relatively poor. This book reminded me that decisions that aren't very important end up being run through mental processes that don't always produce rational ou...

The normal strategy for a publicly-released dataset is for the data collector to impute item missing values and create a single weight that accounts for probability of selection, nonresponse, and noncoverage. This weight is constructed under a model that needs to be appropriate across every statistics that could be published from these data. The model needs to be robust, and may be less efficient for some analyses. More efficient analyses are possible. But in order to do that, data users need more data. They need data for nonresponders. In some cases, they may need paradata on both responders and nonresponders. At the moment, one of the few surveys that I know of that is releasing these data is the NHIS . The European Social Survey is another. Are there others? Of course, not everyone is going to be able to use these data. And, in many cases, it won't be worth the extra effort. But it does seem like there is a mismatch between the theory and practice in this case. Not only wo...