Survey Methods Musings

In an earlier post, I suggested that survey methodologists are "data quality specialists." Our focus on " total survey error " (TSE) is, in many ways, the central defining concept of our field. This focus on data quality could be an important contribution that survey methodologists make to the emerging field of data science. But in order to make that contribution, we may need to test the fit of the TSE concept on evaluations of non-survey data. One of the sources of error in surveys that we examine in surveys is "nonresponse." Does this concept apply to other sources of data? Certainly other sources of data having missing data. But nonresponse is a specific mechanism where we sample a unit and then request data, but the unit fails to supply the data. How does this concept apply to other sources of data? I wouldn't say that Twitter data suffer from "nonresponse" due to the fact that not everyone has a Twitter account or even that not every...

A recent article by Brick and Tourangeau re-examines the data from a paper by Groves and Peytcheva (2008). The original analyses from Groves and Peytcheva were based upon 959 estimates with known variables measured on 59 surveys with varying response rates. They found very little correlation between the response rate and the bias on those 959 estimates. Brick and Tourangeau view the problem as a multi-level problem of 59 clusters (i.e. surveys) of the 959 estimates. They created for each survey a composite score based on all the bias estimates from each survey. Their results were somewhat sensitive to how the composite score was created. They do present several different ways of doing this -- simple mean, mean weighted by sample size, mean weighted by the number of estimates. Each of these study-level composite bias scores is more correlated with the response rate. They conclude: "This strongly suggests that nonresponse bias is partly a function of study-level characteristics; ...

A key question that face-to-face surveys must answer is how to balance local control against the need for centralized direction. This is an interesting issue to me. I've worked on face-to-face surveys for a long time now, and I have had discussion about this issue with many people. "Local control" means that interviewers make the key decisions about which cases to call and when to call them. They have local knowledge that helps them to optimize these decisions. For example. if they see people at home, they know that is a good time to make an attempts. They learn people's work schedules, etc. This has been the traditional practice. This may be because before computers, there was no other option. The "centralized" approach says that the central office can summarize the data across many call attempts, cases, and interviewers and come up with  an optimal policy. This centralized control might serve some quality purpose, as in our efforts here to promote more...

The title of this post is a paraphrase of a saying coined by Tolstoi. " Happy families are all alike; every unhappy family is unhappy in its own way." I'm stealing the concept to think about survey respondents.  To simplify discussion, I'll focus on two extremes. Some people are easy respondents. No matter what we do, no matter how poorly conceived, they will respond. Other people are difficult respondents. I would argue that these latter respondents are heterogenous with respect to the impact of different survey designs on them. That is, they might be more likely to respond under one design relative to another. Further, the most effective design will vary from person to person within this difficult group.  It sounds simple enough, but we don't often carry this idea into practice. For example, we often estimate a single response propensity, label a subset with low estimated propensities as difficult, and then give them all some extra thing (often more money). ...

It seems that intensive measurement is on the rise. There are a number of different kinds of things that are difficult to recall sufficiently over longer periods of time where it might be preferred to ask the question more frequently with a shorter reference period. For example, the number of alcoholic drinks consumed by day. More accurate measurements might be achieved if the questions was asked daily about the previous 24 hour period. But what is the right period of time? And how do you determine that? This might be an interesting question. The studies I've seen tend to guess at what the correct periodicity is. I think it's probably the case that it would require some experimentation to determine that, including experimentation in the lab. There are a couple of interesting wrinkles to this problem. 1. How do you set the periodicity when you measure several things that might have different periodicity? Ask the questions at the most frequent periodicity? 2. How does non...

I heard another interesting episode of the Data Skeptic podcast . They were discussing how a classifier could be assessed (episode 121). Many machine learning models are so complex that a human being can't really interpret the meaning of the model. This can lead to problems. They gave an example of a problem where they had a bunch of posts from two discussion boards. One was atheist and the other board was composed of Christians. They tried to classify each post as being from one or the other board. There was one poster who posted heavily on the Christian board. His name was Keith. Sadly, the model learned that if the person who was posting was named Keith, then they were Christian. The problem is that this isn't very useful for prediction. It's an artifact of the input data. Even cross-validation would eliminate this problem. A human being can see the issue, but a model can't. In any event, the proposed solution was to build interpretable models in local areas of t...

A few years ago, I was presenting at a conference of substantive experts. I gave an update on a progress on a survey of interest to this group. I talked about how nonresponse bias can be complex, and that the response rate might not be a good predictor of when this bias occurs -- based on Groves and Peytcheva . I was speaking with one of the researchers after my presentation, and I was surprised to hear her say that she interpreted my comments to mean that "response rates don't matter." Although that interpretation makes sense, it hadn't really occurred to me in that way until she said it. Since then, it seems like we've seen a lot of published papers and conference presentation where lowering the response rate becomes a tactic for improving the survey. Most studies taking this tactic lower the response rates for groups that tend to respond at higher rates. The purported benefit is  response set balance on known characteristics from the sampling frame is improve...

A case can be made that balanced response -- that is, achieving similar response rates across all the subgroups that can be defined using sampling frame and paradata -- will improve the quality of survey data. A paper that I was co-author on used simulation with real survey data to show that actions that improved the balance of response usually led to reduced bias in adjusted estimates. I believe the case is an empirical one. We need more studies to speak more generally about how and when this might be true. On the other hand, I worry that studies that seek balance by reducing response rates (for high-responding groups) might create some issues. I see two types of problems. First, low response rates are generally easier to achieve. It takes skills and effort to achieve high response rates. The ability to obtain high response rates, like any muscle, might be lost if it is not used. Second, if these studies justify the lower response rate by saying that estimates are not significantly ...

We talk a lot about response propensities. I'm starting to think we actually create a lot of confusion for ourselves by the way we sometimes have these discussions. First, there is a distinction between an actual and an estimated propensity. This distinction is important as our models are almost always misspecified. It is probably the case that important predictors are never observed -- for example, the mental state of the sampled person at the moment that we happen to contact them. So that the estimated propensity and true propensity are different things. The model selection choices we make can, therefore, have something of an arbitrary flavor to them. I think the choices we make should depend on the purpose of the model. We examined in a recent paper on nonresponse weighting whether call record information, especially the number of calls and refusal indicators, were useful predictors of response propensities for this purpose. It turns out that these variables were strong predic...

I've been reading a bit on mixed-mode surveys. I've noticed several discussions of web surveys and coverage error. This is a relatively recent mode, and one of the key issues has been to what extent the population has access to the internet. If someone doesn't have access to the internet, they can't complete a web survey. Everyone agrees upon that. But how do we describe the source of this error? Is it coverage or nonresponse error? In my mind, coverage error is a property of the sampling frame. If the unit is not on the sampling frame, then it is not covered. But many web surveys are general population surveys that don't have a tight association with a frame. That is, since there is not "internet" sampling frame in the way we have RDD or area probability samples. Many surveys start today from ABS sampling and then might do telephone, mail, web, or mixed-mode designs. In this case, a lack of internet access is an impediment to responding and not an imp...

I've been thinking about mixed-mode surveys a great deal over the last few months. And I notice that research publications tend to use a lot of shorthand to describe the approach -- e.g. "Mail-Telephone." Of course, they describe it in more detail, but the shorthand definition focuses on the modes. Since the shorthand describes the sequence, we end up comparing different sequences. But there are other important design features at play that make these comparisons tenuous. Of course, these other design parameters include the dosage of each mode in the sequence. Different dosages may result in different proportions of the interviews be conducted in each mode. For example, in the mail-telephone design, more mailings can increase the proportion of interviews in the mail mode. A recent article by Klausch, Schouten, and Hox includes a parameter for the mixture of modes \(\pi\). I'm concerned that we may do lots of experimentation to design a mixed mode survey that is con...

I've seen several studies that examine whether "balancing response" with respect to a set of covariates available on the frame can lead to reductions in nonresponse bias. Most of the studies indicate that more balanced response is associated with less nonresponse bias. However, there is a strategy for balancing response that worries me a bit -- reducing the response rates of the groups that have the highest response rates and, thereby, reducing the overall response rate. Why does this worry me? Several reasons. First, when does this work? We have some studies that show reductions in bias. The studies that show increases in bias might be suppressed due to publication bias. So, how are we supposed to know when it works and when it doesn't? Second, it's easy to reduce response rates. It's harder to raise them. What's worse, once we reduce response rates, how do we ever get back the skills required for obtaining higher response rates. Maybe we are simp...

I saw this paper recently that talked about how to measure and evaluate nonresponse to surveys that use short, frequently-administered instruments ("measurement-burst survey"). I've been working on a problem with data like these for a while. A complication was that the questionnaire changed based upon the intervals between measurements. For example, questions might begin, "Since you last completed this survey..." or "in the last two weeks..." depending upon the situation. Plus, panel members could choose to respond at different intervals, even though they were asked to respond at a specified interval. This made for a complex pattern of missing data. I ended up defining attrition in several ways.  The most useful was to lay out a grid over time. The survey was designed to be taken weekly, so I looked at each week over the time period to see if any reporting occured. This allowed me to how many cells in the grid were missing. But even that wasn...

Survey methods research first efforts to understand nonresponse started by looking at response rates. The focus was on finding methods that raided response rates. This approach might be useful when everyone has response propensities close to the average. The deterministic formulation of nonresponse bias may even reflect this sort of assumption.  Researchers have since looked at subgroup response rates. Also interesting, but assuming that these rates are a fixed characteristic leaves us helpless.  Now, it seems that we have begun working with an assumprton that there is heterogenous response to treatments and that we should, therefore, tailor the protocol and manipulate response propensities.   I thought this development has a parallel in clinical trials where there is a new emphasis on personalized medicine.  We still have important questions to resolve. For example, what are we trying to maximize?

A colleague sent me a link to a blog arguing that the "long survey" is dead. The blog takes the point of view that anything over 20 minutes is long. There's also a link to another blog that presents data from survey monkey surveys showing that the longer the questionnaire, the less time that is spent on each question. They don't really control for question length, etc. But it's still suggestive. In my world 20 minutes is still a short survey. But the point is still taken. There has been some research on the effect of survey length (announced) on response rates. There probably is need for more. Still, it might be time to start thinking of alternatives to improve response to long surveys. The most common is to offer a higher incentive, and thereby counteract the burden of the longer survey. Another alternative is to shorten the survey. This doesn't work if your questions are the ones getting tossed. Of course, substituting big data for elements of surveys is...

This seems to come up in a number of different ways frequently. We talk a lot about nonresponse and how it may be a selective process such that it produces biases. We might try to model this process in order to correct these biases. Online panels and 'big data' like twitter have their own selection processes. It seems that it would be important to understand these processes. Can they be captured with simple demographics? If not, what else do we need to know? I think we have done some work on survey nonresponse. I'm not sure what is known about online panels or twitter relative to this question.

I conducted a webinar on responsive design this week. I had several interesting questions. One of these was a question about responsive design and quota sampling.  The question was whether these two approaches are, in fact, different? Of course, there are similarities in that the response process is being controlled -- somewhat -- by the researchers. And this may lead to "allocating" nonresponse to some groups over others. For example, if some group is responding at higher rates, we might allocate resources to the lower responding group. Quota sampling will stop data collection for groups that have reached their quota. There are differences, however. Responsive design attempts to provide balanced response, but doesn't necessarily force that to happen. Further, responsive design is attempting to control the data collection process using a variety of approaches. Quota sampling only has one approach -- stop when the quota is full.  I do worry that there may be a conver...

I've been working on a paper about an incentive experiment that we did. It raised some interesting issues. And made me recall one of my favorite papers. Trussell and Lavrakas looked at incentives to a follow-up survey. They found that if someone had refused or been difficult to contact in the initial, screening survey, then a higher incentive was needed than for someone who had not refused or been difficult to contact. The incentives they recommend also differed by some demographic characteristics as well. I liked this example since the adaptation was linked, in part, to the paradata. These are the kinds of adaptations I have the most interest in. They require learning on the part of the survey organization that happens during data collection. I have the feeling that these kinds of adaptations can be particularly powerful since in models predicting response, it is often the case that paradata overwhelm the predictive power of demographic characteristics. There are all sorts of...

There was a debate held yesterday on "Margin of Error" in the presence of nonresponse and using non-probability samples. This is an interesting and useful discussion. In the best of circumstances, "margin of error" represents the sampling error associated with an estimate. Unfortunately, other matters often... errrrr... always interfere. The sampling mechansim is not easily identified or modeled in the case of nonprobability samples. In the case of probability samples, the nonresponse mechanism has to be modeled. Either of these situations involve some model assumptions (untestable) that are required to motivate the estimation of a margin of error. One step forward would be for people who report estimated "margins of error" to reveal all of their assumptions in their  models (weighting models for nonresponse or, in the case of nonprobability samples, selection) and describe the sampling and recruitment mechanisms sufficiently such that others can evalu...

I read this very interesting blog post by Peter Lugtig yesterday. The slides from the talk he describes are also linked to the post. He builds on an analysis of classes of nonresponders. Several distinct patterns of nonresponse are identified. The characteristics of persons in each class are then described. For example, some drop out early, some "lurk" around the survey, some stay more or less permanently. He suggests that it might be smart to identify design features that are effective for each of the groups and then tailor these features to the subgroups in an adaptive design. This makes a lot of sense. And panel studies are an attractive place to start doing this kind of work. In the panel setting, there is a lot more data available on cases. This can help in identifying subgroups. And, with repeated trials of the protocol, it may be possible to improve outcomes (response) over time. I think the hard part is creating the groups. This reminds me of a problem that I read...