Survey Methods Musings

Linking surveys and other sources of data is not a new idea. This has been around for a long time. It's useful in many situations. For example, when respondents would have a difficult time supplying the information (for example, exact income information). Much of the previous research on linkage has focused on either the ability to link data, possibly in a probabilistic fashion; or there have been examinations of biases associated with the willingness to consent to linkage. It seems that new questions are emerging with the pervasiveness of data generated by devices, especially smart phones. I read an interesting article by Melanie Revilla and colleagues about trying to collect data from a tracking application that people install on their devices. They examine how the "meter" as they call the application might be incompletely covering the sample. For example, persons might have multiple devices and only install it on some of them. Or, persons might share devices and no...

I used to work on a CATI system (DOS-based) that allowed supervisors to release cases for calling through an override mechanism. That is, the calling algorithm had certain rules that kept cases out of the calling queue at certain times. The main thing was if something had been called and was a "ring-no-answer," then the system wouldn't allow it to be called (i.e. placed in the calling queue) until 4 hours had passed. But supervisors could override this and release cases for calling on a case-by-case basis. This was handy -- when sample ran out, supervisors could release more cases that didn't fall within the calling parameters. This kept interviewers busy dialing. Recently, I've started to think about the other side of such practices. That is, it is more difficult to specify the protocol that should be applied when these exceptions are allowed. Obviously, if the protocol is not calling a case less than four hours after a ring-no-answer, then the software explicit...

Just continuing the thought from the previous post... Some examples of controlling the variability don't make much sense. For instance, there is no real difference between a response rate of 69% and one of 70%. Except for the largest of samples. Yet, there is often a "face validity" claim that there is a big difference in that 70% is an important line to cross. However, for survey costs, it can be a big difference if the budgeted amount is $1,000,000 and the actual cost is $1,015,000. Although this is roughly the same proportionate difference as the response rates, going over a budget can have many negative consequences. In this case, controlling the variability can be critical. Although the costs might be "noise" in some sense, they are real.

At the Joint Statistical Meetings, I went to a session on responsive and adaptive design. One of the speakers, Barry Schouten, contrasted responsive and adaptive designs. One of the contrasts was that responsive design was concerned with controlling short-term fluctuations in outcomes such as response rates. This got me thinking. I think the idea is that responsive design will respond to the current data, which includes some sampling error. In fact, it's possible that sampling error could be the sole driver of responsive design interventions in some cases. I don't think this is usually the case, but it certainly is part of what responsive designs might do. At first, this seemed like a bad feature. One could imagine that all responsive design interventions should include a feature that accounts for sampling error. For instance, decision rules that attain a level of statistical significance. We've implemented some like that. On the other hand, sometimes controlling samp...

A key question that face-to-face surveys must answer is how to balance local control against the need for centralized direction. This is an interesting issue to me. I've worked on face-to-face surveys for a long time now, and I have had discussion about this issue with many people. "Local control" means that interviewers make the key decisions about which cases to call and when to call them. They have local knowledge that helps them to optimize these decisions. For example. if they see people at home, they know that is a good time to make an attempts. They learn people's work schedules, etc. This has been the traditional practice. This may be because before computers, there was no other option. The "centralized" approach says that the central office can summarize the data across many call attempts, cases, and interviewers and come up with  an optimal policy. This centralized control might serve some quality purpose, as in our efforts here to promote more...

The title of this post is a paraphrase of a saying coined by Tolstoi. " Happy families are all alike; every unhappy family is unhappy in its own way." I'm stealing the concept to think about survey respondents.  To simplify discussion, I'll focus on two extremes. Some people are easy respondents. No matter what we do, no matter how poorly conceived, they will respond. Other people are difficult respondents. I would argue that these latter respondents are heterogenous with respect to the impact of different survey designs on them. That is, they might be more likely to respond under one design relative to another. Further, the most effective design will vary from person to person within this difficult group.  It sounds simple enough, but we don't often carry this idea into practice. For example, we often estimate a single response propensity, label a subset with low estimated propensities as difficult, and then give them all some extra thing (often more money). ...

I have seen this issue on several different projects. So I'm not calling out anyone in particular. I keep running into this issue. Repeated cross-sectional surveys are the most glaring example, but I think it happens other places as well. The issue is that with a slow decline, it's difficult to diagnose the source of the problem. If everything is just a little bit more difficult (i.e. if contacting persons, convincing people to list a household, finding the selected person, convincing them to do the survey, and so on), then it's difficult to identify solutions. One issue that this sometimes creates is that we keep adding a little more effort each time to try to counteract the decline. A few additional more calls. A slightly longer field period. We don't then search for qualitatively different solutions. That's not to say that we shouldn't make the small changes. Rather, that they might need to be combined with longer term planning for larger changes. That...

We recently did an experiment with incentives on a face-to-face survey. As one aspect of the evaluation of the experiment, we looked at the costs associated with each treatment (i.e. different incentive amounts). The costs are a bit complicated to parse out. The incentive amount is easy, but the interviewer time is hard. Interviewers record their time for at the day level, not at the housing unit level. So it's difficult to determine how much a call attempt costs. Even if we had accurate data on the time spent making the call attempt, there would still be all the travel time from the interviewer's home to the area segment. If I could accurately calculate that, how would I spread it across the cost of call attempts? This might not matter if all I'm interested in is calculating the marginal cost of adding an attempt to a visit to an area segment. But if I want to evaluate a treatment -- like the incentive experiment -- I need to account for all the interviewer costs, as b...

I'm thinking again about experiments that we run. Yes, they are usually messy. In my last post, I talked about the inherent messiness of survey experiments that is due to the fact that surveys have many design features to consider. And these features may interact in ways that mean we can't simply pull out an experiment on a single feature and generalize the result to other surveys. But I started thinking about other problems we have with experiments. I think another big issue is that methodological experiments are often run as "add-ons" to larger surveys. It's hard to obtain funding to run a survey just to do a methodological experiment. So, we add our experiments to existing surveys. The problem is that this approach usually creates a limitation. The experiments can't risk creating a problem for the survey. In other words, they can't lead to reductions in response rates or threaten other targets that are associated with the main (i.e. non-methodologic...

A few years ago, I was presenting at a conference of substantive experts. I gave an update on a progress on a survey of interest to this group. I talked about how nonresponse bias can be complex, and that the response rate might not be a good predictor of when this bias occurs -- based on Groves and Peytcheva . I was speaking with one of the researchers after my presentation, and I was surprised to hear her say that she interpreted my comments to mean that "response rates don't matter." Although that interpretation makes sense, it hadn't really occurred to me in that way until she said it. Since then, it seems like we've seen a lot of published papers and conference presentation where lowering the response rate becomes a tactic for improving the survey. Most studies taking this tactic lower the response rates for groups that tend to respond at higher rates. The purported benefit is  response set balance on known characteristics from the sampling frame is improve...

I've seen several studies that examine whether "balancing response" with respect to a set of covariates available on the frame can lead to reductions in nonresponse bias. Most of the studies indicate that more balanced response is associated with less nonresponse bias. However, there is a strategy for balancing response that worries me a bit -- reducing the response rates of the groups that have the highest response rates and, thereby, reducing the overall response rate. Why does this worry me? Several reasons. First, when does this work? We have some studies that show reductions in bias. The studies that show increases in bias might be suppressed due to publication bias. So, how are we supposed to know when it works and when it doesn't? Second, it's easy to reduce response rates. It's harder to raise them. What's worse, once we reduce response rates, how do we ever get back the skills required for obtaining higher response rates. Maybe we are simp...

A colleague sent me a link to a blog arguing that the "long survey" is dead. The blog takes the point of view that anything over 20 minutes is long. There's also a link to another blog that presents data from survey monkey surveys showing that the longer the questionnaire, the less time that is spent on each question. They don't really control for question length, etc. But it's still suggestive. In my world 20 minutes is still a short survey. But the point is still taken. There has been some research on the effect of survey length (announced) on response rates. There probably is need for more. Still, it might be time to start thinking of alternatives to improve response to long surveys. The most common is to offer a higher incentive, and thereby counteract the burden of the longer survey. Another alternative is to shorten the survey. This doesn't work if your questions are the ones getting tossed. Of course, substituting big data for elements of surveys is...

I missed Dr. Groves blog post on this topic. It is an interesting perspective on the strengths and weaknesses of each data source. His solution is to "blend" data from both sources to compensate for the weaknesses of each.  Dr. Couper spoke along similar lines at the ESRA conference last year. An important takeaway from both of these is that surveys have an important place in the future. Surveys gather, relative to big data, rich data on individuals that allow the development and testing of models that may be used with big data. Or provide benchmarks for estimates from big data for which the characteristics of the population are only vaguely known. In any event, I'm not worried that surveys or even probability sampling have outlived their usefulness. But it is good to chart a course for the future that will keep survey folks relevant to these pressing problems.

I've been working on an experiment where evaluating cost savings is an important outcome. It's difficult to measure costs in this environment. Timesheets and call records are recorded separately. It's difficult to parse out the travel time from other time. One study actually shadowed a subset of interviewers in order to generate more accurate cost estimates. This is an expensive means to evaluate costs that may not be practical in many situations. It might be that increasing computerization does away with this problem. In a telephone facility, everything is timestamped so we can calculate how long most call attempts take. It might be that we will be able to do this in face-to-face studies soon/already.

Last week, I mentioned an experiment that we ran with changing the order of tracking steps. I noted that the overall result was that the original, expert-chosen order worked better than the new, proposed order. In this example, the costs weren't all that different. But I could imagine situations where there are big differences in the costs between the different steps. In that case, the order could have big cost implications. I'm also thinking that a common situation is where you have lots of cheap (and somewhat ineffective steps) and one expensive (and effective) step. I'm wondering if it would be possible to identify cases that should skip the cheap treatments and go right to the expensive treatment. Just as a cost savings measure. It would have to result in the same chance of locating the person. In other words, the skipped steps would have to have the same or less information than the costly step. My hunch is that such situations actually exist. The trick is finding ...

I've been reading a number of papers on tracking (aka tracing or locating) of panel members in longitudinal research. Many of the papers are case studies, reporting on what particular studies did. Very few actually conduct experiments. Survey methodologists have produced a few recent experimental papers. Research on HRS showed that higher incentives had persistent effects on response at later waves. McGonagle and colleagues looked at the effects of between-wave cont act methods and incentives . Fumagelli and colleagues   also explore between wave contact methods. These experiments all involve contacting panel members. I found one interesting paper that actually experimented with the order of the steps in the tracking process. Usually, the order starts with the cheapest things to do and goes to the more expensive. If steps have a similar cost, then just choose an order. This paper by Koo et al actually randomized the order of the steps (two different websites). A haven't s...

As I mentioned in my last post, I have been reading an enormous number of papers on locating respondents in panel studies. One interesting thing that I have found is that tracking costs are often described in a manner different than I would have expected. I'm used to thinking of the costs of activities -- telephone calls, internet searches, face-to-face calls, etc. These activity costs can be summed up to total costs, and then averaged over number of cases located or number of cases interviewed. I found a lot of papers reported costs as FTEs. This seemed a lot simpler. I found one review paper that summarized several other studies. They reported all the results as FTEs. This was nifty in that it was simple, and somewhat impervious to inflation and differences in pay rates -- so better than reporting dollar costs. The downside is that the costs can't be rescaled when there are differences among panels in difficulty of being tracked. Some are more difficult and require more ...

I just got back from Amsterdam where I heard the defense of a very interesting dissertation. You can find the full dissertation here . One of the chapters is already published and several others are forthcoming. The dissertation uses optimization techniques to design surveys that maximize the R-Indicator while controlling measurement error for a fixed budget. I find this to be very exciting research as it brings together two fields in new and interesting ways. I'm hoping that further research will be spurred by this work.

I found another interesting article that talked about costs. This one , from Teitler and colleagues, described the apparent nonresponse biases present at different levels of cost per interview. This cuts to the chase on the problem. The basic conclusion was that, at least in this case, the most expensive interviews didn't change estimates. This enables discussing the tradeoffs in a more specific way. With a known amount of the budget that didn't prove to change estimates, could you make greater improvements by getting more cases that cost less? Spending more on questionnaire design? etc. Of course, that's easy to say after the fact. Before the fact, armed with less than complete knowledge, one might want to go after the expensive cases to be sure they are not different. Of course, I'd argue that you'd want to do that in a way that controlled costs (subsampling) until you achieve more certainty about the value of those data.

I'm really enjoying this article   by Andresen and colleagues on the costs and errors associated with tracking (locating panel members). They look at both sides of the problem. I think that is pretty neat. There was one part of the article that raised a question in my mind. On page 46, they talk about tracking costs. They say "...[t]he average tracing costs per interview for stages 1 and 2 were calculated based on the number of tracing activities performed at each stage." An assumption here -- I think -- is that each tracing activity (they list 6 different manual tracing activities) takes the same amount of time. So take the total time from the tracing team, and divide it by the number of activities performed, and you have the average time per activity. This is perfectly reasonable and fairly robust. You might do better with a regression model predicting hours from the types and numbers of activities performed in a week. Or you might ask for more specific information ...