Survey Methods Musings

I've been struggling with the concept of "mode preference." It's a term we use to describe the idea that respondents might have preferences for a mode and that if we can identify or predict those preferences, then we can design a better survey (i.e. by giving people their preferred mode). In practice, I worry that people don't actually prefer modes. If you ask people what mode they might prefer, they usually say the mode in which the question is asked. In other settings, the response to that sort of question is only weakly predictive of actual behavior. I'm not sure the distinction between stated and revealed preferences is going to advance the discussion much either. The problem is that the language builds in an assumption that people actually have a preference. Most people don't think about survey modes. Most don't consider modes abstractly in the way methodologists might. In fact, these choices are likely probabilistic functions that hinge on ...

Following up yet again, on posts about how people choose modes. In particular, it does seem that different subgroups are likely to respond to different modes at different rates. Of course, with the caveat that it's obviously not just the mode, but also how you get there that matters. We do have some evidence about subgroups that are likely to choose a mode. Haan, Ongena, and Aarts examine an experiment where respondents to a survey are given a choice of modes. They found that full-time workers and young adults were more likely to choose web over face-to-face. The situation is an experimental one that might not be very similar to many surveys: Face-to-face and telephone recruitment to the choice of face-to-face or web survey. But at least the design allows them to look at who might make different choices. It would be good to have more data on persons making the choice in order to better understand the choice. For example, information about how much they use the internet might ...

Just a follow-up from my previous post on mixed-mode surveys. I think that one of the things that gets overlooked in discussions of mixed-mode designs is the dosage of each mode that is applied. For example, how many contact attempts under each mode? It's pretty clear that this matters. In general, more effort leads to higher response rates and less effort leads to lower response rates. But, it seems that sometimes when we talk about mixed-mode studies, we forget about the dose. We wrote about this idea in Chapter 4 of our new book on adaptive survey design . I think it would be useful to keep this in mind when describing mixed-mode studies. It might be these other features, i.e. not the mode itself, that account for differences between mixed-mode studies. At least in part.

Ok. The title is a provocative question. But it's one that I've been thinking about recently. A few years ago, I was working on a lit review for a mixed-mode experiment that we had done. I found that the results were inconsistent on an important aspect of mixed-mode studies -- the sequence of modes. As I was puzzled about this, I went back and tried to write down more information about the design of each of the experiments that I was reviewing. I started to notice a pattern. Many mixed-mode surveys offered "more" of the first mode. For example, in a web-mail study, there might be 3 mailings with the mail survey and one mailed request for a web survey. This led me to think of "dosage" as an important attribute of mixed-mode surveys. I'm starting to think there is much more to it than that. The context matters  a lot -- the dosage of the mode, what it may require to complete that mode, the survey population, etc. All of these things matter. Still, we...

Susan Murphy's work on dynamic treatment regimes had a big impact on me as I was working on my dissertation. I was very excited about the prospect of learning from the paradata. I did a lot of work on trying to identify the best next step based on analysis of the history of a case. Two examples were 1) choosing the lag before the next call and the incentive, and 2) the timing of the next call. At this point, I'm a little less sure of the utility of the approach for those settings. In those settings, where I was looking at call record paradata, I think the paradata are not at all correlated with most survey outcomes. So it's difficult to identify strategies that will do anything but improve efficiency. That is, changes in strategies based on analysis of call records aren't very likely to change estimates. Still, I think there are some areas where the dynamic treatment regime approach can be useful. The first is mode switching. Modes are powerful, and offering them i...

The title of this post is a paraphrase of a saying coined by Tolstoi. " Happy families are all alike; every unhappy family is unhappy in its own way." I'm stealing the concept to think about survey respondents.  To simplify discussion, I'll focus on two extremes. Some people are easy respondents. No matter what we do, no matter how poorly conceived, they will respond. Other people are difficult respondents. I would argue that these latter respondents are heterogenous with respect to the impact of different survey designs on them. That is, they might be more likely to respond under one design relative to another. Further, the most effective design will vary from person to person within this difficult group.  It sounds simple enough, but we don't often carry this idea into practice. For example, we often estimate a single response propensity, label a subset with low estimated propensities as difficult, and then give them all some extra thing (often more money). ...

A few years ago, I did an experiment with two sequences of modes for a screening survey. The modes were mail and face-to-face. We found that the sequence didn't matter much for the response rate to the screener, but that the arm that started with face-to-face and then used mail had a better response rate to the main interview given to those who were found to be eligible in the screening interview. There are other experiments that use different sequences of modes. Some of these find that the sequence doesn't matter. For example, Dillman and colleagues looked at mail-telephone and telephone-mail and these had about the same response rate. On the other hand, Millar and Dillman found that for mail-web mixed-mode surveys the sequence does seem to matter, although certainly the number and kind of contact attempts are also important. It does seem that there are times when the early attempts might interfere with the effectiveness of later attempts. That is, we "harden the ref...

I've been reading a bit on mixed-mode surveys. I've noticed several discussions of web surveys and coverage error. This is a relatively recent mode, and one of the key issues has been to what extent the population has access to the internet. If someone doesn't have access to the internet, they can't complete a web survey. Everyone agrees upon that. But how do we describe the source of this error? Is it coverage or nonresponse error? In my mind, coverage error is a property of the sampling frame. If the unit is not on the sampling frame, then it is not covered. But many web surveys are general population surveys that don't have a tight association with a frame. That is, since there is not "internet" sampling frame in the way we have RDD or area probability samples. Many surveys start today from ABS sampling and then might do telephone, mail, web, or mixed-mode designs. In this case, a lack of internet access is an impediment to responding and not an imp...

I've been thinking about mixed-mode surveys a great deal over the last few months. And I notice that research publications tend to use a lot of shorthand to describe the approach -- e.g. "Mail-Telephone." Of course, they describe it in more detail, but the shorthand definition focuses on the modes. Since the shorthand describes the sequence, we end up comparing different sequences. But there are other important design features at play that make these comparisons tenuous. Of course, these other design parameters include the dosage of each mode in the sequence. Different dosages may result in different proportions of the interviews be conducted in each mode. For example, in the mail-telephone design, more mailings can increase the proportion of interviews in the mail mode. A recent article by Klausch, Schouten, and Hox includes a parameter for the mixture of modes \(\pi\). I'm concerned that we may do lots of experimentation to design a mixed mode survey that is con...

I just got back from JSM where I presented the results of an experiment that varied the timing of the mode switch in a web-telephone survey. I'm not going to talk about the results of the experiment in this post, just the premise. The concern that motivated the experiment had to do with the possibility that longer delays before switching modes could have adverse effects on response rates. This could happen for several reasons. If there is pre-notification, then the effect of the prenote on response to the second mode might be reduced with longer delays before switching.  If the first mode is annoying in some way, it can diminish the effectiveness of the second mode. The latter case is particularly interesting to me. It points to the ways that different treatment sequences can have different levels of effectiveness. We saw an impact like this in an experiment we did of two sequences of modes for a screening survey. The two sequences functioned about the same in terms of respo...

A colleague sent me a link to a blog arguing that the "long survey" is dead. The blog takes the point of view that anything over 20 minutes is long. There's also a link to another blog that presents data from survey monkey surveys showing that the longer the questionnaire, the less time that is spent on each question. They don't really control for question length, etc. But it's still suggestive. In my world 20 minutes is still a short survey. But the point is still taken. There has been some research on the effect of survey length (announced) on response rates. There probably is need for more. Still, it might be time to start thinking of alternatives to improve response to long surveys. The most common is to offer a higher incentive, and thereby counteract the burden of the longer survey. Another alternative is to shorten the survey. This doesn't work if your questions are the ones getting tossed. Of course, substituting big data for elements of surveys is...

Survey Practice devoted the entire current issue to a discussion of training in survey methodology. This is a very useful review of what is currently done and suggestions for the future. As they observe, survey methodology is a broad discipline that draws upon a diverse set of fields of research. I expect that increasing this diversity would be positive. That is, there are a number of fields of study that would find applications for their methods in the field of survey research.  A couple of key examples include operations research and computer science. Operations research could help us think more rigorously about designing data collection to optimize specified quantities. That doesn't mean we have to pursue one goal. But it would help, or maybe force us to quantify the vague trade offs we usually deal in. The paper by Greenberg and Stokes is an early example. The paper by Calinescu and colleagues is a recent one.   Computer science is another such field. Researc...

I've been away from the blog for a while, but I'm back. One of the things that I did during my hiatus from the blog was to read papers on mixed-mode surveys. In most of these surveys, there are nonresponse biases and measurement biases that vary across the modes. These errors are almost always confounded. An important exception is Olson's paper . In that paper, she had gold standard data that allowed her to look at both error sources. Absent those gold standard data, there are limits on what can be done. I read a number of interesting papers, but my main conclusion was that we need to make some assumptions in order to motivate any analysis. For example, one approach is to build nonresponse adjustments for each of the modes, and then argue that any differences remaining are measurement biases. Without such an assumption, not much can be said about either error source. Experimental designs certainly strengthen these assumptions, but do not completely unconfound the sources ...

As I have been working on a web survey, I'm following more closely the devices that people are using to complete web surveys. The results from Pew make it seem that the younger generation will move away from PCs and access the internet through portable devices like smart phones. Some of these "portable" devices have become quite large. This trend makes sense to me. I can do many/most things from my phone. I heard on the news the other day, that 25% of Cyber Monday shopping was done with tablets and phones. But some things are easier to do with a PC. Do surveys fit into the latter group? Peter Lugtig posted about a study he is working on that tracks the device used in waves of a panel survey. It appears that those who start on a PC, stay on a PC. But those who start on a tablet or phone are more likely to switch to a PC. He also notes that if you used a tablet or phone in an early wave, you are less likely to do the survey at all in the next wave. I didn't read t...

OK. This actually a survey-related post. I read this short article about an experiment where some kids got a candy bar and other kids got a candy bar and a piece of gum. The latter group was less happy. Seems counter-intuitive, but in the latter group, the "trajectory" of the qaulity of treats is getting worse. Turns out that this is a phenomenon that other psychologists have studied. This might be a potential mechanism to explain why sequence matters in some mixed-mode studies. Assuming that other factors aren't confounding the issue.

In my last post I talked about identifying all the components of a design. At least identifying them is an important step if we want to consider randomizing them. Of course, it's not necessary... or even feasible... or even desirable to do a full factorial design for every experiment. But it is still good to at least mentally list the potentially active components. I first started thinking about this when I was doing a literature review for a paper on mixed mode designs. Most of these designs seemed to confound some elements of the design. The main thing I was looking for -- could I find any examples where someone had just varied the sequence of modes? The problem was that most people also varied the dosage of modes. For example, in a mixed mode web-telephone design, I could find studies that had web-telephone and telephone-web comparisons, but these sequences also varied the dosage. So, telephone first gets up to 10 calls, but telephone second gets 2 calls. Web first gets 3 emai...

I'm still thinking about this problem. For me, it's much simpler conceptually to think of this as a missing data problem. Andy Peytchev's paper makes this point. If I have the "right" structure for my data, then I can use imputation to address both nonresponse and measurement error. If the measurement error is induced differently across different modes, then I need to have some cases that receive measurements in both modes. That way, I can measure differences between modes and use covariates to predict when this happens. The covariates, as I discussed last week, should help identify which cases are susceptible to measurement error. There is some work on measuring whether someone is likely to be influenced by social desirability. I'm think that will be relevant for this situation. That sounds sort of like, "so you don't want me to tell me the truth about x, but at least you will tell me that you don't want to tell me that." Or something li...

I've been working on some mixed-mode problems where nonresponse and measurement error are confounded. I recently read an interesting article on using adjustment models to disentangle the two sources of error. The article is by Vannieuwenhuyze, Loosveldt, and Molenberghs. They suggest that you can make adjustments for measurement error if you have things that predict when those errors occur. They give specific examples. It's things that measure social conformity and other hypothesized mechanisms that lead to response error. This was very interesting to read about. I suppose that just as with nonresponse, the predictors of this error -- in order to be useful -- need to predict when those errors occur and the survey outcome variables themselves. This is a new and difficult task... but one worth solving giving the push to use mixed mode designs.

I just got back from Amsterdam where I heard the defense of a very interesting dissertation. You can find the full dissertation here . One of the chapters is already published and several others are forthcoming. The dissertation uses optimization techniques to design surveys that maximize the R-Indicator while controlling measurement error for a fixed budget. I find this to be very exciting research as it brings together two fields in new and interesting ways. I'm hoping that further research will be spurred by this work.

I recently read an article about a very interesting experiment. Luiten and Schouten report on an experiment to improve the Statistics Netherlands' Survey of Consumer Sentiment. Their task was to improve representativity (defined as increasing the R-Indicator) of the survey without increasing costs and without lowering the response rate. This sounds like a difficult task. We can debate the merits of lowering response rates in "exchange" for improved representativity. But who can argue with increasing representativity without major increases in costs or decreases in response rates. The experiment has a number of features all built with the goal of meeting these constraints. One of the things that makes their paper so interesting is that each of the design features is "tailored" to the specifics of the sampled units. For those of you who like the suspense of a good survey experiment, spoiler alert: they managed to meet their objectives.