Survey Methods Musings

This is another post in a series on currently used survey design features that could be "relabeled" as adaptive. I think it is helpful to relabel for a couple of reasons. 1) It demonstrates a kind of feasibility of the approach, and 2) it would help us think more rigorously about these design options (for example, if we think about refusal conversions as a treatment within a sequence of treatments, we may design better experiments to test various ways of conducting conversions). The design feature I'm thinking of today has to do with a card that interviewers leave behind sometimes when no one is home at a face-to-face contact attempt. The card says "Sorry I missed you..." and explains the study and that we will be trying to contact them. Interviewers decide when to leave these cards. In team meetings with interviewers, I heard a lot of different strategies that interviewers use with these cards. For instance, one interviewer said she leaves them every time,

In the same vein as previous posts, I'm continuing to think about current practices that might be recast as adaptive. Call limits are a fairly common practice. But they are also, at least for projects that I have worked on, notoriously difficult to implement. For example, it may happen that when project targets for numbers of interviews are not being met, then these limits will be violated. We might even argue that since the timing of the calls is not always well regulated, that it is difficult to claim that cases have received equal treatments prior to reaching the limit. For example, three calls during the same hour is not likely to be as effective as three calls placed on different days and times of day. Yet they would both reach a three-call limit. [As an aside, it might make more sense to place a lower-limit on "next call" propensities estimated from models that include information about the timings of the call, as Kreuter and Kohler do here .] In any event, su

This isn't a technique that gets much attention. I can think of three articles on the topic. I know of one article (Fuse and Xie, 2007)that investigates refusal conversions in telephone surveys and collects information (observations) from interviewers. And I just googled another one (Beullens, et al., 2010) that investigates the effects of time between initial refusal and first converstion attempt. There is a third article (Burton, et al. 2006) on refusal conversions in panel studies. This one adds another element in that a key consideration is whether refusers that are converted will remain in the panel in subsequent waves. This problem seems to fit really well into the sequential decisionmaking framework. The decision is at which waves, for any given case that refuses, should you try a refusal conversion. You might, for instance, optimize the expected number of responses (completed interviews) over a certain number of waves. Or, you might maximize other measures of data qual

I have two feelings about talking about adaptive or responsive designs. The first feeling is that these are new concepts, so we need to invent new methods to implement them. The second feeling is that although these are new concepts, we can point to actual things that we have always (or for a long time) done and say, "that's an example of this new concept" that existed before the concept had been formalized. I think refusal conversions are a good example. We never really applied the same protocol to all cases. Some cases got a tailored or adaptive design feature. The rule is something like this: if the case refuses to complete the interview, then change the interviewer, make another attempt, and offer a higher incentive. I'm trying to think systematically about these kinds of examples. Some are trivial ("if there is no answer on the first call attempt, then make a second attempt"). But others may not be. The more of these we can root out, the more we can