Survey Methods Musings

I'm really enjoying this article   by Andresen and colleagues on the costs and errors associated with tracking (locating panel members). They look at both sides of the problem. I think that is pretty neat. There was one part of the article that raised a question in my mind. On page 46, they talk about tracking costs. They say "...[t]he average tracing costs per interview for stages 1 and 2 were calculated based on the number of tracing activities performed at each stage." An assumption here -- I think -- is that each tracing activity (they list 6 different manual tracing activities) takes the same amount of time. So take the total time from the tracing team, and divide it by the number of activities performed, and you have the average time per activity. This is perfectly reasonable and fairly robust. You might do better with a regression model predicting hours from the types and numbers of activities performed in a week. Or you might ask for more specific information ...

I spent a few posts cataloging design features that could be considered adaptive. No one labelled them that way in the past. But if we were already doing it, why do we need the new label? I think there are at least two answers to that: 1. Thinking about these features allows us to bring in the complexity of surveys. Surveys are multiple phase activities, where the actions at different phases may impact outcomes at later phases. This makes it difficult to design experiments. Clinical trials, some have labelled this phenomenon as " practice misalignments ." They note that trials that focus on single-phase, fixed-dose treatments are not well aligned with how doctors actually treat patients. The same thing may happen for surveys. When something doesn't work, we don't usually just give up. We try something else. 2. It gives us a concept to think about these practices. It is an organizing principle that can help identify common features, useful experimental me...

I really enjoyed this paper by Peter Lynn on targeting cases for different recruitment protocols. He makes a solid case for treating cases unequally, with the goal of equalizing response probabilities across subgroups. It also includes several examples from panel surveys. I strongly agree that panel surveys are a fertile ground for trying out new kinds of designs. They have great data and there is a chain of interactions between the survey organization and the panel member. This is more like the adaptive treatment setting that Susan Murphy and colleagues have been exploring. I believe that panel surveys may be a fertile ground for bringing together ideas about adaptive treatment regimes and survey design.

This is a highly tailored strategy. The idea is that certain kinds of interviewer observations about contact with sampled households will be used to tailor a letter that is sent to the household. For example, if someone in the household says they are "too busy" to complete the survey, a letter is sent that specifically addresses that concern. It's pretty clear that this is adaptive. But here again, thinking about it as an adaptive feature could improve a) our understanding of the technique, and b) -- at least potentially -- its performance. In practice, interviewers request that these letters be sent. There is variability in the rules they use about when to make that request. This could be good or bad. It might be good if they use all of the "data" that they have from their contacts with the household. That's more data than the central office has. On the other hand, it could be bad if interviewers vary in their ability to "correctly" identify c...

This is another post in a series on currently used survey design features that could be "relabeled" as adaptive. I think it is helpful to relabel for a couple of reasons. 1) It demonstrates a kind of feasibility of the approach, and 2) it would help us think more rigorously about these design options (for example, if we think about refusal conversions as a treatment within a sequence of treatments, we may design better experiments to test various ways of conducting conversions). The design feature I'm thinking of today has to do with a card that interviewers leave behind sometimes when no one is home at a face-to-face contact attempt. The card says "Sorry I missed you..." and explains the study and that we will be trying to contact them. Interviewers decide when to leave these cards. In team meetings with interviewers, I heard a lot of different strategies that interviewers use with these cards. For instance, one interviewer said she leaves them every time, ...

In the same vein as previous posts, I'm continuing to think about current practices that might be recast as adaptive. Call limits are a fairly common practice. But they are also, at least for projects that I have worked on, notoriously difficult to implement. For example, it may happen that when project targets for numbers of interviews are not being met, then these limits will be violated. We might even argue that since the timing of the calls is not always well regulated, that it is difficult to claim that cases have received equal treatments prior to reaching the limit. For example, three calls during the same hour is not likely to be as effective as three calls placed on different days and times of day. Yet they would both reach a three-call limit. [As an aside, it might make more sense to place a lower-limit on "next call" propensities estimated from models that include information about the timings of the call, as Kreuter and Kohler do here .] In any event, su...

This isn't a technique that gets much attention. I can think of three articles on the topic. I know of one article (Fuse and Xie, 2007)that investigates refusal conversions in telephone surveys and collects information (observations) from interviewers. And I just googled another one (Beullens, et al., 2010) that investigates the effects of time between initial refusal and first converstion attempt. There is a third article (Burton, et al. 2006) on refusal conversions in panel studies. This one adds another element in that a key consideration is whether refusers that are converted will remain in the panel in subsequent waves. This problem seems to fit really well into the sequential decisionmaking framework. The decision is at which waves, for any given case that refuses, should you try a refusal conversion. You might, for instance, optimize the expected number of responses (completed interviews) over a certain number of waves. Or, you might maximize other measures of data qual...