Survey Methods Musings

One of the sticking points that I've had with reconciling 'adaptive designs' and 'responsive designs' has been Groves and Heeringa' s description of responsive design which includes the notion of 'design phases.' Their definition says: "A design phase is a time period of a data collection during which the same set of sampling frame, mode of data collection, sample design, recruitment protocols and measurement conditions are extant." This definition didn't seem to fit too well with the notion of adaptive design. In adaptive design, the treatments are tailored to the individual. In the survey context, a treatment can be an incentive, additional calls, etc. The tailoring variable is a time-varying variable -- that is, it changes during the field period. When a persons value on the tailoring variable changes, their treatment changes as well. Under this approach, the design phase is a person-level attribute. One person can be in 'phase 1...

I'm still struggling to find a method that improves contact rates in the refusal conversion process for the experiment with call scheduling. As a reminder, the experimental method improves contact rates for calls prior to a refusal, but then contact rates for calls after a first refusal have lower contact rates than calls to those cases in the control group. Ouch. I already tried calling households at times other than the time at which the first refusal was taken. The hypothesis was that people were screening us out and that calling at a different time might lead to someone else in the household picking up the phone. But that didn't work. In looking at the data, searching for a reason that this is happening, I noticed that the control group seemed to be "exploring" better than the experimental group. The figures below demonstrate this. The upper figure shows calls prior to a refusal. It shows the average number of windows that have been called by call number for c...

For me, the idea of adaptive design was influenced by work from the field of clinical trials on multi-stage treatments. Susan Murphy introduced me to adaptive treatment regimes as an approach to the problem. She points to methods developed in the field of reinforcement learning as useful approaches to problems of sequential decisionmaking. Reinforcement learning describes some policies (i.e. a set of decision rules for a set of sequential decisions) as myopic. A policy is myopic if it only looks at the rewards available at the next step. I'm reading Decision Theory by John Bather right now. He uses an example similar to the following to demonstrate this issue. The following is a simple game. The goal is to get from the yellow square to the green square with the lowest cost. The number in each square is the cost of moving there.Diagonal moves are not allowed.   The myopic policy looks only at the next option and goes down a path that ends up with only expensive options to reac...

My dissertation was entitled "Adaptive Survey Design to Reduce Nonresponse Bias." I had been working for several years on "responsive designs" before that. As I was preparing my dissertation, I really saw "adaptive" design as a subset of responsive design. Since then, I've seen both terms used in different places. As both terms are relatively new, there is likely to be confusion about the meanings. I thought I might offer my understanding of the terms, for what it's worth. The term "responsive design" was developed by Groves and Heeringa (2006) . They coined the term, so I think their definition is the one that should be used. They defined "responsive design" in the following way: 1. Preidentify a set of design features that affect cost and error tradeoffs. 2. Identify indicators for these costs and errors. Monitor these during data collection. 3. Alter the design features based on pre-identified decision rules based on ...

There have been several important studies of interviewer variance in face-to-face surveys. O'Muircheartaigh and Campanelli (1998) report on a study that used an interpenetrated design to evaluate the impact of interviewers on variance estimates. There are also studies that show interviewers vary in their ability to establish contact (Campanelli et al., Can you hear me knocking? 1999) and elicit response ( Durrant, Groves, Staetsky, Steele, 2010 ). Although O'Muricheartaigh and Campanelli  account for the clustering of the sample design, they don't account for differences in response (due to contact or refusal). It may be that variation in response rates or the composition of response may explain some (certainly not all) of the interviewer variation. If that is the case, then attempting to control interviewer recruitment protocols (like call timing) might help reduce interviewer variance.

We just completed a month of data collection on the RDD survey that is running my experiment on call scheduling. I discussed an interesting problem in a previous post . Basically, the algorithm seems to work for calls prior to any refusal. But it is actually less efficient for calls made after an initial refusal (i.e. refusal conversion calls). One hypothesis about why this occurred was that the person who refused would be screening calls and would not pick up if they saw that we were calling again. The model, which is tuned to contact, might lead you to call back during the same call window as that in which the first refusal was taken. If you call during another call window, you might reach another person in the household or, perhaps, the person who refused would be less likely to be screening calls. The change was to make the window in which the first refusal occurred the lowest priority window. The results were... no change (12.0% contact rate for controls, 10.1% for experimen...

The experiment on call scheduling in a telephone survey required specialized programming to make it work. We use Blaise SMS in our telephone facility. My colleagues here, Joe Matuzak and Dave Dybicki, are planning to present what they did to make this experiment work at the International Blaise Users Conference (IBUC) in October. They asked me to show some of the results. The first problem we faced was how to make sure that the experimental and control groups were called at the same pace. I produce files every day that show how I want the sample sorted. The control group is sorted using a different algorithm. But we had to make sure that the cases were mixed up -- we didn't want to call one group and then the other. Dave wrote a program that reads the sorted list for each group (experimental and control). It pulls a record from each list and then checks if it is still active. Maybe it was finalized after the sort occured. When it finds 5 active cases from the top of the sort in...