Survey Methods Musings

I've been continuing with the experiments in call scheduling. January was the first month where there was a difference in response rates by treatment group. Generally, the response rates across the treatment arms (control and experiment) have been similar. But that doesn't necessarily mean the two methods obtain the same result. When I look at response rates by phase, even in prior months, it appears that the experimental method has a higher response rate in the calls prior to a refusal and a lower response rate in the calls after a refusal (even though both sets of calls are now governed by the algorithm). The following tables shows the results from December and January (AAPOR RR2 by refusal status, NOT overall RR): January is the first month where the experimental group outperformed the control group in the refusal conversion phase. At first, I thought the refusal calls were more difficult in the experimental group than the control. But maybe not. We are repeating the...

The experiment on the timing of calls in an RDD survey has been continuing. It looks like the change that we made in the experimental algorithm has an impact.  The change basically meant to govern the timing of refusal conversion calls as well. These calls were not governed by the algorithm (mainly for technical reasons) in the first few months of the experiment. The following table shows the experimental ("MLE") and control ("CON") groups broken down by whether the case was governed by the original algorithm (In Algorithm=1) or whether it was not (In Algorithm=0, i.e. the refusal conversion calls). In prior months, the "In Algorithm=0" cases were much less efficient, such that the overall efficiency was about the same for the experimental and control groups.   The efficiency of the refusal conversion calls is still a bit lower for the experimental group. I'm still trying to understand why that's the case. It's better, but still not what ...