Wednesday, July 28, 2010

Refusal Conversions, Some Results

We just completed a month of data collection on the RDD survey that is running my experiment on call scheduling. I discussed an interesting problem in a previous post. Basically, the algorithm seems to work for calls prior to any refusal. But it is actually less efficient for calls made after an initial refusal (i.e. refusal conversion calls).

One hypothesis about why this occurred was that the person who refused would be screening calls and would not pick up if they saw that we were calling again. The model, which is tuned to contact, might lead you to call back during the same call window as that in which the first refusal was taken. If you call during another call window, you might reach another person in the household or, perhaps, the person who refused would be less likely to be screening calls.

The change was to make the window in which the first refusal occurred the lowest priority window.

The results were... no change (12.0% contact rate for controls, 10.1% for experimental group). The control group was still more efficient than the experimental group at contacting refusal conversion cases.

I had one other thought. Could it be that the algorithm is not allowing less time between the refusal and the conversion attempt? This is a parameter in our sample management system, and it should lead to a minimum lag time. But it is possible the algorithm was creating a difference in the lag time between first refusal and the first conversion attempt. Alas, the following table shows this is probably not the case.

Back the drawing board...

Wednesday, July 21, 2010

How do they do it?

The experiment on call scheduling in a telephone survey required specialized programming to make it work. We use Blaise SMS in our telephone facility. My colleagues here, Joe Matuzak and Dave Dybicki, are planning to present what they did to make this experiment work at the International Blaise Users Conference (IBUC) in October.

They asked me to show some of the results. The first problem we faced was how to make sure that the experimental and control groups were called at the same pace. I produce files every day that show how I want the sample sorted. The control group is sorted using a different algorithm. But we had to make sure that the cases were mixed up -- we didn't want to call one group and then the other.

Dave wrote a program that reads the sorted list for each group (experimental and control). It pulls a record from each list and then checks if it is still active. Maybe it was finalized after the sort occured. When it finds 5 active cases from the top of the sort in one group, it pulls 5 active cases from the other group. It continues this process until the whole sample is sorted, with 5 active experimental cases followed by 5 control cases. Or vice versa.

The following table shows the proportion of calls to each type of case by hour. I'm ignoring day of week (mainly weekend vs weekday) just to simplify the table.

It looks like the algorithm works pretty well at keeping the distribution of calls even across hours of the day. There were some other jazzy features that Dave and Joe set up for the experiment. You'll have to catch their presentation at IBUC if you want to learn more about those.

Thursday, July 1, 2010

Refusal Conversions and Timing of the Call

In the previous post, I talked about an interesting problem that a call scheduling experiment produced in relationship to refusal conversions. In the experiment, calls prior to a first refusal were more efficient under the experimental algorithm. But calls after the refusal were less efficient (in terms of contact) such that the experimental condition was only as efficient as the control when looking at call pre- and post-first refusal. Ouch.

My question is: what can be done to change the call scheduling of calls after the refusal to improve their efficiency? My colleagues here in Survey Research Operations suggested that calling back at the same time as the first refusal might be bad. You might get the same person.

For that to be the case, it seems as if the person who refused would have to screening their calls. While the control algorithm calls back at different times and finds someone else at home.

As a test of this hypothesis, I changed the algorithm to put the window in which the first refusal was taken at the bottom of the list. In other words, avoid calling at the window in which the first refusal was taken. We'll see how or if this works...