We recently did an experiment with incentives on a face-to-face survey. As one aspect of the evaluation of the experiment, we looked at the costs associated with each treatment (i.e. different incentive amounts).
The costs are a bit complicated to parse out. The incentive amount is easy, but the interviewer time is hard. Interviewers record their time for at the day level, not at the housing unit level. So it's difficult to determine how much a call attempt costs.
Even if we had accurate data on the time spent making the call attempt, there would still be all the travel time from the interviewer's home to the area segment. If I could accurately calculate that, how would I spread it across the cost of call attempts? This might not matter if all I'm interested in is calculating the marginal cost of adding an attempt to a visit to an area segment. But if I want to evaluate a treatment -- like the incentive experiment -- I need to account for all the interviewer costs, as best as I can.
A simple approach is to just divide the interviewer hours by the total number of call attempts. This gives an average that might be useful for some purposes. Or I can try to account for differences in lengths of different types of call attempt outcomes. If the distribution of types of outcomes differ across treatments, then the average length of any attempt might not be a fair comparison of the costs of the two treatments.
I suspect that the problem can only be "solved" by defining the specific purpose for the estimate. Then thinking about how errors in the estimate might impact the decision. In other words, how bad does the estimate have to be to lead you to the wrong decision? I think there are a number of interesting cost problems like this, where we haven't measured the costs directly, but need to use some proxy measure that might have errors of different kinds.
The costs are a bit complicated to parse out. The incentive amount is easy, but the interviewer time is hard. Interviewers record their time for at the day level, not at the housing unit level. So it's difficult to determine how much a call attempt costs.
Even if we had accurate data on the time spent making the call attempt, there would still be all the travel time from the interviewer's home to the area segment. If I could accurately calculate that, how would I spread it across the cost of call attempts? This might not matter if all I'm interested in is calculating the marginal cost of adding an attempt to a visit to an area segment. But if I want to evaluate a treatment -- like the incentive experiment -- I need to account for all the interviewer costs, as best as I can.
A simple approach is to just divide the interviewer hours by the total number of call attempts. This gives an average that might be useful for some purposes. Or I can try to account for differences in lengths of different types of call attempt outcomes. If the distribution of types of outcomes differ across treatments, then the average length of any attempt might not be a fair comparison of the costs of the two treatments.
I suspect that the problem can only be "solved" by defining the specific purpose for the estimate. Then thinking about how errors in the estimate might impact the decision. In other words, how bad does the estimate have to be to lead you to the wrong decision? I think there are a number of interesting cost problems like this, where we haven't measured the costs directly, but need to use some proxy measure that might have errors of different kinds.
Comments
Post a Comment