Survey Methods Musings

Once more on this theme that I discussed on this blog several times last year. This is a central problem for the field of research known as reinforcement learning. I'd recommend taking a look at Sutton and Barto' s book if you are interested. It's not too technical and can be understood by someone without a background in machine learning. As I mentioned in my last post, I think learning in the survey environment is a tough problem. The paper that proposed the upper confidence bound rule said it works well for short run problems -- but the short run they envisioned was something like 100 trials. In the survey setting, there aren't repeated rewards. We're usually looking for one interview. You might think of gaining contact as another reward, but still. We're usually limited to a relatively small number of attempts (trials). We also often have poor estimates of response and contact probabilities to start with. Given that reward structure, poor prior informatio...

One of the issues with looking at average contact rates (like with the heat map from a few posts ago) is that it's only helpful for average cases. In fact, some cases are easy to contact no matter what strategy you use, other cases are easy to contact when you try a reasonable strategy (i.e. calling during a window with an average high contact rate), but what is the best strategy for the hard-to-reach cases? I've proposed a solution that tries to estimate the best time to call using the accruing data. I know other algorithms might explore other options more quickly. For instance, choosing the window with the highest upper bound on a confidence interval. It might be interesting to try these approaches, particularly for studies that place limits on the number of calls that can be made. The lower the limit, the more exploration may pay off.

I recently pointed out this article by Calinescu and colleagues that uses techniques (specifically Markov Decision Process models) from operations research to design surveys. One of the citations from Calinescu et al. is to this article, which I had never seen, about using nonlinear programming techniques to solve allocation problems in stratified sampling. I was excited to find these articles. I think these methods have the promise of being very useful for planning survey designs. If nothing else, posing the problems in the way these articles do at least forces us to apply a rigorous definition of the survey design. It would be good if more folks with these types of skills (operations research, machine learning, and related fields) could be attracted to work on survey problems.