Survey Methods Musings

I found this really interesting article ("Deciding what to observe next") from the field of machine learning. They address the problem of building a regression model using data from a "data stream." A data stream is incoming data. The example they use is daily measurements of weather at different locations. But monitoring paradata during data collection also may have this flavor. They use statistical techniques that I've seen before -- the Lasso for model selection and the EM algorithm for dealing with "missing" data.  The missing data in this case are variables that you choose not to observe at certain points. The neat thing is that their method continues to explore data that are judged to be "not useful" (i.e. not included in the model) at certain points.

I'm still thinking about this problem that gets posed in machine learning -- exploitation vs. exploration. If you want to read more on the topic, I'd recommend Sutton and Barto's Reinforcement Learning . The idea is deceptively simple. Should you take the highest reward given what you currently know, or explore actions for which you don't know the reward? In machine learning, they try to balance the two objectives. For example, in situations of greater uncertainty, you might spend more resources on exploration. In surveys, I've tended to look at experiments as discrete events. Run the experiment. Get the results. Implement the preferred method. But we know that the efficacy of methods changes over time. The simplest example is incentive amounts. What's the right amount? One way to answer this question would be to run an experiment every so often. And change your incentive amounts based on each experiment. Another approach might be to keep a low level of expl...