Survey Methods Musings

I saw this interesting article that just came out. It called to my mind a talk that was hosted here a few (8?) years ago. The talk was someone from a major corporation who talked about how they switched product testing from church basements to online panels. They found that once they switched, the data became worse. The online panels picked products that ended up failing at higher rates. This seemed like a tough problem. There isn't much of a "nonresponse" kind of relationship here. But at least understanding the mechanism that got people into online panels and how they were then selected and agreed to participate in this kind of product testing seemed important. It's not my area, so I'm wondering if this has ever been done. Not that anyone would understand the process of recruiting people to participate in product testing in church basements. But that process at least worked. This new article looks at an old process -- mall intercepts -- for recruiting people...

In my last post I talked about identifying all the components of a design. At least identifying them is an important step if we want to consider randomizing them. Of course, it's not necessary... or even feasible... or even desirable to do a full factorial design for every experiment. But it is still good to at least mentally list the potentially active components. I first started thinking about this when I was doing a literature review for a paper on mixed mode designs. Most of these designs seemed to confound some elements of the design. The main thing I was looking for -- could I find any examples where someone had just varied the sequence of modes? The problem was that most people also varied the dosage of modes. For example, in a mixed mode web-telephone design, I could find studies that had web-telephone and telephone-web comparisons, but these sequences also varied the dosage. So, telephone first gets up to 10 calls, but telephone second gets 2 calls. Web first gets 3 emai...

I've been reading papers on email prenotification and reminders. They are very interesting. There are usually several important features for these emails: how many are sent, the lag between messages, the subject line, the content of the email (length etc.), the placement of the URL, etc. A full factorial design with all these factors is nearly impossible. So folks do the best they can and focus on a few of these features. I've been looking at papers on how many messages were sent, but I find that the lag time between message also varies a lot. It's hard to know which of these dimensions is the "active" component. It could be either, both, and may even be synergies (aka "interactions") between the two (and between other dimensions of the design as well). Linda Collins and colleagues talk about methods for identifying the "active components" of the treatments in these complex situations. Given the complexity of these designs, with a large nu...

I missed Dr. Groves blog post on this topic. It is an interesting perspective on the strengths and weaknesses of each data source. His solution is to "blend" data from both sources to compensate for the weaknesses of each.  Dr. Couper spoke along similar lines at the ESRA conference last year. An important takeaway from both of these is that surveys have an important place in the future. Surveys gather, relative to big data, rich data on individuals that allow the development and testing of models that may be used with big data. Or provide benchmarks for estimates from big data for which the characteristics of the population are only vaguely known. In any event, I'm not worried that surveys or even probability sampling have outlived their usefulness. But it is good to chart a course for the future that will keep survey folks relevant to these pressing problems.