Skip to main content

Posts

Showing posts from 2011

Fractional Factorial Experiments and Surveys

I'm reading and greatly enjoyed Statistics for Experimenters. The book emphasizes careful thinking about experimentation in addition to careful explanation of statistical methods.

One of the main topics is the use of fractional factorial designs. In these designs, some of the experimental conditions are confounded with higher order interactions (the latter are assumed to be negligible). These designs can be more efficient when the assumptions are correct. And fitting these designs into a sequence of experiments can make them very powerful.

This has me thinking about how we often conduct experiments on surveys. We often vary a single factor and then report the results with a slim description of the other "essential survey conditions" -- i.e. other factors which are not explored. These other conditions are potential confounders in these experimental designs.

If more of these essential conditions were reported, it might help make sense of experimental results that are somet…

Mixed Mode Experiments

I'm going to be at MAPOR talking about a mixed-mode experiment that we did last year. We tried to randomize the sequence of two modes -- mail and face-to-face. The latter mode is (obviously) interviewer-administered. One of the difficulties in administering this type of experience is that it's difficult to apply the treatment (face-to-face) attempts evenly across all the cases.We can look at outcomes and not if the treatments were applied differently using simple measures like number of attempts. But that certainly doesn't capture the whole picture.

In the end, we usually default to an "intent-to-treat" analysis that acknowledges that cases will get different dosages of the treatment, but ignores the differences in actual treatment for the analysis (i.e. even cases with fewer than the prescribed number of attempts are included in the analysis. I imagine that different survey organizations would differ on these sorts of outcomes. It seems that it is important to d…

Hazards of analyzing call records

I've been worrying about how to use information about the number of calls in propensity models for a while. To me, it seems that you shouldn't expect simple, linear relationships between the number of calls and the propensity of response. The distance between 1 and 2 calls, is greater than the distance between 12 and 13 calls. Maybe some kind of transformation (natural logarithm) can patch that up.

But I recently saw a presentation (at the International Statistical Institute conference in Dublin) by Paul Biemer, Patrick Chen, and Kevin Wang from RTI. They found that there were reporting errors in the call records. Interviewers didn't record when they drove by housing units and saw that no one was home. They also sometimes didn't record calls since they had limits on the number of calls they could place on any housing unit. I'm sure these errors are endemic in many/all face-to-face surveys. Biemer and his colleagues also demonstrated how these errors can bias resul…

Adaptive and Responsive Design

I've raised this topic a couple of times here. Several years ago, Groves and Heeringa (2006) proposed an approach to survey data collection that they called "Responsive Design." The design was rolled out in phases with information from prior phases being used to tailor the design in later phases.

In my dissertation, I wrote about "Adaptive Survey Design." For me, the main point of using the term "adaptive" was to link to the research on adaptive treatment regimes, especially as proposed by Susan Murphy and her colleagues.

I hadn't thought much about the relationship between the two. At the time, I saw what I was doing as a subset of responsive designs.

Since then, Barry Schouten and Melania Calinescu at Statistics Netherlands have defined "adaptive static" and "adaptive dynamic" designs. Adaptive static designs tailor the protocol to information on the sampling frame. For example, determining the mode of contact for each case …

Mode Switching Algorithms

After running an experiment using sequences of modes (for contacting sampled households), I've been thinking about how to decide when to switch modes. In our experiment, we had a specified time when the switch would occur (after 5 weeks of the first mode, switch to the second mode).

It seems like better "switching" rules should be possible. Ideally, we would want to identify some best mode as quickly as possible. The amount of time it might take to determine this would vary across sampled cases.

The hard part is that we generally have very little feedback. We don't get a lot of information back from failed attempts. For example, a letter doesn't generally generate much feedback other than an interview occurred, it didn't occur, or the letter was returned. It might be that interviewer-administered modes are more promising for this kind of tailoring, since they do generally obtain more feedback.

Experimental Design in Surveys

Sorry for the long layoff! I had a very busy Spring.

I've been working with the results of an experiment we ran on a survey last year. The experimental condition that we wanted to vary was the mode of contact. The results were a bit messy. We didn't have complete control of all the conditions. The main issue was that we couldn't insure that sampled units in each arm of the experiment received the same treatment (equality of effort -- number of calls distributed over windows and across time in equivalent manner).

This is a common problem for experiments that we run. Most of our experiments are 'piggy-backed' onto data collections for which the experiment is a lower priority. They actually need to collect data.

I've been focused on the negatives, the messiness of this situation. But there is a positive. Most of these experiments are embedded in real-world situations. Hence, they should have greater external validity. If we try them again, many of the same essent…

'Design Phases' and 'Responsive Design'

One of the sticking points that I've had with reconciling 'adaptive designs' and 'responsive designs' has been Groves and Heeringa's description of responsive design which includes the notion of 'design phases.' Their definition says: "A design phase is a time period of a data collection during which the same set of sampling frame, mode of data collection, sample design, recruitment protocols and measurement conditions are extant."

This definition didn't seem to fit too well with the notion of adaptive design. In adaptive design, the treatments are tailored to the individual. In the survey context, a treatment can be an incentive, additional calls, etc. The tailoring variable is a time-varying variable -- that is, it changes during the field period. When a persons value on the tailoring variable changes, their treatment changes as well. Under this approach, the design phase is a person-level attribute. One person can be in 'phase 1'…