Survey Methods Musings

One aspect of responsive design that hasn't really been considered is when to stop collecting data. Groves and Heeringa (2006) argue that you should change your data collection strategy when it ceases to bring in interviews that change your estimate. But when should you stop? It seems like the answer to the question should be driven by some estimate of the risk of nonresponse bias. But given that the response rate appears to be a poor proxy measure for this risk, what should we do? Rao, Glickman and Glynn proposed a rule for binary survey outcome variables. Now, Raghu and I have an article accepted at Statistics in Medicine that proposes a rule that uses imputation methods and recommends stopping data collection when the probability that additional data (i.e. more interviews) will change your estimate is sufficiently small. The rule is for normally distributed data. The rule is discussed in my dissertation as well.

One problem that we face in evaluating the experiments in face-to-face surveys where the interviewer decides when to call, leave SIMY cards,etc. is that we don't know whether the interviewer followed our recommendation. Maybe they just happened to do the very thing we recommended without viewing our recommendation. I'm facing this problem with both the experiment involving SIMY card use and the call scheduling experiment. We could ask them if they followed the recommendation, but their answers are unlikely to be reliable. My current plan is to save the statistical recommendation for all cases (experimental vs control) and compare how often the recommendation is followed in both groups. In the control group, the recommendation is never revealed to the interviewer. If the recommendation is "followed" more in the group where it is revealed, then it appears that it did have an impact on the choices the interviewers made.

Although we don't have any evidence, our prior assumption seems to be that SIMY cards are generally helpful. Julia D'arrigo, Gabi Durrant, and Fiona Steele have a working paper that presents evidence from a multi-level multinomial model that these cards do improve contact rates. A further question that we'll be attempting to answer is whether we can differentiate among cases for which the SIMY card improves contact rates and those for which it does not. Why would a card hurt contact rates? It might be that for some households, the card acts as a warning and they work to avoid the interviewer. Or, they may feel that leaving the card was somehow inappropriate. We have anecdotal evidence on this score. In the models I've been building, I have found interactions between observable characteristics of the case (e.g. is it in a neighborhood with access impediments? Is it in a neighborhood with safety concerns?, etc.) that indicate that we may be able to differentiate our ...

I mentioned the R-Indicators in a recent post. In addition to the article in Survey Methodology , they also have a very useful website . The website includes a number of papers and presentations on the topic.

I've actually already done a lot of work on imputing eligibility. For my dissertation, I used the fraction of missing information as a measure of data quality. I applied the measure to survey data collections. In order to use this measure, I had to impute for item and unit nonresponse (including the eligibility of cases that are not yet screened for eligibility). The surveys that I used both had low eligibility rates (one was an area probability sample with an eligibility rate of about 0.59 and the other was an RDD survey with many nonsample cases). As a result, I had to impute eligibility for this work. An article on this subject has been accepted by POQ and is forthcoming. The chart shown below uses data from the area probability survey. It shows the distribution of eligibility rates that incorporate imputations for the missing values. The eligiblility rate for the observed cases is the red line. The imputed estimates appear to be generally higher than the observed v...

In a previous blog I talked about an experiment that I'm currently working. The experiment is testing a new method for scheduling calls. For technical reasons, only a portion of the calls were governed by the experimental method. Refusal conversion calls were not governed by the new method. The experiment had the odd result that although the new method increased efficiency for the calls governed by the algorithm, these gains were lost at a later step (i.e. during refusal conversion -- see the table for results).   This month, we resolved the technical issues (maybe we should have done this in the first place). Now we will be able to see if we can counteract this odd result. If not, then we'll have to assume either:  Improbable sampling error explains this Some odd interaction between the method and resistance/refusal I'm hoping this moves things in the "expected" fasion.