Survey Methods Musings

I recently read an article by Tourangeau, Kreuter, and Eckman on undercoverage in screening surveys. One of several experiments on which they report explores how the form of the screening questions can impact eligibility rates. They compare taking a full household roster to asking if there is anyone within the eligible age range. The latter produces lower eligibility rates. There was a panel at JSM years ago that discussed this issue. Several major screening surveys reported similar undercoverage issues. Certainly the form of the question makes a difference. But even on screening surveys that use full household rostering, there can be undercoverage. I'm wondering what the mechanism is. If the survey doesn't advertise the eligibility criteria, how is that some sampled units avoid being identified as eligible? This might be a relatively small source of error in the survey, but it is an interesting puzzle.

There was another interesting thing that happened in our mixed mode that varied the sequences of modes. As a reminder, the survey was a screening survey. It identified households with an eligible person. An interviewer later returned to eligible households to complete a 'main' interview. We found that eligible households in locked buildings that had received the FtF-Mail treatment on the screening interview responded at higher rates to the main interview. Better even than Mail-FtF. The results were significant, even when accounting for the clustering in the sample. It may be that the FtF-Mail sequence displays our earnestness to the respondent more clearly. It would be nice to replicate these results.

The genesis of the idea for imputing "e" came from my process for estimating the fraction of missing information for an ongoing survey. I had to impute eligibility for cases each day so that I could impute survey values for the subset of eligible case (including those with imputed eligibility). I thought, "hey, I'm already imputing 'e.' I just need to set it up that way." Along the way, I had to compare the method to the life table product-limit approach advocated by Brick et al. (POQ, 2002). I found a very nifty article by Efron (JASA, 1988) that compares life table methods to logistic regression. Essentially, for the discrete time case, the life table model produces the same results as if we had a logistic regression model with a dummy variable for each time point. Efron then paramterized the model with fewer parameters ($t$, $t^2$, and $t^3$), I believe, and shows how this compares to the life table product-limit nonparametric estimate. This artic...

I'm finishing up a presentation that I'll be giving at AAPOR (Saturday, May 15th at 2:15) on using imputation methods to estimate "e." I posted on this topic a while ago. I wanted to post one of the graphics that I developed for that presentation. I start from a very simple model that predicts eligibility using the natural logarithm of the last call number as the only predictor. That generates the following distribution of imputed eligible cases. The blue line shows the eligibility rates for the cases for which the eligibility status is known. The blue dashed line shows the model (a logistic regression model predicting eligibility using the natural log of the call number) prediction of eligibility. The green line shows the eligibility for the cases where the eligibility flag is imputed. The green line is the line used to estimate "e."

I've actually already done a lot of work on imputing eligibility. For my dissertation, I used the fraction of missing information as a measure of data quality. I applied the measure to survey data collections. In order to use this measure, I had to impute for item and unit nonresponse (including the eligibility of cases that are not yet screened for eligibility). The surveys that I used both had low eligibility rates (one was an area probability sample with an eligibility rate of about 0.59 and the other was an RDD survey with many nonsample cases). As a result, I had to impute eligibility for this work. An article on this subject has been accepted by POQ and is forthcoming. The chart shown below uses data from the area probability survey. It shows the distribution of eligibility rates that incorporate imputations for the missing values. The eligiblility rate for the observed cases is the red line. The imputed estimates appear to be generally higher than the observed v...

AAPOR defines response rates that include an adjustment factor for cases that have unknown eligibility at the end of the survey. They call the factor "e". Typically, people use the eligibility rate from the part of the sample where this variable (eligible=yes/no) is observed. This estimate is sometimes called the CASRO estimate of e. But in a telephone survey, this estimate of "e" is likely to be biased upwards for the unknown part of the sample. Many of the cases that are never contacted are not households. They are simply numbers that will ring when dialed, but are not assigned to a household. These cases are never involved in estimates of "e". A paper in POQ (Brick and Montaquila, 2002) described an alternative method of estimating e. They use a survival model. This lowers estimates of e relative to the CASRO method. But it's still upwardly biased since many of the noncontacts could never be contacted. I like the survival method since it's clos...