Survey Methods Musings

There was a debate held yesterday on "Margin of Error" in the presence of nonresponse and using non-probability samples. This is an interesting and useful discussion. In the best of circumstances, "margin of error" represents the sampling error associated with an estimate. Unfortunately, other matters often... errrrr... always interfere. The sampling mechansim is not easily identified or modeled in the case of nonprobability samples. In the case of probability samples, the nonresponse mechanism has to be modeled. Either of these situations involve some model assumptions (untestable) that are required to motivate the estimation of a margin of error. One step forward would be for people who report estimated "margins of error" to reveal all of their assumptions in their  models (weighting models for nonresponse or, in the case of nonprobability samples, selection) and describe the sampling and recruitment mechanisms sufficiently such that others can evalu...

I read this very interesting blog post by Peter Lugtig yesterday. The slides from the talk he describes are also linked to the post. He builds on an analysis of classes of nonresponders. Several distinct patterns of nonresponse are identified. The characteristics of persons in each class are then described. For example, some drop out early, some "lurk" around the survey, some stay more or less permanently. He suggests that it might be smart to identify design features that are effective for each of the groups and then tailor these features to the subgroups in an adaptive design. This makes a lot of sense. And panel studies are an attractive place to start doing this kind of work. In the panel setting, there is a lot more data available on cases. This can help in identifying subgroups. And, with repeated trials of the protocol, it may be possible to improve outcomes (response) over time. I think the hard part is creating the groups. This reminds me of a problem that I read...

I've been away from the blog for a while, but I'm back. One of the things that I did during my hiatus from the blog was to read papers on mixed-mode surveys. In most of these surveys, there are nonresponse biases and measurement biases that vary across the modes. These errors are almost always confounded. An important exception is Olson's paper . In that paper, she had gold standard data that allowed her to look at both error sources. Absent those gold standard data, there are limits on what can be done. I read a number of interesting papers, but my main conclusion was that we need to make some assumptions in order to motivate any analysis. For example, one approach is to build nonresponse adjustments for each of the modes, and then argue that any differences remaining are measurement biases. Without such an assumption, not much can be said about either error source. Experimental designs certainly strengthen these assumptions, but do not completely unconfound the sources ...

I've been reading a very interesting book on daily diary surveys. One of the chapters, by Norbert Schwarz, makes some interesting points about how frequent measurement might not be the same as a one-time measurement of similar phenomena. Schwarz points to the well-known studies that he did where they varied the scale of measurement. One of the questions was about how much TV people watch. One scale had a maximum of something like 10 or more hours per week, while the other had a maximum of 2.5 hours per week. The reported distributions changed across the two different scales. It seems that people were taking normative cues from the scale, i.e. if 2.5 hours is a lot, "I must view less than that," or "I don't want to report that I watch that much TV when most other people are watching less." He points out that daily surveys may provide similar context clues about normative behavior. If you ask someone about depressive episodes every day, they may infer that...

As I have been working on a web survey, I'm following more closely the devices that people are using to complete web surveys. The results from Pew make it seem that the younger generation will move away from PCs and access the internet through portable devices like smart phones. Some of these "portable" devices have become quite large. This trend makes sense to me. I can do many/most things from my phone. I heard on the news the other day, that 25% of Cyber Monday shopping was done with tablets and phones. But some things are easier to do with a PC. Do surveys fit into the latter group? Peter Lugtig posted about a study he is working on that tracks the device used in waves of a panel survey. It appears that those who start on a PC, stay on a PC. But those who start on a tablet or phone are more likely to switch to a PC. He also notes that if you used a tablet or phone in an early wave, you are less likely to do the survey at all in the next wave. I didn't read t...

I came across this interesting po st about building a Bayesian model with careful specification of priors. The problem is that they have "tiny" data. So the priors play an important role in the analysis. I liked this idea of "tiny" data. The rush to solve problems for "big data" has obscured the fact that are interesting problems for situations where you don't have much data. Frost Hubbard and I looked at a related problem in a recently published article . We look at the problem of estimating response propensities during data collection. In the early part of the data collection, we don't have much data to estimate these models. As a result, we would like to use "prior" data from another study. However, this prior information needs to be well-matched to the current study -- i.e. have the same design features, at least approximately. This doesn't always work. For example, I might have a new study with a different incentive than I...

The Director of the Census Bureau, John Thompson, recently blogged about a field test for the 2020 Decennial Census Nonresponse Follow-up. They are testing a number of new features, including the use of smartphones in data collection. I've been working with GPS data from smartphones used by field interviewers. The data are complex, but may offer new insights into interviewer travel. Think of travel as a broad concept -- it's not just an expense or efficiency issue. The order in which calls are made may also relate to field outcomes like contact and response rates. Perhaps these GPS data can help us understand how interviewers currently make decisions about how to work their sample. For example, do they move past sampled housing units when they first arrive to the area segment? Is this action associated with higher contact rates? Of course, travel is also an expense or efficiency issue. I wouldn't want pushing for more efficient travel to interfere with other aspects o...