Friday, January 20, 2017

What is the right periodicity?

It seems that intensive measurement is on the rise. There are a number of different kinds of things that are difficult to recall sufficiently over longer periods of time where it might be preferred to ask the question more frequently with a shorter reference period. For example, the number of alcoholic drinks consumed by day. More accurate measurements might be achieved if the questions was asked daily about the previous 24 hour period.

But what is the right period of time? And how do you determine that? This might be an interesting question. The studies I've seen tend to guess at what the correct periodicity is. I think it's probably the case that it would require some experimentation to determine that, including experimentation in the lab.

There are a couple of interesting wrinkles to this problem.

1. How do you set the periodicity when you measure several things that might have different periodicity? Ask the questions at the most frequent periodicity?

2. How does nonresponse/attrition fit into this? If some people will only respond at a certain rate, what should you do? Is it better to force the issue with them, i.e. make an ultimatum that they participate at the rate we desire or not at all; or better to allow them to participate at their preferred rate?

I'm sure the answers vary across the substantive areas of interest. But it does seem like an interesting set of problems in the evolving world of survey research.

Friday, January 13, 2017

Slowly Declining Response Rates are the Worst!

I have seen this issue on several different projects. So I'm not calling out anyone in particular. I keep running into this issue. Repeated cross-sectional surveys are the most glaring example, but I think it happens other places as well.

The issue is that with a slow decline, it's difficult to diagnose the source of the problem. If everything is just a little bit more difficult (i.e. if contacting persons, convincing people to list a household, finding the selected person, convincing them to do the survey, and so on), then it's difficult to identify solutions.

One issue that this sometimes creates is that we keep adding a little more effort each time to try to counteract the decline. A few additional more calls. A slightly longer field period. We don't then search for qualitatively different solutions.

That's not to say that we shouldn't make the small changes. Rather, that they might need to be combined with longer term planning for larger changes. That's often difficult to do. But another argument for ongoing experimentation with new methods.