Survey says: “I don’t know”

Overview

It is standard survey best-practice to give respondents the option to answer “I don’t know” to questions. That is, unless “I don’t know” isn’t a reasonable response, say if the question is asking a person his/her name. For many researchers, including “I don’t know” as an answer option is a reflex, alongside a “refuse” option, for most closed-ended survey items, that is, questions with fixed answer choices.

But “I don’t know” responses can compromise analysis if the respondent uses it for key questions that might influence the survey’s outcome. We recommend deciding to include “I don’t know” in a questionnaire on a case-by-case basis, after careful consideration.

In this post, we share our internal debate and some ideas on making the decision to force respondents to give a substantive answer by not allowing ‘I don’t know’ as an acceptable response.

Scenario:

A social enterprise called d.light provides solar-powered energy products for homes and small businesses in under-resourced communities. They wanted to understand whether purchasing one of their home solar-powered systems gave households a positive financial return on their investment and if it influenced other outcomes like a family’s productivity (e.g. working, studying) or improved their health.

To answer these questions, d.light commissioned an impact evaluation. In addition to the key outcomes mentioned above, our IDinsight team saw that there might be other variables affecting these results, like total number of light hours used in the home the previous day, and from which sources. We decided to measure these as well.

Problem:

When the results came in, a challenge arose. Thirteen percent of respondents said “I don’t know” when asked the number of hours they used at least one light source. Twenty eight percent of respondents answered, “I don’t know” to at least one outcome variable (their return-on-investment, productive time, or improved health).

This is a mood-dimmer because:

1. “I don’t know” is not very informative.
2. It compromises analysis and interpretation of the results.
3. There are a few options for moving analysis forward, but all have substantial downsides (see addendum).

Debate:

If you scour the internet you can find that debate around addressing this challenge usually falls into two camps. Anti-DK (anti-don’t know) folks think that sometimes eliminating the ‘don’t know’ answer choice produces sounder data and clearer analysis. Pro-DK (pro don’t-know) folks think a ‘don’t know’ answer choice must always be included in the set of answer choices for a close-ended survey item — when ‘don’t know’ is a plausible answer — for the soundest data.

Anti-DK: Conducting analysis without information about key outcome variables can destroy a piece of research. It is very common that a respondent won’t know the exact answer to a question, but in general their best estimate is acceptable — and preferred to missing data — since they have information about the household that can be informative.

Pro-DK: We don’t like destroying research; that’s why we need to “include all reasonable response possibilities as explicit response options”¹. Sometimes the most accurate or truthful answer is, legitimately, “I don’t know.” The respondent could have reasonably been away from the house most of the day prior to the survey and not be aware of how long the solar lights were used. The best guess estimate may be meaningless.

It is the surveyor’s role to prompt the respondent to thoughtfully estimate the answer. Respondents say “don’t know” for a variety of reasons, including, according to Converse and Schuman 1974²:

1. I don’t know (and frankly I don’t care);
2. I don’t know (and it’s none of your business);
3. I don’t know (actually, I do know, but you wouldn’t be interested in what I have to say about that);
4. I don’t know (and I wish you’d change the subject because this line of questioning makes me uncomfortable); and
5. (I wish I could help you but) I really don’t know.

We need to be diligent about training surveyors with the tools and confidence to sufficiently draw out real (if only estimated) answers for the first four types of “I don’t know.” We could, in addition, script specific prompts for survey items that directly affect key outcomes.

As you may have guessed, the two sides of this debate have strong feelings about surveyors stepping in.

Anti-DK: But it is a general rule of (closed-ended) surveying to give as little discretion to surveyors as possible. If there’s a “don’t know” option, it will be overused to avoid pushing respondents beyond the minimum possible response³

Pro-DK: Doesn’t that come back to surveyor incentives?

[Applause for both sides of the debate.]

In short…

Including “don’t know” in a survey is a difficult decision that needs to be made on a case-by-case basis. At IDinsight, we haven’t taken a definitive stance on how to guide these case-by-case decisions.

Please weigh in in the comments — because we… don’t know!

Addendum:

What are the analysis options you can use when you have high percentages of “don’t know” responses? What are their downsides?

1. You can do regression analyses with different samples, depending on the number of definitive responses received for each outcome. But this won’t clarify whether your finding is due to the theory of change or just due to a different sample.
2. You can restrict the sample size to the set of respondents that answered all questions. But this likely reduces your sample and precision. Also, if “don’t know” instances are not exogenous (that is, if saying ‘don’t know’ is also related to the values of key outcomes of interest), it can cause bias.
3. You can use multiple imputation to generate the missing values. But for dependent variables, this doesn’t add any additional information to the system and may introduce extra noise.⁴).