This is a blog post from
If you’ve always suspected that surveys are highly dependent on the way the question is asked, here’s some empirical proof that you’re right. The way questions about sanitation are asked has significantly affected data on the extent of open defecation in India, new research reveals.
In a >working draft paper , economists Diane Coffey and Dean Spears of the >Research Institute for Compassionate Economics compared seven recent studies on toilet use in rural India. They found that surveys which asked a balanced question about open defecation or latrine use, individually, for each person in the household were able to document more open defecation among those who owned toilets than unbalanced survey questions that grouped household members.
So who interviews each person in the household? Three important national surveys do not - The 2011 Census of India, the India Human Development Survey (IHDS), and the National Family Health Survey (NFHS), all report a single household-level measure that combines behaviour and ownership. Four studies considered by Ms. Coffey and Mr. Spears partially disaggregate members of the household. Three others fully disaggregate, i.e. they ask every member of the household.
What is a balanced question? One, say the researchers, in which multiple options are presented without evidence of a bias in favour of one of the options. “Do you use a toilet or do you defecate in the open?” would be a balanced question in this case, as opposed to “Do you use a toilet?” “When asking unbalanced questions, it is easier for surveyors and respondents to speed through yes/no questions, or to tick off default responses, rather than noticing the verbal and non-verbal cues of respondents, which leads to the recording of more accurate information,” they say.
This difference in open defecation rates measured by the two types of surveys is statistically significant (meaning very unlikely to be due to chance alone) and is not because of differences in overall open defecation rates across study sites.
These findings are, for me, particularly significant because the National Sample Survey Office, India’s official source of statistics, does not ask either a balanced question or ask each member of the household about toilet use. This sadly means that in its >most recent round (2011-12), India likely underestimated the extent of open defecation in India. Of the seven studies Ms. Coffey and Mr. Spears looked at, the NSSO in fact found the least open defecation among people who have toilets.
