column
Another Episode of Damn Lies and Statistics
How a question is phrased makes a huge difference in the results.By Sharon Fisher, 3-22-10
We get it all the time, by phone, paper, or online. “Would you like to take a survey?” Then, through the magic of statistics, we learn all sorts of amazing things.
Not so fast.
As Mark Twain (or Disraeli, depending on which Internet quote engine you use) once said, there are three kinds of lies: lies, damn lies, and statistics. And it doesn’t take a lot of learning about surveys to realize how often and how easily statistics can be misused.
Take this week.
The survey itself was actually from January, but I just encountered it a few days ago when it was mentioned in a Facebook feed. “In December, [Idaho Freedom Foundation] commissioned a poll that found that about two-thirds of Idahoans support state-passed legislation that keeps the federal government from dictating whether Idahoans buy health insurance.”
Really? That was interesting. I asked for details. So here’s the question that elicited that response:
Should lawmakers pass legislation that protects Idahoans from being forced to buy health insurance and protects Idahoans from having to enroll in a government insurance program?
Oh, God. Where to start.
1. There is no information about the size of the survey population or where they were drawn from. For statistics to work and be valid, it’s important that the population be of at least a certain minimum size and to be randomly chosen. A survey population where everyone is the same demographic, or from the same area, or who responded to a request for survey participants, isn’t random, any more than asking only Idahoans would give you a clear picture of the entire United States.
2. For a survey to work, the questions need to be objective. If it’s too obvious what sort of answer the surveyor is fishing for, it’s going to skew the results.
So let’s look at the question. It talks about ‘protecting Idahoans’ twice in a single sentence, as well as ‘being forced to buy health insurance’ and ‘having to enroll in a government insurance program.’ Using phrasing like this pretty much guarantees that people are going to say “yes,” because it sounds scary. In fact, if anything surprises me, it’s that the result is only 66 percent.
Around the same time, the IFF did another survey. This one gives a little more information, such as the fact that 400 people were surveyed in six regions of Idaho—meaning they weren’t all picked from a single town, for example. So the population is large enough, and diverse enough geographically, at least, to show some indication that the population may be random.
However, there’s the same problem with the question. Idahoans were asked, The Legislature is getting ready to cut spending this year. Should the Legislature cut taxes as well in an effort to spur economic development, increase personal income and create jobs? Again, about the same two-thirds number agreed with this sentiment.
Well, sure they did. Lots of people are going to agree with a proposal to cut taxes, particularly when they’ve just been told the Legislature is also going to cut spending. Plus, who’s not going to go along with something that’s going to “spur economic development, increase personal income and create jobs?” Hell yes. Sign me up.
I’m not going to debate the issue of whether cutting taxes spurs economic development and creates jobs. The point is, phrasing the question in that way makes it more likely to elicit that response, just like asking “Should the Legislature cut taxes even though it will affect children’s education and make it harder for senior citizens to get medical care?” might elicit a different response.
This doesn’t mean, of course, that Idahoans are in favor of a federal health care plan or are against cutting taxes. What it does mean is that with questions phrased that way, we can’t be sure, and we can’t trust the responses we get to such questions.
Full disclosure: Sharon Fisher is a candidate for the Idaho Legislature, District 21.
Like this story? Get more! Sign up for our free newsletters.
Comments
Add your comment below
Push polls on the other hand use biased questions to change the opinions of people as they are being asked the questions. Push polls are a propaganda technique.
The public rarely draws the distinction: a poll is a poll.
The news media could help, but most news organizations also use unscientific "polls" to gather garbage data about "public opinion" on their Web pages. A proper survey costs thousands of dollars but cash-strapped newspapers can put up a question on their Web site at no cost and make it look like they're doing their job.
The Idaho Freedom Foundation is not a reliable source from everything I've seen. When it comes to rooting around and re-publishing public records, I can give them the benefit of the doubt, even as I presume WHICH things the dig up are designed to push their agenda.
For surveys and stats and the like, I have no reason to believe Wayne and whoever works for him are competent, and I have every reason to believe they're strongly biased by their ideology. Either fault by itself will produce untrustworthy results; together, they make trustworthy results exceedingly hard to come by.
But IFF is about lobbying, first and foremost, entertainment to the extent that it serves its first purpose, and promulgating objective information not at all.
-Jon Cheever.
“If it’s too obvious what sort of answer the surveyor is fishing for, it’s going to skew the results.”
This has nothing to do with the problem, and when I read the question it doesn’t become obvious to me what the desired result is…..However. Most Nouns, Verbs, and Adjectives have either positive or negative connotations. For example Child like and childish, they both mean the same thing but one is an insult and one can be a compliment. And as you hinted at the wording is biased although you decide to call it “scary”. A few more problems with the question; depending on what side of the aisle you read it from it can mean two very different things. This is no good, every respondent needs to understand the question the same way. The question is double barreled meaning it asks two separate questions. Am I supposed to answer whether or not I want “lawmakers pass legislation that protects Idahoans from being forced to buy health insurance”, or if I want protection “from having to enroll in a government insurance program?” This is no good. And possibly the biggest problem is the fact that there are only two answers to chose from YES or NO. So here is what I would have done.
The federal government recently passed legislation requiring everyone enroll in a health insurance plan. How would you feel about the Idaho sate legislature passing a law in order to void this provision?
Strongly Agree
Agree
Don’t know
Disagree
Strongly Disagree
And then if you want you could ask another question about the government insurance program. I find it somewhat interesting that you take them to task over the way the posed the question but didn’t offer a better version. Not to sound like a mean spirited person but next time do more research. There is tons of info out there on best practices for survey design. May your political career take you as far as you can take it.
"Congress recently passed a law intended to provide health insurance to more Americans. Part of the way this is implemented requires Americans to buy health insurance (with financial assistance provided to low-income people). Should lawmakers pass a law stating that Idahoans do not have to follow this requirement?"
I'm sure it's not perfect; it's just what I could come up with at the spur of the moment.
Bloomberg's NRA-owner poll by Luntz was a recent classic, as was Trout Unlimited's poll about Tester's wilderness bill. If S-1470 actually did the things the poll question claimed, I'd support it.
I have to wonder about telephone polls any more....who the heck is around to answer the phone, and d'ya think anyone with a BRAIN is going to sit through a push-poll when it becomes clear?
Not me.
I think maybe I've been called twice. The first thing I do is ask the questioner to spill for whom the poll tolls. Both times, the poll terminated....
Eventually your phone number disappears from the data bases many of the polling companies sell back and forth to each other, and you're not bothered any more.
Q: Some people believe a federal law requiring every American to buy or obtain health insurance is unconstitutional. If that law passes, some states have announced that they will sue the federal government to fight that requirement. If the health care reform passes, would you favor having your state sue the federal government to prevent it from becoming law?