Why Polls Matter - A Little

By Jill Kuraitis, 5-26-08

There are two basic types of polls:  those with a statistically near-accurate sample of people who match the demographics of a certain population, and those which are self-selecting and therefore meaningless.

If a blog or website posts a “poll” meant to survey the general population and you answer it, you’ve selected yourself to participate – along with another thousand people who happen to read that site – say it’s a liberal blog’s liberal readers. They may recruit their friends to answer the “poll” to drive up the results to their satisfaction. The population who answer the questions have no statistical relationship to the general population, so the poll means nothing. It’s a PR trick.

If the “poll” is meant to survey only the blog’s readers, that’s another matter.

But a poll which means to survey the general American population questions a representative sampling of American demographics – urban, rural, young, old, this percent white, Protestant, Republican, brown, Asian, those who vote, those who don’t – in proportions closely matched to the statistics of the whole population – then the poll means something.

Just not as much as some pollsters like to think.

Of course, which criteria is used to choose a sample is very important.  Let’s say you want to know how many Asian-Americans will vote for Obama or Clinton.  Obviously then your sample needn’t include Polish-Americans or white Scientologists.

In the early 1990’s, I was partners with a pollster.  He did the research, and I took his results and made political media using it as guidelines. The care he took in getting samples exactly right, his questions perfect, and the telephoners properly trained taught me a lot about how polling works.

He was stunningly accurate.  In our time as partners, he was never wrong about the outcome of an election.  Once, he said a district race inside one county was the tightest he’d ever seen – and our candidate took that county by one vote.  Just one.  In another, a county commission race, he predicted we’d win “by a whisker”, and we did – by 77 votes.

Polling is an art as well as a science, with experience, keen observation skills, and plain talent all making a difference.

Professional, statistical polling can be useful.  It’s especially useful in showing trends in gains and losses in a candidate’s chances of winning.  It can point out generally how people react to your candidate after she’s come out in favor of strip mining or against home schooling.

But the margin of error – the number that is sometimes called the confidence interval, or the “fuzz factor” is so misunderstood that many news organizations draw false conclusions, and most people don’t truly understand it.

Even some pollsters.

There are many ways a poll can include errors in its calculations. Some people won’t respond to the questions or aren’t available. Telephone interviewers could influence people by their tone of voice or casual comments or in other unconscious ways.  The way the questions are written and the order in which the questions are asked can matter. 

But those variables are not always included in the margin of error statement by the pollster.  If a poll states its margin or error at plus or minus three percent, that may be only the theoretical number based on a response rate of 100% in a poll which is a masterpiece of perfection, and there are no other sources of error.

On one day last week, the Gallup poll had Clinton at 39 and Obama at 55 – among Democrats and D-leaners.  Gallup’s margin of error is stated as plus or minus three percent.

The ABC/Washington Post poll had Clinton at 41 and Obama at 53 among D’s and D-leaners - a week before the Gallup poll. They don’t post their margin of error.

The Quinnipiac poll had Clinton at 41 and Obama at 45, with 14 undecided, among Democrats.  They don’t state if they included D-leaners in their sample.  Their margin of error is stated as plus or minus 4.3 percent.

But all three polls’ sample sizes were different, ranging from 1,000 to 1,700.  The precise day of the ABC and Quinnipiac polls isn’t posted, and daily news events can change polling numbers fast.  The questions were written and asked differently on all three polls.  And there were more variables.

Apples and shoes.

What in the world to make of polling numbers, then? Here’s one way to decide what you think:  Simple averaging of the inconsistently-performed above three polls put Clinton at 40% and Obama at 50%, a fairly dramatic ten-point difference.  With a three percent margin of error, that could be Clinton 37, Obama 53 – even more dramatic.  Or Clinton 43, Obama 47, which paints a different picture indeed.

It could be even wackier, if the margin of error included all the other possible errors written about here.

The body of evidence and research about polling is enormous, and this is just a basic overview without the detail the subject needs to be well understood.

Then there’s the issue of a campaign’s own polling being used to put a positive spin on her chances – but that’s a subject all by itself which I’ll hold for later.

The National Council on Public Polls has some great information.