What You Should Know About Polling

MSNBC is running a Live Vote asking "Should the motto 'In God We Trust' be removed from U.S. currency?" If you want to participate, the poll is here.

As of a few minutes ago, 13% say that Yes, it is a violation of the principle of the separation of church and state, and 87% say that No, the motto has historical and patriotic significance and does nothing to establish a state religion.

These results, and others like them, will of course provide fodder for the likes of Rush Limbaugh, Glenn Beck, and others who will wave the flag and the Constitution when it supports their cause, and wave poll numbers such as these when they do not.

Polls are tricky things and even the "pollsters" hired by news organizations to explain them to us do so in such a way as to present the picture that they are hired to present. There are several kinds of polls, and several ways to gather and present data.

Probably the most important thing to know about polling, is that web polls (Live Votes) like the one currently on MSNBC or the ones frequently offered on local news stations (mine frequently asks me to call or go online and vote yes or no to an open-ended or specious question) are never accurate and may be fun or interesting, but offer no valid or reliable information.

Statistically, results to be at all useful, must be reliable and valid. A scientifically designed experiment - including a poll - is repeatable. Repeatable means that if done over and over again under the same conditions, the same results with minimal changes will occur.

If a test is valid, it means that the answers reached, are correct. You might design a test (poll) that achieves the same results over and over again, but with the wrong answer, over and over again. For example, during the Clinton/Lewinsky scandal, MSNBC had another Live Vote with 200,000 participating. That poll found that 73% felt that Clinton should resign, however, an NBC News/WSJ poll discovered that only 34% of 2,000 felt the same way. (information available through the same link as above).

If you knew nothing about polling, you would have no way of knowing which poll to believe. The Wall Street Journal is considered a fairly conservative publication while MSNBC is considered more liberal, or at least more neutral, so perhaps your politics might guide you. You might decide based on past history of polling results as the best indicator of future performance is past history, or, you might decide which you believed based on the reporter you were listening to or whether or not you liked Clinton or which party you belonged to.

What is interesting about this particular Clinton poll, is that news organizations and political pundits talked to each other more than people outside the beltway. They payed attention to polls like MSNBC's Live Vote because it validated their own opinion. Much of what happened during that time was driven by those in Washington believing that the scandal was a big deal while most of the rest of the country did not.

Anyway, back to polling. If a test is reliable, it means that the results will be the same each time the instrument is used. In the case of a poll, if opinions have changed, the results will change, but if used correctly with a properly drawn sample, the results will be valid. In the case of a political poll, a reliable poll will accurately reflect the opinions of the people being polled.

A poll can be reliable but invalid, but it can never be unreliable and valid. Or in other words, a poll can have well-written questions that elicit good answers, but poorly written questions can never reliably elicit accurate answers.

There are several kinds of polls, and if anyone is interested in more information about the specifics, just write me a comment and I'd be happy to write another post (or write me a comment about anything you'd like me to go into more depth about). What is most important to understand though, are 3 things.

1) A poll that you participate in on your own is virtually meaningless. People who respond to requests to participate whether it's an internet poll or a request to call and vote usually have strong feelings about the issue. If you don't care, you are unlikely to bother to vote.

2) Polls that are valid are conducted using a carefully drawn sample. There is a very scientific method governing how people are selected to participate so that the sample is representative of the population being studied - gender, race, ethnicity, socioeconomic status, geographic location, etc. A poll is more valid if it contains at least 500 people, but is usually no more valid if it contains more than 1,500. There are several ways to design the sample and it depends on the kind of poll and the purpose of the poll. People get degrees on this.

3) Most polling is done by telephone for fairly obvious reasons. In the past, there were flaws in this as whole segments of the population were left out as not everyone had a telephone, or people were on party-lines (not too long ago, rural areas had fewer telephone lines and people had to share them). Then, who was home to answer the phone? During the day, it was typically the woman until more and more two-income families. Calls shifted to the dinner hour, which created more people unwilling to participate. The advent of answering machines and caller ID became a factor, and most recently, more and more people use their cell phones as their primary or only telephone - and cell phone numbers are unlisted. So, designing an accurate poll becomes very difficult.

The results - at least for political pollsters - is that they poll the people they can reach. The registered party members. They use online polling which attracts people who already have a strong opinion.

Unethical pollsters use push-polling, leading questions, poorly drawn samples, small samples (40% of the people polled said .... is meaningless if they polled 10 people). Remember, how many people did they poll, who were they, what questions did they ask, and how did they ask them?

Thanks for stopping by. Come back soon.
k