In our latest dig, we examine political public opinion polls. Mainstream Media (MSM) organizations often publish, quote, and discuss opinion polls live on television, but do they even matter? Look at this opinion poll article titled Slim majority of Americans want Senate to convict Trump: Reuters/Ipsos poll that Reuters recently published online. I am not specifically picking on Reuters because I believe they are a reputable news source.
The following paragraph is the disclaimer for the poll, which is the subject of the article:
"The Reuters/Ipsos poll was conducted online, in English, throughout the United States. It gathered responses from 1,115 American adults, including 538 Democrats and 373 Republicans. It has a credibility interval, which is a measure of precision, of 3 percentage points." (Kahn, 2021)
The disclaimer is professionally written, accurate and gives the article a semblance of legitimacy. We are only going to focus on the disclaimer today. Please read the poll yourself if the headline piques your interest. There are two dimensions to consider. What does it say? And what does it not say?
The disclaimer immediately highlights the existence of built-in bias. I won't presume whether this was intentional or unintentional. The poll was "conducted online," so the reader immediately knows that this poll can miss older age groups in the population who vote but often do not frequent the internet. It also excludes people who cannot afford the internet or may have poor internet access options. The Pollster did not use telephone or man/woman on the street collections for this poll, limiting the sample group. This poll cannot deliver a truly random sample from the entire population based on these factors alone. (Huff, 1954, p13-28)
How was the poll taken online? Was it on a website or via an e-mail? We can assume either because the same rules would apply to both. If the Pollster collected data from a website, then the population only includes those who frequent that particular website. If the Pollster collected data using e-mail, it only consists of a portion of the people who reside on the e-mail list. Does that sound like a valid random sample of the entire population of America? It is not. The poll is biased.
It is easy to find two separate opinion polls that focus on the same topic with different and opposing outcomes. Opposing opinion polls may have a different built-in bias that gravitates the respondents in opposite directions. Any public opinion poll is merely a snapshot at a particular point in time and based on a specific group of people. The media should never use opinion polls to forecast future events. There is too much uncertainty in these polls, and opinions can vary drastically from one day to the next.
The phrase "throughout the united states" tells us something but leaves out a lot as well. Throughout means in all parts by definition. We can assume that the poll was conducted in all 50 states, but did the Pollster receive respondents from all 50 states? We do not know. Did the respondents live predominately in urban cities or rural areas? We do not know. All of these things will inherently bias the final results.
The biggest thing we don't know is the age groups of respondents, which must be understood to make the poll information useful and trustworthy. American adults indicate that all respondents were age 18 or older, but what was the cross sampling? Did the poll have respondents from every age group? We do not know. Once you start getting a large amount of "we do not know" from the poll disclaimer, its validity begins to falter and eventually craters into meaningless information.
How many people did Reuters/Ipsos poll who responded vs. those who did not respond? They only tell you how many people responded, and this a relatively tiny poll, by the way. Could it be that 10% of those polled responded? 20%? 50%? 75%? This matters in statistics. It's crucial information. As a casual reader, you cannot apply tests of significance or come to exact conclusions about the sample's adequacy based on Pollster's provided disclaimer. The media often quote biased opinion polls and seemingly try to predict the future with them, and Republicans and Democrats are equal opportunity offenders.
The credibility interval is a creative way to make you feel good about the data presented. It makes people feel like these people are an authority, but is the sample large enough to draw any reliable conclusion? Based on Reuters' disclaimer, I would have to testify under oath with a No response. An economical way to opinion poll is to use stratified random sampling. "Divide the universe into several groups in proportion to their known prevalence." Translation, get this many respondents from different ethnic backgrounds, sexes, sexual preferences, income brackets, jobs, ages, etc. The numbers provided by Reuters tell us none of that; therefore, we do not know. (Kahn, 1954, p. 23)
The poll I discussed here has many blind spots from bias. It is not helpful or useful, and it certainly should not be referenced to make a point or win any public debate or argument. Always bring a healthy dosage of skepticism whenever reading public opinion polls. Find the poll disclaimer first because it will tell you all that is seen and unseen by the Pollster.
Good Luck!
FYI - We announce all new digs on Twitter. Follow us @newdigger_org
References:
Kahn, C. (2021, January 22) Slim majority of Americans want Senate to convict Trump: Reuters/Ipsos poll. Reuters. https://www.reuters.com/article/us-usa-election-poll/slim-majority-of-americans-want-senate-to-convict-trump-reuters-ipsos-poll-idUSKBN29R2M9?fbclid=IwAR3JAIfSYSMaaJHHIO-hJKyokIIL-XNAyjLasOzNJ8muxn_wYgOIrsp2RgE
Huff, D. (1954) How to lie with statistics. W.W. Norton & Company, Inc.
Comments