So what about that Ipsos Reid survey which gives credence to the federal government proposed pension changes?
The results of a poll, conducted by Ipsos Reid for Sun Life Financial, were recently released as part of Sun Life Financial’s “Unretirement Index.” The publication of those results followed The Prime Minister’s unexpected announcement of changes to the Old Age pension during a speech in Davos at the World Economic Forum. The results of that poll appeared to support the changes.
The best article I found which tells the story of announcement and survey is Andrew Moran’s “Study: Majority of Canadians to work past 65 amid uncertainty”
Andrew Moran is a busy independent journalist/blogger. His credits include “libertarian op-ed pieces to Crucial Politics.” The story appeared Feb 22, 2012 in Politics
An aside on the article: The article, while it gave a good general survey of current media reporting, had a couple of errors. It identified the pension under discussion as the CPP [Canadian Pension Plan, which is partially funded by work deductions] rather than the OAS [Old Age Security Plan which is wholly funded by the federal government, and one of the few remains of the safety net. One wonders if the misunderstanding is general. The other error may be innocent, but … He called Statistics Canada an agency, rather than a government department. Is this an indication of Stats Can’s continued loss of status?
Various media broadcast the results of the poll with headlines such as: "Canadian seniors expect to keep on working: Poll” [Headline, 24 H NEWS, Thursday, February 23 2012, pg. 5.]; “Canadians expect to work past age 66: Poll” [Metro News, Thursday, February 23, 2012, p. 18]; “Most Canadians plan to work past age 65: poll” 02/22/2012 | Marcia Chen, CityNews.ca.
I wondered about those results. The poll appeared to support the government’s reported plan to change the age of qualification for the OAS from 65 to 67. The conclusion that, for many seniors, the delay in retirement was financial hardship appeared to be glossed over. Many of the news stories repeated Sun Life’s own list of conclusions, without attribution. They concluded “a majority of Canadians expect to work past the age of 66 due to a variety of reasons, such as longevity, an increase in debt, the rising costs of healthcare and the lack of money for retirement.” I found the order of that listing troubling.
Here’s the thing about statistics and polls, surveys, etc. First, a lot of their influence is formulated in the reporting. That is, the results that are highlighted or included in the capsule account appear to be the most important. In fact, the conclusions should be weighted evenly. Of course, where the results can be divided into those supported by strong definitive results and those who have only a suggestion of an implication, it is proper to report that difference, and proper for news stories to reflect that difference.
The second thing about statistics, especially when derived from polls and surveys, is they are only as good as the methodology. There are rules about how such tools are used. That the conclusions end with the confidence formula does not mean that you should accept the results without question.
In this case, I found the results and the reporting, when fully reported in the more comprehensive news story, to not fit my bottom-up view of the world. They included:
I felt that the reason for the change in the expectations for retirement should have received much more media attention. Ipsos Reid’s own report included this striking paragraph:
“Fully six-in-ten (61%) of those who will be working at age 66, say the main reason is because they “need to”, not because they “want to”. The number one most often cited reason they will be working at the age of 66 is to earn enough money to pay basic living expenses, (23%, up from 15% in 2009), followed by to earn enough to live well (20%), I don’t believe government benefits will be enough (19%) and to stay mentally active (14%)” ( Ipsos Reid press release “HALF OF CANADIANS PLANNING A PHASED-IN RETIREMENT” )
For me, that was the story.
I really worried about the methodology. You see, I belong to an Ipsos Reid online “I-Say” panel. These panels are self-selected and therefore not representative of any population. My experience on the panel leads me to suggest that it is possible that Ipsos-Reid would further select respondents based on their own formulae for an even representation. Further, they stipulate that they “employed weighting to balance demographics and ensure that the sample’s composition reflected that of the adult population according to Census data.”
Because they are represented as using a sample that reflects Canada’s adult population, it seems fair to look at the methodology used in surveys which attempt to be truly representative of a population. The following guidelines from the UN (2003), describe the rules of sampling generally used collect information on a national population. One of the first things emphasized is the necessity for using Probability Samples:
“It is this mathematical nature of probability samples that permits scientifically-grounded estimates to be made from the survey. More importantly it is the foundation upon which the sample estimates can be inferred to represent the total population from which the sample was drawn. A crucial feature and by-product of probability sampling in surveys is that sampling errors can be estimated from the data collected from the sample cases, a feature that is not mathematically justifiable when non-probability sampling methods are used.” (Anthony G. Turner, “Sampling strategies” Expert Group Meeting to Review the Draft Handbook on Designing of Household Sample Surveys, Statistics Division, UNITED NATIONS SECRETARIAT (ESA/STAT/AC.93/2) 03 November 2003.)
So, basically, if the survey or poll does not follow probability sample rules, then the poll or survey cannot be used to make inferences about the population and it is not possible to know the size of the sampling error. What element makes the probability sample mathematically solid? The paper cited above gives the following:
“Probability sampling in the context of a household survey refers to the means by which the elements of the target population – geographic units, households and persons – are selected for inclusion in the survey. The requirements for probability sampling are that each element must have a known mathematical chance of being selected and that chance must be greater than zero and numerically calculable. It is important to note that the chance of each element being selected need not be equal but can vary in accordance with the objectives of the survey.”
Doing research using surveys: methods: questions, numbers of respondents
Let’s begin with the source of the poll, as reported by Sun Life Financial. They conclude their article with this paragraph:
“These are the findings of an Ipsos Reid/Sun Life Financial poll conducted from November 29, 2011 to December 12, 2011. For the survey, a sample of 3,701 adult, working Canadians between the ages of 30 and 65 was interviewed from Ipsos’ Canadian online I-Say panel. Ipsos employed weighting to balance demographics and ensure that the sample’s composition reflected that of the adult population according to Census data. A survey with an unweighted probability sample of this size and a 100% response rate would have an estimated margin of error of sample of ±1.6 percentage points, 19 times out of 20, of what they would have been had the entire adult Canadian population been polled. The margin of error would be larger within regions and for other sub-groupings of the survey population. All sample surveys and polls may be subject to other sources of error, including, but not limited to, methodology change, coverage error and measurement error.” [ web address of their article]
[ Further information was found in their downloads, the Press Release and the Detailed Tables, found on the same page.The numbers from the survey are taken from their "Detailed Tables”]
Does the Ipsos Reid Poll present itself as representative of a national population? They stipulated that they “employed weighting to balance demographics and ensure that the sample’s composition reflected that of the adult population according to Census data.” Did they used probability sampling? By using the online survey method, Ipsos Reid eliminated certain portions of the population. They eliminated:
- Anyone who chooses not to indulge or participate in online surveys. I dare say that the portion of the population who take part in online surveys is fairly select.
- Anyone without east access to the internet.
- Anyone who does not want to reveal their economic status.
You wind up with a fairly small portion of the population. It is surprising that their conclusions included the data on those who felt they had to keep working.
From my experience taking part in Ipsos Reid surveys, I believe it is entirely possible that exclusionary questions were used. An exclusionary question could be “Have you used any agents or services to assist in planning for your retirement?” Another example could be “What is your work status?” with choices that include “not working, ” “employed part time,” “employed part time,” etc. With the first question you can eliminate anyone who cannot plan for the future by reason of their low income. With the second question, you can eliminate anyone not employed.
So we see how the portion of the population of Canada which is represented in the poll has been significantly narrowed.
There is another surprising way in which the results of the poll have been made even less representative. They stipulated that they “employed weighting to balance demographics. . .” This is normally done to correct for demographic skew. One would employ a set of correct percentage data on the focus group from a trusted source, compare it with the percent used in the poll, and then correct the poll results to reflect the difference in the ratios.
Their own statement reveals that it was the comparison of their sample to the general population of Canada, including the proportion of respondents in each of three income categories which was weighted to reflect “the adult population according to Census data.”
So, if we examine the evidence:
Their numbers or respondents by income: under $50,000 [1025] ; $50,000 – $100,000 [1421] , over $100,000 [801]. [The Detailed Tables, available here ]
The Statistics Canada data is available on Table 111-0008. It lists income brackets by increments of $5000 and gives numbers for the stated income and over. To arrive at the number and percent of the population for the three categories used here, it was necessary to subtract the amount for the income level from the total before changing it into a percentage.
Compare that with the Population of Canada by income: under $50,000 [ 24,964,290 -6,458,940 = 18,505,350 ] = 74%; $50,000 to $100,000 [6,458,940 - 1,343,150 = 5115790 ] = 20%; $100,000+ [ 1,343,150 ] = 5%.
{Statistics Canada. Table 111-0008 - Neighbourhood income and demographics, taxfilers and dependents with income by total income, sex and age group, annual (number unless otherwise noted), CANSIM (database). } Online
The Ipsos Reid survey weighing according to the tables gives: under $50,000, 1217 respondents; $50,000 to $100,000, 1436; and +$100,000, 633. They had a total of 3701 respondents of survey.
This means that their under $50,000 = 33% of respondents, $50,000 to $100,000 = 39%, and +$100,000 = 17%.
{www.ipsos-na.com, (home/ news and polls/ Half of Canadians Planning a Phased-in Retirement.}
This means that for the portion of the Population with a yearly income of under $50,000, which according to Statistics Canada is 74% of the total, is represented by 33% of the respondents in this survey. The portion between $50,000 and $100,000, 20% of the population, represents 39% of the respondents in this survey. The portion with an income of over $100,000, at 5% of the population, is represented by 17% of the respondents in the survey.
Is this representative of the population of Canada? Could this mean that a much higher proportion of Canadians are actually worried about life after retirement and preparing to work as long as they are able because it is necessary?
Never mind the dumb horse.
May 8
Posted by mollyclendon
About the horse? I baled in January. Mostly done with partnership and electronics.
I am gearing up for gardening with two girlfriends. Look for commentary on public land use, encouraging action and working together.
I’ve been spending a lot of time and effort doing political commenting on Facebook. I know, I know. Pearls before swine. I will be moving that here. Some of what’s going on just needs bigger commentary. I might even throw in the not-obviously-related rant or two.
I’m also getting seriously into new family positions: great-aunt and grandma. How sweet it is. And isn’t it amazing how some home knit gifts to toddlers can warm up a relationship.
Hmmmm — I might even talk about the horse – a little -
Posted in personal experiences and issues, politics, social justice
Leave a Comment
Tags: commentary, community garden, Facebook., family, knitting, Online Communities, politics, ranting, Social Networking