Most people involved in the SCI BC Peer Program will tell you that it’s special—that it creates a space for ‘magic moments’ to happen between people with shared experiences of SCI.
The challenge with ‘magic moments’ is that they’re hard to measure. And capturing the magic of these programs in a quantifiable way is needed to show their impact to the funders who make them possible, or at least, affordable.
That’s why the SCI Peer Support Community-University Partnership—a group of students, researchers, and community-based organizations, including SCI BC, that aim to better understand, promote, and optimize SCI peer support programs and services—recently developed a new tool to assess the outcomes of peer support programs in SCI organizations.
“The idea behind the tool is that the organizations want to show impact. There’s a need for it. And from a research side we also wanted to highlight that impact quantitatively,” says Dr. Shane Sweet, the partnership lead and an Associate Professor at McGill University. Sweet is also a Tier 2 Canada Research Chair in Participation, Wellbeing, and Physical Disability.
“In our first partner meeting, one of our partners was like, ‘We really want to show the magic of peer support.’ So, a running theme throughout our entire partnership was, ‘How are we going to measure this magic?’” he says.
The Magic of Peer Support
Peer support programs foster interactions between individuals who share similar lived experiences, such as SCI, with the goal of helping them to adapt and thrive. They most often use a discussion-based approach featuring one-on-one or group gatherings.
While peer support programs are widely available in Canada and are believed to improve overall wellbeing and quality of life for people with SCI, there isn’t a lot of high quality-evidence to show that they work—despite the magic we so often hear about through testimonials from peers.
At SCI BC, the Peer Program offers four broad types of services for members that go beyond the traditional one-on-one peer mentorship model.
“SCI BC is unique in that we have the peer meetups, which is your scheduled weekly or monthly meetups, often coffee groups or dinners. Then you have peer connection, which is that one-on-one peer mentorship. Then we have large-scale events, which can be outings like bigger barbeques or kayaking experiences or overnight fishing. And we also have peer coaching, which is more directed towards a specific outcome or goal that you’re trying to achieve through peer support,” says Dr. Rob Shaw, SCI BC’s National Peer Support Program Lead.
“We offer over 300 different events each year. These range from larger scale overnight trips, education panels, and book clubs to smaller, intimate online sessions… So, it’s a really big mix, which brings in a lot of people, and it kind of shows the utility of peer support, that it can be used in more than just one way,” he adds.
Before coming to SCI BC, Shaw was a volunteer SCI peer mentor with SCI Ontario and completed a Postdoctoral Fellowship with Sweet and UBC Okanagan’s Dr. Kathleen Martin Ginis, where he played a central role in developing the new SCI Peer Support Evaluation Tool. He is putting the tool to good use in his role with SCI BC, which involves leading SCI Canada’s National Peer Support Working Group, a collective of provincial SCI organizations working together to improve the consistency and quality of peer support programming across Canada.
The SCI Peer Support Evaluation Tool
The SCI Peer Support Evaluation Tool came from a need identified by partner organizations, including SCI BC, SCI Alberta, SCI Ontario, and Ability New Brunswick, in the SCI Peer Support Community-University Partnership.
What they needed was a tool to help them demonstrate the impact of their peer support programs with the goal of promoting sustained funding and, in turn, improvements to programming.
The first step in developing the tool involved identifying outcomes associated with peer support. “We believed that understanding the outcomes associated with peer support could help take us one step closer to identifying its magic,” says Sweet.
To do this, the team reviewed existing research and interviewed peer support users, identifying 87 different outcomes. They then conducted two Delphi studies, a consensus-based process that allowed them to identify the most important outcomes for SCI peer support, through which they identified 20 key outcomes.
“We went through a Delphi procedure where we had peer mentors, mentees, and staff rate the 87 [outcomes] based on specific criteria… then we did another Delphi with the national directors and peer support managers,” explains Sweet. “So, it was all through consultation and consensus methodologies…peer mentees, mentors, and norganizations are the ones that voted to identify which outcomes were most important for this context.”
The next step involved finding existing measurement tools to assess each of these outcomes. Similar to the process for identifying key outcomes, the team reviewed existing research and went through a consensus-based process with researchers and community partners to identify measurement items that best represented the 20 outcomes.
At the end of this process, the team had identified 20 items, one to assess each outcome—ensuring that the tool could be completed quickly and easily by program users. Each item began with a neutral statement, followed by a five-option rating scale. For example, to assess the ‘community engagement’ outcome, the item reads: Thinking about my experience with [the peer support program], I participate in programs, activities, or events in the community: (1) much less often, (2) less often, (3) as often, (4) more often, or (5) much more often.
The final step involved gathering feedback on the tool from leaders and peer support program coordinators at provincial community-based SCI organizations, then asking peer support users to complete the tool while providing live feedback to a researcher in what’s known as a ‘think aloud’ method. “The think aloud process asked an individual to speak out loud and to have all the thoughts that are running through their head while they’re answering that question. And so, through that process, you’re able to gauge how clearly the question is being understood,” explains Shaw.
Then, 10 days later, the participants completed the tool again on their own. Overall, participants found the items easy to understand and relevant to their peer support experiences, and the two tests generally produced the same results for each participant, highlighting the overall rigour and consistency of the tool.
Measuring the Magic
The tool offers several benefits to both program providers and program users, like those of you involved in the SCI BC Peer Program. For example, organizations like SCI BC can use the tool to identify the strongest and weakest parts of SCI peer support programs, providing a guide for how to improve program delivery. It can also be used to show the impact of programming to funders, which could bring in more funding to support programming.
“This tool has the potential to provide a lot of really strong evidence of the benefits and impact of peer support, which then could strengthen grants, which then could bring in more money, which would allow for more programming or higher quality programming,” Shaw says.
The tool is also customizable, meaning that individual outcomes and items can be selected to best match the goals of a specific program. “It’s an extremely practical tool in the sense that we can’t state how organizations will use it because it’ll be up to them,” explains Shaw. “Some might choose to administer it online. Some might choose to do it over the phone. Some might choose to have all 20 items included in their outcome tool. Some might only choose to have three or four. So that’s kind of the really neat thing about this tool is it can be customized to the exact needs of each organization.”
Taking the time to reflect on how the elements of a peer support program target specific outcomes is important for organizations to identify which outcomes they should measure. This process can provide organizations with more valuable information on whether their peer support program is impacting the outcomes they are trying to achieve.
Another key benefit is that the tool can be administered quickly and easily to program participants. According to Shaw, it takes about 15 to 25 minutes for a participant to complete the 20- item tool. “We tried to keep it pretty brief with the 20 items… but it could be as short as three or four questions, which would obviously take much shorter,” he says.
Putting the Tool into Practice
Shaw presented the tool to the National Peer Support Working Group in May 2024 and the response so far has been extremely positive, he says. While some organizations have been able to start using the tool to evaluate their programs right away, others are using the tool to support program planning and design.
“Some organizations don’t have a peer program, but they’re using the tool to then decide what might be core components of the peer program. So, it’s used not only as an evaluation, but also as a planning tool,” explains Sweet.
As a planning tool, organizations could figure out the outcomes that are important for their organization and then design the elements of the peer support program to target those outcomes.
“There’s also a nice spot in the middle where you can use [the tool] to revamp your program. Learn what you did wrong. Learn why it didn’t impact what you wanted it to. Revise what you’re doing and try it again. And so, through that process, I think you’ll have programming and services that are a little more highly tuned if you want it to be that way,” adds Shaw
According to Shaw, SCI BC is currently reviewing the tool to determine which outcomes are most applicable to its Peer Program, which involves identifying how the tool will be applied to each of the different types of programming.
As a next step, the working group has discussed how this tool can be used collectively to benefit all organizations. Through a series of consensus-based surveys and in-person meetings, the working group decided on three ‘core’ outcomes from the tool that every organization will evaluate (community engagement, quality of life, and reduced isolation).
Having consistent, nation-wide data is intended to support joint efforts to secure funding for peer support programs across the country. “It’s harder to get grant money in the smaller provinces… so if we can get some more consistency with how we’re evaluating these programs across the country, then maybe there is the opportunity to join together and apply for federal level funding that can then be dispersed out to some of those provinces that face more challenges than others,” says Shaw. “So, it’s not necessarily all about funding, but I think for the smaller organizations, there’s a really big opportunity there to have that pooled data.”
A Final Word for Our Peers
Of course, the success of the tool depends on how well participants of peer support programs are engaging with it. “The outcome tool is only going to be important and impactful if we have people using it and filling it out,” says Shaw. “A fear of all of us in the working group is, ‘How are we going to get people to fill out this survey?’ If we want to sustain the programming and improve on it, then we need people to engage in the evaluation of it.”
So, for our readers out there who are current or future members of the SCI BC Peer Program (maybe you’re planning to join the program now that you’ve heard all about the magic it offers) please consider answering a few questions from us about your program experience if they ever come your way. We’re asking because we value your feedback, we want to understand your
experience, and ultimately, we want to make the program better for you.
To learn more about the SCI BC Peer Program, check out sci-bc.ca/peer-program. For more information on the SCI Peer Support Community-University Partnership and the SCI Peer Evaluation Tool, visit mcgill.ca/scipm.
