New Tools for Evaluation

Beacon, January 14, 2025

For most of the life of the Bree Collaborative, we have had no formal way to evaluate or align evaluations done by organizations that represent the audiences in our reports. In 2024 the Bree embarked on a new strategy to develop tools that will help support the evaluations of our guidelines, from the state level through to the smallest organizations. The Bree now has a whole suite of tools and resources that organizations can draw on to develop their evaluation plan, identify the data needed to conduct an evaluation, share their evaluation plans, methods and results, and more.

For our recently released 2024 reports we have three new topic-specific tools. The first set of tools, audience specific score cards, were originally developed in 2022 and have been updated for this report revision. The intent of the score cards is to measure process changes recommended by the Bree. For other topics, they have been used by organizations like the Puget Sound High Value Network to measure their current state of Perinatal care and develop and improvement plan and the Thurston County Climate Action Team to support the evaluation of a pilot program.

As part of the work group process we drew individuals from work groups who had expertise in implementation science, evaluation, and health research to help identify metrics and create frameworks for the evaluation of our reports. The frameworks “outlines future evaluation activity that is intended to measure the uptake, concordance of care, outcomes, and impacts of the Bree Collaborative’s guidelines” and “provides guidance for different types of evaluations at different levels across the healthcare ecosystem”. In this framework you will find detailed definitions of the recommended metrics and guidance on what to include in specific types (program, process, etc.) evaluations and what kinds of evaluations may be useful for different guideline audiences.

In parallel with the framework, the Bree staff developed an Evaluation Matrix that acts as a sort of summary the objectives and goals of the recommendations made by the work group and a theory of change which illustrates how the work group expects that these process changes will result in greater system-wide changes to patient experiences and impacts on the health of the population as a whole. Along with the theory of change, the matrix can be used to give you an ecosystem-wide overview of the intent of the guidelines.

These resources are now available online for our Opioid Use Disorder Treatment Report and will be released for Health Impacts of Extreme Heat and Wildfire Smoke and Youth Behavioral Health – Early Interventions, as those reports are published:

• OUD Evaluation Matrix
• OUD Evaluation Framework
• Implementation Guide (score cards located here)

As our reports are implemented throughout the health care system in Washington, we hope that these new evaluation tools will allow organizations to share their ideas and knowledge with their peers as well as share their successes with us. Like our guidelines, we see evaluation as a collaborative effort. Consider this your invitation to join us in learning together.

Karie Nicholas, M.A., G. Dip, Measurement and Evaluation Manager