Evaluation and Improvement

Over the past decade, “outcomes” has gone from a buzzword in the nonprofit, government, and foundation worlds to a full-fledged movement. As the outcomes movement and outcome-based decisionmaking have grown, many models and frameworks for applying this thinking have emerged.

Though evaluators and practitioners have benefited greatly from the development of various tools to guide outcomes thinking, understanding the unique advantages of each model and how to select the right one can be challenging. Responding to this challenge, the Rensselaerville Institute for Outcomes published Outcome Frameworks: An Overview for Practitioners in 2004. This book offers insights into which model tool might be appropriate for the particular needs of a program at a given time. The tools described in Outcome Frameworks fall into three main categories:

  • Program planning and management: Program planning and management tools are outcome models that support an effort’s proposal, funding, and implementation phases. They illustrate the logic, theory of change, and anticipated flow of an intervention, providing markers against which both incremental and ultimate progress may be measured.
  • Program and resource alignment: Program and resource alignment tools ensure that resources and efforts are expended in support of organizational goals.
  • Program reporting: Program reporting tools allow organizations to capture and communicate the fullness of the results they have achieved.

The matrix below provides an overview of major outcome-based tools in use, with information about which one might be appropriate to the particular needs of a program at a given time and how they may be used in an evaluation. Understanding the unique advantages of each tool and what it is well-suited for will help state leaders and evaluators determine which tool to use in evaluations.

Table 1. Overview of Evaluation Tools for State and Territory Leaders[1]

Program Planning and Management

Logic Model: diagram representation of a program showing what it is supposed to do, with whom, and why

Key Concepts Strong Points Uses
Inputs, outputs, outcomes; arrows show relationships between elements in the model Easy to use; provides an easily understood representation of a program’s theory of change Program overview; presentations; program and evaluation planning

Outcome funding framework: key management focus on the achievement of specific, sequential results for customers of services; emphasis on results, not activity

Key Concepts Strong Points Uses
Investor return, results, customers, milestones, performance targets, outcome statements Highly disciplined approach that serves both program investors and implementers; web-based software has strengthened usability Government and philanthropic grantmaking; program and organization management

Results-based accountability: real-time approach that describes what desired results look like, defines results in measurable terms, and uses measures to drive action plans for improvement

Key Concepts Strong Points Uses
Results, experience, indicators, baselines, strategy, action plan and budget, accountability Thorough system for planning community-change efforts and improvements in program, agency, or system performance; uses lay language; provides direct link to budgeting; useful for integrating different outcome systems Project planning and start-up; development of community report cards; program and agency improvement plans and budgets; grantmaking and evaluation design

Targeting outcomes of programs: tracking progress toward achievement of targets; evaluating degree to which programs impact targeted conditions

Key Concepts Strong Points Uses
Knowledge, attitude, skills, aspiration; process, outcome, and impact evaluation Fairly easy to use; helps integrate program development and evaluation; implementers and managers can use same concepts Program design and evaluation

Collective-impact evaluation[2]: not focused on assessing programmatic impact but on complex initiatives working in complex environments where progress is not linear or predictable.

Key Concepts Strong Points Uses
Nine propositions to help evaluators navigate the unique characteristics of complex systems, improve their practice, and serve the social sector Seeks to understand and describe the whole system; focuses attention on context and being responsive to changes as they occur Evaluating systems initiatives and complex, multifaceted efforts

Program and Resource Alignment

Balanced scorecard: business-based model designed to provide integrated management and accounting for multiple variables impacting organization performance by connecting them to a set of performance indicators

Key Concepts Strong Points Uses
Strategy, alignment, short- and long-term objectives, financial and nonfinancial measures, lagging and leading indicators, performance measures and drivers, internal and external indices of success Allows for a graphic assessment of the degree to which an organization’s resources and efforts support its goals Monitoring either a single program with several associated initiatives or multiple programs within an organization; analyzing alignment of resources and initiatives to strategic targets

Program Reporting

Scales and ladders: graphic tool that centers around a series of scales and their placement within a matrix designed to illustrate progress along a continuum of stages

Key Concepts Strong Points Uses
Scales; mutually exclusive, multiple, and floating indicators Places a client, community, or program on a continuum; shows incremental and relative progress, stabilization, or decline; individual data together tell a complete story; behaviorally anchored description of levels of change Demonstrating aggregate progress; measuring concepts that are not easily quantified

Results mapping: outcome-based evaluation tool designed to systematically capture otherwise nonquantifiable anecdotal evidence

Key Concepts Strong Points Uses
Causal and synchronistic attribution; levels and milestones Way to systemize, standardize, gather, and use lessons embedded in anecdotal information Turning anecdotal information into a useful tool for program presentation, evaluation, and assessment

Program results story: uses stories to capture organizations’ achievements and present them in a results-based format

Key Concepts Strong Points Uses
Results, stories, anecdotal evidence Easily understood approach for presenting results; brings outcomes to human-interest level; captures and conveys richness of information Presenting program and results to multiple audiences
 

[1] Penna, R. Elena, & Phillips, W.. (2005). Outcome Models. The Evaluation Exchange: Evaluation Methodology: Promising Practices. Newsletter, Volume XI, 2, p.5. Retrieved from http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluation-methodology/eight-outcome-models

[2] Preskill, H., & Gopal, S. (2014). Evaluating complexity: Propositions for improving practice. Boston: FSG.