OVERVIEW OF OUTCOME-BASED PROJECT EVALUATION - OBE
- Why OBE?
- Essentials of OBE
- OBE Definitions
- OBE FAQ's
- Measuring Program/Project Outcomes for Influencers (Stakeholders)
- Program/Project Purpose
- Logic Model for Measuring Project Outcomes
- OBE Planning Forms
The Government Performance and Results Act of 1993 (GPRA) requires federal agencies to focus on planning and managing for results. Federal programs must assess the impact investment of federal tax dollars has on the lives of people. This requires partnerships with states, local governments and other federal agencies to produce measurable results that matter to citizens. The Institute of Museum and Library Services (IMLS) is challenged to be accountable for outcomes under the Library Services and Technology Act (LSTA) of 1996. Since states receive 91.5% of LSTA funding, it is important to obtain credible results from the very diverse library programs being implemented in all the states.
Outcome-based (program/project) evaluation has been adopted by the United Way organization and by other agencies to change the way grants are administered. The premise is to require grant applicants to develop more critical thinking and planning with their grant projects. They must determine in advance what the results of a federally funded grant project will be and how they will know that they will have achieved success. There is less emphasis on describing the project in terms of strategies and activities, although these are not ignored.
Inputs (money, staff time, equipment, or supplies) support activities ((training, installing and supporting networks, equipment, purchasing databases). Activities result in outputs (number of classes held or people trained, number of computers available, or hits on the databases). The next step beyond outputs is to identify the results or outcomes (increased skills, knowledge, changes in attitudes, behaviors, status or life condition) on the project participants.
ESSENTIALS OF OUTCOME-BASED EVALUATION:
- OBE is a systematic way to assess the extent to which a program or project has achieved its intended results. It focuses on key questions:
"What difference did the project make?"
"How much better are the lives of the participants because of the project?" - What OBE can achieve
Increase the program/project effectiveness
Communicate program/project value & benefits
Provide a logical framework for program/project development
Enhance the planning process
Generate information for decision-making
Satisfy funder requirements - What OBE cannot do
Pass for formal research
Although OBE suggests cause and effect, it doesn't prove it
OBE demonstrates contribution, not attribution - What is a program/project?
Activities and services generating intended outcomes
Usually has a definite beginning and end
Designed to change attitudes, behaviors, knowledge, or increase skills and abilities based on assumed need - Assumptions:
Programs/projects developed based on assumptions about people's needs
Assumptions based on personal experiences, formal, or informal research (surveys, observations, needs assessments)
Assumption: A need of a group of people exists based on their common characteristics - Solution: A program/project will change or improve participants behaviors, knowledge, skills, attitudes, life condition or status.
- Desired results: The change or improvement you expect to occur.
- Influencers (stakeholders): Individuals, agencies, funding sources, competition, community groups and professional affiliations that influence
Type and nature of services provided
Who is served
Desired outcomes
How results are communicated
Examples: Target audience, organization mission, administration and board, community, media, funders, program partners.
FREQUENTLY ASKED OBE QUESTIONS (adapted from the IMLS FAQīs statement)
What is outcome-based evaluation (OBE)?
Outcome-based evaluation, sometimes described as outcomes measurement, is a systematic way to determine if a program has achieved its goals. The organized process of developing an outcome-based program and an evaluation plan (logic model) can help libraries specify program benefits (outcomes), identify ways to measure those program benefits (indicators), clarify the specific individuals or groups for which the programīs benefits are intended (target audience), and design program services to reach that audience and achieve the desired results.
What is an "outcome and how do you evaluate (measure) them?
An outcome is a benefit that occurs to participants of a program; when the benefit to many individuals are viewed together, they illustrate the programīs impact. Outcomes generally represent an achievement or a change in behavior, skills, knowledge, attitude, status or life condition of participants in a program. In OBE, an outcome always focuses on what participants will say, think, know, feel, or be - not on mechanisms or processes which programs use to create their hoped-for results. Well-designed programs usually choose outcomes that participants would recognized as benefits to themselves. To simplify planning for evaluation, state the outcome you want to produce in simple, concrete, active terms.
What is the difference between outputs and outcomes?
Outputs are measures of the volume of a programīs activity: products created or delivered, people served, activities and services carried out. Outputs are the "things" component of evaluation. Outputs are almost always numbers: the number of loans, the number of materials digitized, the number of workshops held. Outcomes are the "people" or the "so what" component, the what happened because of the outputs.
| Outputs | Outcomes |
|---|---|
| 42 staff members will complete training | Library staff will provide faster, more accurate, and more complete answers to reference questions. |
| 37 libraries will participate in reference training | Customers will report high satisfaction with reference service |
| 4 automated ILL workshops will be held | Participants will know how to transact ILL requests over the Web |
How do you choose outcomes for your program?
First, carefully think out and describe the purpose of the program. A program (or project) is developed for a reason. It assumes a need by certain people for specific kinds of information, or services.
Next, decide why you are proposing this program (or project). What do you want to accomplish and who will benefit from using or participating in this program. Think about what the results of a successful program will look like for the people you are serving. Of equal importance is knowing your audience, needs and wants, and what your program can do to help them achieve their aims.
The answers to those questions should allow you to describe the changes or impact that you want to see as a result of the program. Those hoped-for changes become the intended program outcomes.
What is an Indicator?
Indicators are the specific, observable, and measurable characteristics, actions, or conditions that reveals whether a desired achievement or change has occurred because of a program. To measure outcomes accurately, indicators must be concrete, well-defined, and observable; usually they are also countable.
| Poor Indicators | Better Indicators |
|---|---|
| The number and percent of students who know how to use the Web | The number and percent of participating students who can bring up an Internet search engine, enter a topic in the search function and bring up one example of the information being sought within 15 minutes. |
| Patrons will report high satisfaction with the automated ILL service | The number and percent of patrons who report they are "satisfied" or "very satisfied" with the automated ILL service and who use the service at least once a month for six months. |
| Users will make healthier choices | The number and percent of users who report they made one or more lifestyle changes from a list of 10 key lifestyle health factors in the last six months |
It is easy to construct a good indicator if you use the format: Number and/or percent of a specific target population who report, demonstrate, exhibit an attitude, skill, knowledge, behavior, status or life condition in a specified quantity in a specified time-frame and/or circumstance.
- Number and percent: Both number and percent are usually specified to provide adequate information. If only two people participate in your program, after all, reporting that 50% of them benefited could be misleading. Examples: 30% of 150, 75% of 25, 10% of 1,500.
- Target audience: The group of people the program hopes to affect. Effective programs keep the characteristics of the people they want to benefit clearly in mind. The more narrowly and specifically the group of people who are expected to participate in a program can be described, the greater likelihood that a program will be designed to actually reach them. Examples (low to high definition): Salt Lake County residents; Murray high-school students; Salt Lake City adult immigrants at literacy level 1 or below.
- Report, demonstrate, exhibit: Note that these are active, observable behaviors or characteristics that donīt depend on guesswork or interpretation.
- Attitude What someone feels or thinks about something; e.g., to like, to be satisfied, to value . . .
- Skill What someone can do; e.g., log on to a computer, format a word processing document, conduct an Internet search . . .
- Knowledge What someone knows; e.g., the symptoms of diabetes, the state counties, how to use a dictionary . . .
- Behavior How someone acts; e.g., listens to others in a group, reads to children, votes . .
- Status Someoneīs social or professional condition; e.g., registered voter, high-school graduate,
employed . . . - Life condition Someoneīs physical condition; e.g., ex-smoker, weight loss, cancer-free . . .
- Specified quantity and specified time-frame or circumstance: This is the measurable part of an indicator. It asks the project director to choose a quantity of achievement or change that is enough to show the desired result happened, and the circumstances or time-frame in which the result will be demonstrated. Examples: three times per week, in 15 minutes or less, 6 months after the program ends, 4 or higher on a 5 point scale.
What kinds of programs/projects are best suited for OBE?
Most programs can incorporate OBE as an effective management tool. But specially, OBE is geared toward measuring the impact of a program on a specific group of people regarded as the target audience. Any program that proposes to educate or train participants (to change or build attitudes, skills, knowledge, behavior, status or life condition) can be designed with outcomes and evaluated using OBE concepts.
How many program participants need to be evaluated, all or a sample?
For many programs it is possible to evaluate the impact to all participants. Others will have access to only a sample of participants. Such is true for programs providing digital resources - collections, exhibits, curriculum tools, or Web sites. Many programs will seek volunteers to answer questionnaires or fill out surveys or to participate in focus groups to provide outcome information.
What if the outcomes are small or appear insignificant?
For IMLS it is less about small or large outcomes than what you hoped to achieve for an audience, what you learned in the process and what was reasonable to expect for that audience. In some cases a 10% improvement is very significant, while in others, a 90% impact is reasonable to expect. When that is done, and outcomes still fall short of goals, OBE allows libraries to assess, explain, and learn from why outcomes fell short of goals. Without OBE, it can appear as if a program just didnīt do what it said it would. With OBE, you have the opportunity to learn why and make improvements for the next offering, or the next program.
Finally, the "size" of the outcome is proportionate to the size of the target audience and the duration or the intensity of their experience in a program. If a project works closely with a small number of participants, the outcome might look small, but might be profound for those participants. If a project offers a rapid service to a very large number of participants or users, the outcome is likely to be minor, but may reach many people.
Many program/project proposals make idealized claims for anticipated contributions without offering any concrete information about how project directors will know if their intentions were realized. It is increasingly important to resource allocators (grant funders) and policy makers that programs or projects have concrete audience benefits, with services designed to achieve them for a clearly-defined audience, and that librarians demonstrate that the benefits were achieved.
What does OBE cost?
For small programs/project or Mini Grants, the cost for OBE can be minimal involving the project directorīs time to ascertain from questionnaires, surveys, interviews or perhaps focus groups what the impact was for the target audience. The important thing is for this assessment to be completed and interpreted and the results submitted in the Final Summary & Evaluation Report. LSTA funds may be used for evaluation purposes.
For Regular and Major Grants, LSTA funds may be used for evaluation purposes. Costs should be of a reasonable amount to get the job done and be included in the budget.
If OBE is not the same as academic research, and the results may not be reliable evidence of outcomes, then why should you do it?
Formal research is one way of capturing information, not the only way. OBE is a strong, effective and reliable management tool that provides your library with information on the degree to which a program did what it intended to do. While it does not allow you to determine and claim unique or complete credit for an outcome, it does allow you to demonstrate the degree to which a program contributed to the outcome for individuals. If you have no information, you cannot claim credibly claim any contribution to impact.
With OBE you look to see if your program achieved its intended results. That information will help you make decisions - whether to continue the program, expand it, improve it, or replace it with another. OBE looks at a programīs participants for logical, credible evidence that a limited number of very specific, observable attributes or phenomena happened in relatively close proximity to an experience or service designed to produce them for the target audience.
OBE doesnīt usually look for signs that participants have more or better of what itīs evaluating than non-participants. It is not intended to prove that one program did something more effectively than another.
If a project intends to show unique attribution, to demonstrate the relative worth of one approach measured against others, or to provide a tool for use by other libraries or organizations, then it requires the tools and criteria of research. Since the use of the data provided by OBE is limited, we can usually be satisfied if the information is accurate, without requiring statistical vigor, blind or random sampling, or other characteristics of research
How many outcomes should a program have?
A program needs to have at least one outcome, however, programs may have more than one outcome. It is important to consider what the purpose of the program is and the ways participants are expected to benefit from it. These benefits will likely be the outcomes for your program; you need not measure everything. Prioritize this list and determine what you and your programīs stakeholders need to know about the programīs impact.
What is a logic model and is it necessary?
A logic model is a step-by-step approach for defining and measuring outcomes. It is your programīs evaluation plan. It shows how you will measure outcomes, what information you need to collect, who you will collect information about, when you will get the information and what targets you have chosen for the outcomes.
A logic model is essential to the success of your programīs outcome-based evaluation. Without this, outcome-based evaluation will not become a reality for your library
How complicated is outcome based evaluation?
Once the concepts are understood and you have implemented them a few times, it is a very simple process to understand and manage. The key to success is the commitment of the project director and the clear identification of the roles in managing OBE.
How much time will it take?
It isnīt possible to prescribe a time for all programs. It does take a commitment of time and resources to get it done. The majority of time comes at the front end. Once incorporated, OBE can save significant time in planning and management by identifying the right questions and answers early on in the planning process.
What are the benefits of outcome-based evaluation for your library?
First, it can help your library tell its story in ways your stakeholders and the general public can understand and appreciate. It helps convey important information about the collective impact on program participants, while maintaining the ability to communicate the very powerful and personal stories that reveals how important the program was to specific individuals.
Second, it can help better position the library to request and receive funding because intended benefits and impact of a proposed program can be described in very specific terms by identifying what the program will do for participants.
Third, when OBE is incorporated into the libraryīs management routine, its programs can be improved as a result. Program goals are well planned and established, and these goals are regularly reviewed. Stakeholders are informed about the impact of funded programs. In turn, outcome-based evaluation will help the libraryīs program staff better communicate the benefits they intend to deliver to program participants - it can aid recruitment and marketing.
Arenīt some things difficult to measure?
Some things will seem more difficult to measure (evaluate) than others, and not all things programs accomplish need to be measured. It is often more straightforward to measure "hard" impact, such as knowledge, behavior, and skills than it is to measure "soft" impact such as attitudes. Measuring attitude changes or other "soft" impacts is not actually more difficult, but it may require more creativity. Regardless, clarifying the relationship between an outcome and measurable and observable "indicators" is key to success.
How will you know if your outcomes are good enough?
Outcomes are effective if they 1) are closely associated with the purpose of a program and describe what the library wants to make happen for people, 2) are realistic and within the scope of what the program can affect, and 3) have indicators that allow them to be measured.
How do you report outcome based evaluation information?
Consider what your programīs stakeholders want to know about the results of your program when developing reports from outcome-based evaluation data. The library board, community and funders may want similar information, but this does not mean that one report will satisfy everyone. In general consider the following as desirable information for reports:
- Needs identified
- Outputs ( what we produced)
- Inputs (what we used)
- Outcomes (what impact we acheived and how we know
- Activities and Services
- Interpretation (what it all means, why it matters)
- Audience (characteristics and participation)
- Where do we go from here?
Do you have to do this?
IMLS does not currently require its grantees to conduct outcome-based evaluation, but is supports and encourages it as a valuable management tool. At the same time, IMLS is required to report to Congress in outcome-based terms and cannot do so without input from libraries. IMLS considers consistent use of outcome-based evaluation to be an effective and efficient way for all programs to capture critically important information and to tell their story persuasively. It is considering the benefit of making outcome-based evaluation for funded programs a requirement at some future time.
MEASURING PROGRAM/PROJECT OUTCOMES FOR INFLUENCERS (STAKEHOLDERS)
| Influencer (Stakeholder) | What they want to know | How the results will be used |
|---|---|---|
| Library Board | Does the project meet the patrons' needs? |
|
| Friends of the Library | Who does the program/project serve? |
|
| Literacy Project Partners |
Are the program/project responsibilities equal?
Which services produce outcomes? |
|
PROGRAM/PROJECT PURPOSE: (Driven by assumptions about need; relates to libraryīs mission statement; defines audience, services, & outcomes)
| We Want to do What? | Summary of key proposed solutions |
| For Whom? | Target audience |
| For What Outcome or Benefit? | Project benefits in terms of changed, improved or demonstrated skills, knowledge, behaviors, attitudes, status of life condition. |
LOGIC MODEL FOR MEASURING PROGRAM OUTCOMES
| Outcome | Indicator | Data Source | Applied to | Data Interval | Target (Goal) |
|---|---|---|---|---|---|
| Definitions: | |||||
| Intended Impact | Observable and measurable behaviors and conditions | Sources of information about conditions being measured | The specific group within an audience to be measured | When data will be collected | The amount of impact desired |
| Examples: | |||||
| Students will have basic Internet skills | The # and % of participating students who can activate an Internet search engine, enter the topic in the search function, and bring up one example of the information being sought within 15 minutes | Searching exercise, trainer observation | Salt Lake City 7th-8th graders who complete the workshop | At end of workshop | 85% of approximately 125 participants |