Overarching Components
Conduct a Summative Evaluation
To determine the effectiveness of a program after implementation, it is useful to follow a framework initially developed by Donald L. Kirkpatrick and revised by Jack J. Phillips. The framework outlines five levels of evaluation, each of which builds on information from the previous level:
Level 1: Reaction and Planned Action
Reaction is essentially what the target population thinks of the program. Planned Action is what the target population plans to do with the new learning. Level 1 evaluations are generally conducted using surveys, interviews, course evaluation forms, and feedback forms (Kirkpatrick, 1994; National Highway Institute, 2000). While Level 1 evaluations are important for the summative evaluation, they are also often used in formative evaluations for continuous improvement of the program.
Level 2: Learning
Level 2 evaluations involve measuring how much was learned between the start of the program and the end of the program. This typically involves pre-tests and post-tests. Pre-tests may be part of a program’s initial assessment process (Kirkpatrick, 1994), and they may be part of the demographic data collected prior to the start of the program. Post-tests are typically the same as the pre-test in order to measure the learning gains. Post-test results help to determine whether a program participant is ready to move on to further education and training or to higher level employment (See the description of the Work Readiness Credential as an example of an innovation based on a standards-based post-test.)
Level 3: Application
While the first two evaluation levels were concerned with how the training affected the individual, the next level goes beyond the individual to measure the extent to which the skill gains result in improved performance or behavior in the workplace. The question asked is whether the training participant was able to apply their new knowledge in a work setting (similarly, a Level 3 evaluation for a non-vocational ESL program might measure the extent to which the student could use English in a real-life setting.) (Kirkpatrick, 1994).
Level 4: Impact
Level 4 measures whether a workforce training program had an impact on business performance. Typical objective measures include cost savings, output increases, turnover decrease, time savings or quality improvements. Subjective measures might include customer satisfaction, employee satisfaction, customer retention, response time to customers, reduction in re-work and so on. This can be a very important evaluation level for employers, but it is also – with Level 5 – the most difficult to carry out because it can be extraordinarily difficult to isolate the effects of the training alone on certain business metrics. Employers often implement more than one improvement strategy at a time (Winfrey, 1999; Kirkpatrick, 1994).
Level 5: Return on Investment (ROI)
The final level is the major contribution from Jack Phillips. It attempts to capture the true monetary value of the program by comparing the monetary benefits with the actual costs of administering the program. This, too, can be very difficult to administer, but it is critically important for showing employers the strategic, bottom-line importance of training and workforce development (Phillips, 2003).
With each successive evaluation level, the information gained is of greater value, but it also becomes more difficult to obtain. A Level 1 evaluation might only use a course evaluation form, while Level 5 might require a number of different data tracking systems to be in place prior to, during, and after training as well as the ability to isolate the training impact from other system improvements or changes in HR practices.
Although Level 4 or 5 evaluations may seem out of reach because of their complexity and cost, they have been successfully carried out by workforce development organizations. Some, such as San Francisco Works and partners of the Aspen Institute, have measured the impact of their training programs on business clients (Level 4); others, such as Valley Initiative for Development and Advancement (VIDA) and Twin Cities Rise!, have measured the ROI for local governments who have invested in training (Level 5):
- San Francisco Works conducted a Level 4 analysis of five employers and found that the employers gained several benefits from training, including an expanded applicant pool, reduced training costs, increased tax credits, increased productivity, higher employee retention, and enhanced community and brand reputation (Marquardt & Feeley, 2005).
- The Aspen Institute recently reported that in healthcare, one workforce development organization showed through data that patients were more satisfied by aides it trained than by aides of other providers, while another program’s entry-level placements were about as likely or more likely than traditional hires to demonstrate soft skills which results in time and cost savings for the employer. Similarly, workforce development organizations in manufacturing showed that their programs reaped the following benefits: higher employee retention rates, greater supervisor ratings on job performance, and improvements in scrap, on-time delivery and efficiency rates (Aspen Institute, 2005).
- Recent analysis from VIDA shows that the city of McAllen’s investment of $2.4 million in employment training has resulted in more than $5 million in benefits to the city (increased sales tax, property tax and other fees), for a return on their investment of 211% (Valley Initiative for Development and Advancement, 2005).
- Twin Cities RISE! receives support from the State of Minnesota through a pay-for-performance contract. Through an analysis of benefits to the state – including savings as a result of a reduction in public assistance, a reduction in incarceration and sentencing costs, and an increase in tax revenue to the State of Minnesota as a result of higher incomes – Twin Cities RISE! has calculated that the State has already received a 20% return on its investment, with the return potentially reaching 300% or more as graduates continue to work and pay taxes.