JITE v32n4 - Under Review - Flex: A flexible tool for continuously improving your evaluation of training effectiveness

Flex: A flexible tool for continuously improving your evaluation of training effectiveness

Charles L. Sidell
Southern Illinois University at Carbondale

This book is primarily a training and development workbook for conducting training effectiveness evaluations and is directed at training practitioners rather than evaluation theorists. Evaluation processes are approached through a series of worksheets, designed along eight primary dimensions, to enable trainers to evaluate training effectiveness. Flex is designed as a cookbook and practitioners can choose those "recipes" that are needed to evaluate training programs adequately.

Author's Purpose and Objectives

Continuous improvement has become a watchword of business and industry throughout the world. Schouborg designed Flex as a comprehensive, yet flexible tool for continuously improving the evaluation of training effectiveness, intended for use when rigorous laboratory evaluation methodologies are not feasible. Flex is specifically designed to assist trainers, instructors, training directors, and human resource personnel in evaluating training programs as thoroughly, and as in as much detail, as circumstances permit; to have benchmarks for continuously improving evaluation processes; and to provide a perspective against which one can assess limitations of a particular evaluation.

Schouborg designed a system of evaluation instruments around three constraints of evaluation:

1. Theoretical constraints: problems encountered even if one has all the resources and time to conduct an evaluation.
2. Cost constraints: when the cost of the evaluation is disproportionate to the possible benefits of the training.
3. Time constraints: when the availability of time impacts the validity and reliability of the evaluation.

Structure and content

This book is divided into eight primary sections and six appendices, flanked by an introduction and references. The eight sections, called "dimensions" by Schouborg , deal with evaluation from different perspectives. Each primary dimension contains three or more sub-dimensions. The primary dimensions are (a) business results, (b) business planning, (c) transfer, (d) learning, (e) delivery, (f) design, (g) reaction, and (h) vendor relations. According to the author, these sections are presented in order of value to decision makers and are designed to assist trainers' and supervisors' decisions to adopt, improve, maintain, or eliminate training programs. To assist users, each primary section contains a short introduction that describes the purpose of the section, its relative position in Schouborg's hierarchy of evaluation worksheets, required expertise, and a methodology for conducting the evaluation. The worksheets are part of a hierarchical system, in which a generalized overview sheet is supported by information presented on specific appraisal sheets. Each dimension contains two parts: (a) major elements of the dimension to be evaluated, and (b) implications of the dimension as it relates to evaluation of business results. For example, the Transfer dimension has two major sub-dimensions: Facilitation of Transfer and Performance Change . The sub-dimension Facilitation of Transfer is further divided into three evaluation areas: Before Training, During Training, and After Training.

The first two dimensions, Business Results and Business Planning , are comprised of identical summary sheets containing benefit to cost ratio worksheets. They differ only in their focus. Business Results consists of ongoing and back-end evaluations, while Business Planning adds front-end evaluations to the ongoing and back-end evaluations. These dimensions are designed to meet the needs of financial decision makers. The third dimension, Transfer , deals with the transfer of training from a place of learning to the workplace. While most evaluation publications deal only with evaluation of the training program itself, Schouborg specifically included these three dimensions because he felt that trainers must understand and be able to address financial issues and implications of training programs on workplace performance.

The five remaining dimensions-- Learning, Delivery, Design, Reaction, and Vendor Relations --provide the data collection instruments needed for training program evaluation. Learning is used to measure the extent to which learning is a product of training, while Delivery addresses the issues surrounding the quality of the program content. The Design dimension evaluates the effectiveness of program design, and Reaction allows for student input about the program in question. The final dimension, Vendor Relations , concerns evaluation of training provided by an external source.

Strengths and Weaknesses

A major strength of this manual is its relative ease of use and simple layout. The many worksheets and examples make the content easy to understand. Another strength is its treatment of evaluation designs and methods. Schouborg provides several evaluation models and identifies the advantages and disadvantages of each. He explains the theoretical and practical aims of evaluation and provides a good discussion of subjective versus objective evaluation. The author also provides a detailed strategy guide for selecting evaluation strategies by outlining the comparative strengths and weaknesses of each approach.

Another major strength is the inclusion of financial considerations, which are often overlooked by books of this type. Schouborg delves deeply into cost-benefit evaluation. He appreciates the bottom line issues that frequently affect training decisions. Finally, the glossaries, while repetitious, prove to be extremely valuable. Excellent guidance and examples are provided in the appendices.

Less successful are the author's references to traditional management systems while touting Total Quality (TQ). Schouborg submits that Flex will fit into TQ organizations but makes several statements about providing information to managers so that they can make decisions. In a truly TQ organization, the people conducting evaluations would contribute heavily to decisions about training programs. This seems to constitute a fundamental flaw of the book.

Another weakness is the apparent conflict between the book design and actual application. The design of the workbook is said to assure that experience in evaluation is not necessary for utilization of the workbook. Users are guided to make judgments about applied theory during the evaluation process. Yet Schouborg states several times that graduate level understanding or training experience is necessary for valid use of a particular section. These minor drawbacks aside, Flex is well-designed and constructed, well-presented, and thoughtful. This book would be a valuable addition to any training department's store of evaluation tools.