JVTE v15n2 - The Modified Delphi Technique - A Rotational Modification

The Modified Delphi Technique - A Rotational Modification

Rodney L. Custer
Illinois State University

Joseph A. Scarcella
California State University, San Bernardino

Bob R. Stewart
University of Missouri-Columbia

---

Abstract

This study describes and illustrates a modification of the Delphi technique that was designed to extend its applicability to large competency data sets. The approach consisted of rotating sub-sets of a larger set of competencies through three sub-panels and then examining the viability of the approach. The context of the study was the identification of competencies needed to update training requirements for 21st century plastering contractors. Based on the outcomes of this study, the rotational approach represents a promising mechanism for extending the usefulness of the Delphi tool to larger data sets. The findings also indicate that the factors that erode the validity and usefulness of traditional modified Delphi studies also negatively affect the rotational approach.

An obstacle facing many educators has been identifying an appropriate research methodology for developing curriculum when teaching vocational subjects. This is especially true when a research topic is very broad and when new curricular areas are being developed. In such situations there is often a large number of competencies to be evaluated by recognized experts in the content area. A number of techniques have been used that are generally associated with the competency-based approach to curriculum development. These include such tools as occupational analysis, DACUM (Developing a Curriculum), and the Delphi technique ( Finch & Crunkilton, 1989 ; Frykland, 1970 ; Miller, 1990 ; Rothwell & Kazanas, 1992 ). These techniques span a range of approaches from observing workers on the job to conducting meetings with workers to conducting analyses based on work previously conducted on related occupations ( Blank, 1982 ). Each technique has advantages and disadvantages, especially in dealing with diverse groups of experts involving relatively large data sets.

This manuscript describes a modification developed for use with the Delphi technique, which was selected for use in a curriculum competency study. The Delphi technique was selected for use in this study due to its ability to obtain expert input from individuals who were widely dispersed geographically. This technique has been used in a number of fields for long-range planning - including education, international affairs, transportation, leisure activities and the like ( McCampbell & Stewart, 1992 ).

The Delphi survey technique was developed in the 1950s by two research scientists working at The Rand Corporation, Olaf Helmer and Norman Dalkey. They developed the procedure as a tool for forecasting future events using a series of intensive questionnaires interspersed with controlled-opinion feedback ( Dalkey & Helmer,1993 ; McCampbell & Stewart, 1992 ; Weaver, 1971 ). Participants were solicited experts in the issues related to national defense such as forecasting probable bombing targets the Russian government might choose in the event of an attack on the United States ( Dalkey & Helmer, 1963 ).

The Delphi begins with an open-ended questionnaire that is given to a panel of selected experts to solicit specific information about a subject or content area. In subsequent rounds of the procedure, participants rate the relative importance of individual items and also make changes to the phrasing or substance of the items. Through a series of rounds (typically three) the process is designed to yield consensus.

The modified Delphi technique is similar to the full Delphi in terms of procedure (i.e., a series of rounds with selected experts) and intent (i.e., to predict future events and to arrive at consensus). The major modification consists of beginning the process with a set of carefully selected items. These pre-selected items may be drawn from various sources including related competency profiles, synthesized reviews of the literature, and interviews with selected content experts. The primary advantages of this modification to the Delphi is that it (a) typically improves the initial round response rate, and (b) provides a solid grounding in previously developed work.

Additional advantages related to the use of the modified Delphi technique include reducing the effects of bias due to group interaction, assuring anonymity, and providing controlled feedback to participants ( Dalkey 1972a, 1972b, & Judd 1972 ). Brooks (1979) noted that three mailings are usually sufficient in order to arrive at consensus.

Additional Modifications to the Delphi Technique

One significant problem that is often encountered in Delphi studies has to do with the rigors involved in maintaining focus when rating competency sets containing large numbers of items. Beyond problems with maintaining sufficient levels of concentration, large competency sets can consume large blocks of time and thus represent a common source of panel attrition. Many Delphi studies contain as many as 50 items; however, some studies contain considerably more than that amount.

This article reports on a research study that employed an additional modification to the modified Delphi technique. This was termed a "Rotational" Delphi technique and was specifically designed to explore the viability of a procedure for rotating subsets of larger competency sets through sub-panels in order to reduce the level of fatigue on panelists and to increase the volume of competencies that can be effectively and efficiently studied using the Delphi procedure.

Context and Purpose of the Study

The research study that used the rotational technique was conducted to identify competencies needed to update training requirements for 21st century plastering contractors (i.e., lath, plaster, drywall, metal-stud framing, fire proofing insulation, exterior insulation finish systems, specialty, and decorative & ornamental plastering). Due to changes in the construction industry that have occurred over the last two decades as a result of new technology and management practices, a competency analysis was needed to develop a foundation for curriculum and training program development for use in union and nonunion sectors ( Scarcella, 1997 ).

The purpose of this article is to describe and illustrate the viability of using the rotational Delphi procedure through an actual study of competencies needed for 21 ^st century plastering contractors. The specific question that provided focus for the study was, Does the rotational modification to the Delphi procedure represent a valid and efficient means of identifying and validating large competency sets in occupationally-specific areas?

Procedures and Methodology

Sample

A sample of 22 contractors and 22 policy makers from selected contractor associations and organizations was asked to verify and rate the relative importance of the set of identified competency items. The research goal was to obtain consensus regarding what competencies are important for plastering construction contractors. A descriptive research design was utilized to achieve this objective because the panel members under investigation were attempting to describe what knowledge, skills, and abilities are associated with plastering contractors ( Crowl, 1993 ).

It was determined that contractors and policy makers from different regions of the country, as well as industry experience and perspective would provide some breadth of insight into certification requirements for plastering contractors. The rationale for using the two different types of panelists to react to the common competencies list was that, although both groups come from different parts of the industry, they represent unique perspectives of expert opinion. Both perspectives are important as certification, licensing, and curriculum programs are developed to educate the workforce.

Three rounds of the modified Rotational Delphi technique were conducted. Each round presented an instrument to panel members who completed and returned it to the researcher. The responses were analyzed and compiled to build the next round's instrument. For each item, interquartile ranges were calculated as measures of dispersion and median scores were calculated as measures of central tendency. The combination of these indices was used to determine the degree of importance and consensus for each item.

Instrumentation

The Plastering Contractors Competency Rating Scale (PCCRS) was developed for use in the study. The procedures for developing the instrument began with identifying and collecting documents pertinent to the certification requirements for plastering contractors. This included sources such as licensure examinations, textbooks, apprenticeship training materials, and job descriptions. Based on a table of specifications developed from the review of the literature, 75 competencies were selected and divided into two major classifications: 48 having to do managerial functions (e.g., accounting, hiring, personnel, training, etc.) and 27 concentrating in technical areas (e.g., skills associated with the application of materials, use of tools, etc.). The instrument was subjected to scrutiny by a panel of experts and was pilot tested by a sample of plastering contractors and policy makers. Items were rated on a scale of 1=Important to 6=Not Important.

After the instrument had been validated and pilot tested, it was subdivided into three forms. Each form of the instrument contained two-thirds of the entire competency set. This was done by systematically eliminating every third item as illustrated in Table 1. This procedure provided a mechanism for exposing panelists to every item in the total competency set at least once during the first two rounds of the modified Delphi process. An alternative method of developing sub-forms of the instrument would have been to have divided the competency set into three forms with each form containing unique items (i.e., one-third of the total set with no duplication of items across forms). The rationale for using the first approach (i.e., two-thirds of the total set on each form of the instrument) was to reduce the total number of items rated during each round while maintaining some continuity of ratings across the entire panel.

Table 1

Instrument Master Data Rotation For Rounds One and Two

Master Form
Item Number	Form One	Form Two	Form Three
Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7	x x . x x . x	. x x . x x .	x . x x . x x

The Rotational Process

The total panel was randomly subdivided into three sub-panels on a stratified basis (e.g., equal numbers of contractors and policy makers in each of the three groups). The three forms of the instrument were then "rotated" through the subpanels on a systematic basis during rounds one and two (see Figure 1). The round three instrument was comprised of all items that had not reached consensus by the end of the second round. This involved 13 (27%) of the managerial items and 19 (70%) of the technical items (i.e., 43% of the total competency set).

Round One Round Two Round Three F1 F2 F3 F3 F1 F2 C n n n n n n n G1 G2 G3 G1 G2 G3 CP ¬ ¬ ¬ ¬ ¬ ¬ ¬

Figure 1 . Illustration of the rotational process.

Note: F = 3 forms of 1 instrument

G = Groups of panelists

C = Composite forms of instrument comprised of all non-consensus items

CP = Combined panel

Data Gathering Procedures

During the first two rounds, each of the three sub-panels received a form of the instrument containing 50 items along with a cover letter that included an explanation of the study and the six point rating scale. In addition to rating each of the items, raters were asked to make additional suggestions related to (a) the phrasing of items or (b) ideas about any new items that they believed should be added. The only difference between the first and second round procedure was that interquartile ranges of the first round ratings were placed on each of the second round items.

If any of the panelists second round ratings fell outside of the interquartile range (indicated in brackets), they were asked to provide a brief explanation of their rating in the comments section that was located on the back side of each page. If panel members agreed with the majority rating, no additional commentary was required. Panelists were also requested to make written comments or reword those items, which lacked clarity as part of the feedback on the second round instruments (if applicable). In round three, interquartile ranges were placed in brackets directly on the instrument along with any comments obtained during round two.

Data Analysis Procedures

The data analysis process included three primary components. The first component (and primary consideration of the Delphi process) consisted of analyzing each item for consensus. For this study, consensus was considered to have been achieved when an Interquartile Range score of less than 1.2 was obtained ( Zeliff & Heldenbrand, 1993 ). A second component of the analysis was to evaluate the perceived importance of the items. To accomplish this, the six-point scale was evenly divided into high, medium, and low importance. Items were then classified into one of six categories based on an analysis of consensus combined with importance.

The third component of the analysis process consisted of examining the effectiveness of the rotational modification to the Delphi process. This process consisted of comparing the interquartile range scores of first and second time raters on each item at the end of round two. As noted previously, during round two, one-half of the items that each group received were being rated by that group for the second time. The other half of the items was new to each group (rotated onto the group from the first round).

The analysis process was designed to compare the second round ratings of "first" and "second" time raters. The rationale for this procedure was that strong similarities between the two groups could suggest that little consensus-building information was lost when all competencies were not rated by all panelists during the initial round. Strong dissimilarities, on the other hand, would suggest that the rotational process had not worked effectively.

Findings

The analysis procedure consisted of examining the differences in interquartile ratings for each item. Separate frequency and percentage scores were compiled for managerial skills (see Table 2) and technical skills (see Table 3). For managerial skills, there was no difference between first and second time raters on 50% of the items. An interquartile range difference greater than one was detected on only one item. It is important to note that there was strong consensus across the panel on the managerial skill set. By the end of round three, raters for 95.8% of the managerial skill set had achieved consensus. Therefore, it is likely that the low interquartile rating differences between first and second time raters was to some extent a function of the strong overall consensus on the managerial item set.

Table 2

Difference in Managerial Skills Competency Items/Interquartile Ratings Between First and Second Time Raters

Interquartile Differences	Frequency	%
.0 .25 .50 .75 1.00 1.25 1.50 1.75 2.00	23 5 0 3 15 0 0 0 1	48.9 11.6 0.0 6.6 31.9 0.0 0.0 0.0 2.8
N =	47	100.0

Note: 36 out of 46 (78.2%) first time rater's managerial skills interquartile ranges were equal or lower than second time raters.

24 out of 46 (52.1%) second time rater's managerial skills interquartile ranges were equal or higher than first time raters.

The differences between the interquartile ratings of the technical skill set items were somewhat greater. Seventy-two percent of the items were within an interquartile range of one while the difference between first and second time raters was greater than one for seven items in the set (see Table 3). The degree of consensus achieved by raters for the technical skill set at the end of round three was less (44.4%) than with the managerial skills. Thus, it could be expected that this would translate into greater interquartile differences between first and second raters than was the case with the managerial skill comparisons.

Table 3

Difference in Technical Skills Competency Items/Interquartile Ratings Between First and Second Time Raters

Interquartile Differences	Frequency	%
.0 .25 .50 .75 1.00 1.25 1.50 1.75 2.00	4 4 1 2 7 1 1 0 5	16.0 16.0 4.0 8.0 28.0 4.0 4.0 0.0 20.0
N =	25	100.00

Note: 15 out of 25 (60.0%) first time rater's technical skills interquartile ranges were equal or lower than second time raters.

14 out of 25 (56.0%) second time rater's technical skills interquartile ranges were equal or higher than first time raters.

The relationship between the extent of total group consensus and first/second round rater comparisons requires additional comment, particularly as it relates to interpreting the results from the technical skills component of this analysis. Whereas high consensus corresponded closely with high first and second round comparisons with the managerial skills, a relatively low level of consensus yielded a relatively high degree of first and second round correspondence for the technical skills. This pattern of relationships for the technical skills makes sense, given that (a) high interquartile range scores indicates a lack of consensus, and (b) a relatively high percentage of raters agreed with the relative lack of consensus. The correspondence scores (72% within one interquartile) between first and second round raters is relatively high given the relatively low degree of consensus. In other words, the first and second time raters tended to agree on a relative lack of consensus.

Discussion

This study was designed to explore the feasibility of using a rotational modification to the Delphi process in order to expand the total number of competencies that can be analyzed effectively. The analysis of the rotational modification was conducted as part of a study of competencies needed to update training requirements for 21st century plastering contractors. Based on the findings of the study and an analysis of the rotational process, the following comments and observations can be reported.

1. The validity of the modified Delphi process depends on the careful and systematic application of procedures for initial competency selection (e.g., reviewing the literature, developing a table of specifications, conducting a pilot test, etc.). In a traditional modified Delphi, this careful selection process is necessary in order to (a) avoid biasing panelists by including inappropriate or unnecessary items and (b) increase the probability that consensus can be achieved in an efficient and timely manner. For the rotational modification, the careful initial selection is important because it increases the probability that first and second time raters would rate individual items in somewhat the same way.
2. The extent to which the Delphi process is capable of achieving consensus is a function, not only of the quality of the initial competency selection process, but also of the degree of controversy or clarity that exists in a given content area or profession. In this study of the plastering industry, the Delphi provided a mechanism for refining the understanding of the skills required to accomplish the job. Given this, the use of a rotational modification for emerging professions or controversial content areas is not recommended.
3. As a corollary to the previous point, in those situations where the Delphi is being used as a refinement tool and where the required pool of experts is readily available (as was the case in this study), the rotational modification represents a promising approach for expediting the consensus process while reducing the workload on panelists.
4. There are limitations to the rotational delphi procedure which should be acknowledged. Specifically, since all round two instruments contained brackets indicating interquartile ratings for round one, first time raters (during round two) could well have been biased by the previous first round ratings. In this study, where the items were relatively homogeneous in terms of consensus (particularly on the management set), the concern with bias is less problematic.
5. On reflection, it seems clear that the same factors that threaten the validity of a pure or modified Delphi (non-rotational) are nearly identical to those that erode the quality of a rotational Delphi; namely, (a) a lack of expertise on the panel, (b) lack of clear content definition, and (c) a poorly developed initial data set (for the modified Delphi). Stated in the reverse, we may reasonably conclude that Delphi studies that are conducted according to established criteria and procedures are candidates for the rotational modification.

In conclusion, the rotational modification to the Delphi process represents a viable alternative to the established modified Delphi as a mechanism for use with large data sets. The rotational procedure that was used in this study should be viewed as a preliminary and exploratory process that holds promise for refinement and use in other Delphi studies.

References

Blank, W. E. (1982). Handbook for developing competency-based training programs . Englewood Cliffs, NJ: Regents/Prentice Hall.

Brooks, K. W. (1979, winter). Delphi technique: Expanding applications . North Central Association Quarterly, 54 (3), 377-385.

Crowl, T. K. (1993). Fundamentals of educational research . Madison, WI: WCB Brown and Benchmark Publishers.

Dalkey, N. C. (1972). The Delphi method: an experimental application of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds.) Studies in the quality of life . Lexington, MA: Lexington Books.

Dalkey, N.. C. (1970). Occupational analysis: Techniques & procedures . New York, NY: The Bruce Publishing Co.

Judd, R. C. (1972). Forecasting to consensus gathering: Delphi grows up to college needs . College and University Business, 53 (1), 35-38, 43.

McCampbell C. , & Helmer, O. (1993). An experimental application of the Delphi method to the use of experts. Management Science, 9 (3), 458-467.

Finch, C. R., & Crunkilton, J. R. (1989). Curriculum development in vocational and technical education . Rockleigh, NH: Allyn and Bacon.

Frykland, V, W. H., & Stewart, B. R. (1992). Career ladder programs for vocational educators. Journal of Vocational Education Research, 17 (1), 53-68.

Miller, W. R. (1990 ). Instructors and their jobs . Homewood, IL: American Technical Publishers.

Rothwell, W. J., & Kazanzs, H. C. (1992 ). Mastering the instructional design process: A systematic approach . San Francisco, CA: Jossey-Bass.

Scarcella, J. A. (1997 ). Plastering competencies identified as important for 21 ^st century contractors: A rotational Delphi . Unpublished doctoral dissertation. University of Missouri-Columbia.

Weaver, W. T. (1971). The Delphi forecasting method. Phi Delta Kappan, 52 (5), 267-271.

Zeliff, N. D., & Heldenbrand, S. S. (1993). What's being done in the international business curriculum? Business Education Forum, 48 (1), 23-25.