Journal of Career and Technical Education
THE DELPHI TECHNIQUE: A RESEARCH STRATEGY FOR CAREER AND TECHNICAL EDUCATION
Wanda L. Stitt-Gohdes
University of Georgia
Tena B. Crews
University of South Carolina
ABSTRACTCareer and technical education research often centers around quantitative research designs. The Delphi Technique provides a structured communication process designed to produce a detailed examination of a topic and/or problem and discussion from the participating group. The contributions of individuals via this tool produce a group perspective not otherwise attainable. This paper provides the foundations of the Delphi Technique, discusses its strengths and weaknesses, explains the use and stages followed, discusses panel selection, and explains how consensus among participants is reached. Examples of how the Delphi Technique has been used in career and technical education are also included.
Research literature in career and technical education is replete with studies using survey research as a means of seeking answers to any number of different questions. On occasion, however, one wishes to collect opinions of experts and, to the greatest extent possible, achieve consensus among them. For example, consensus among educators in a specific area about what technologies should be taught in today's technology-based world would be important to many educators and their students. This goal presents a unique research situation.
Survey research was first used in 1790 with the first U.S. Census (CASRO, 2004). Due to fluctuations in the economy and World War II, marketing surveys became important later in history as a tool to investigate the needs of consumers. Companies and educational institutions today are still concerned about customers' and students' wants and needs, but many times are also interested in the opinions of experts in the field and not simply a market analysis.
It would be relatively simple to ask a sample of the appropriate population their opinion about topic X. But then, how does one achieve consensus or agreement about topic X? Survey research doesn't provide a vehicle through which to achieve this goal. With the rate of change in today's workplace, it is necessary to investigate future trends and needs in career and technical education. The Delphi technique affords the researcher such an opportunity.
FOUNDATIONS OF THE DELPHI TECHNIQUE
The Delphi technique had its genesis at the Rand Corporation in the early 1950s as a forecasting tool for the military. Its purpose is to solicit reliable responses from a panel of experts regarding a particular problem or dilemma. This research tool permits researchers to combine the reports or testimony of a group of experts into one, useful statement.
Adler and Ziglio (1996) describe the Delphi method as a communication process that is structured to produce a detailed examination of a topic/problem and discussion from the participating group, but not one that forces a quick compromise. Thus, the structuring of group communications is an essential component of the Delphi. The contributions of individuals via this tool produce a group perspective not otherwise obtainable.
Linstone and Turoff (1975) characterize the Delphi as a method for structuring a group communication process. This structure permits the process to be an effective way to allow a group of individuals, as a whole, to deal with a complex problem. Critical to this communication process are four factors: "some feedback of individual contributions of information and knowledge; some assessment of the group judgment or view; some opportunity for individuals to revise views; and some degree of anonymity for the individual responses" (Linstone & Turoff, 1975, p. 5). This research technique allows educators, amongst others, to communicate and effectively develop trends, needs, or other factors relative to a particular area of education. In selecting the most appropriate research tool, however, Linstone and Turoff (1975) caution the researcher to consider the circumstances surrounding the "necessarily associated group communication process" (p. 6). They suggest these guiding questions: "Who is it that should communicate about the problem, what alternative mechanisms are available for that communication, and what can we expect to obtain with these alternatives" (p. 6)? Depending on the answers to these questions, one may then choose the Delphi as the most effective research tool for the study at hand.
Typically one of the following leads to the need for using the Delphi:
- The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis.
- More individuals are needed than can effectively interact in a face-to-face exchange.
- Time and cost make frequent group meetings infeasible.
- Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured.
- The heterogeneity of the participants must be preserved to assure validity of the results, i.e., avoidance of domination by quantity or my strength of personality (Linstone & Turoff, 1975).
Linstone and Turoff (1975) have provided a mass of information about the Delphi technique. The Delphi exists in two basic forms: (1) The Paper-and-Pencil Version (Conventional Delphi), and (2) The Delphi Conference. The Conventional Delphi includes questions being sent to a group and, based on their responses, a new questionnaire is developed and disseminated to the same group. This continues through several rounds, providing the group the opportunity to reevaluate their own answers based upon examination of the group response. The Delphi Conference uses a computer program to compile and distribute the group results. The main advantage of this method is that there is no delay in summarizing feedback from each round and returning the group responses. Essential components of the Delphi technique, to be discussed in more detail later, include the communication process, a group of experts, and essential feedback.
There are four individual phases in the Delphi process. The first phase explores the subject being researched, giving participants the opportunity to contribute information they feel is appropriate. The second phase moves to determine an understanding of how the entire group views the issue. If significant disagreement is determined, the third phase is used to explore that disagreement and determine reasons for differences. The fourth phase is a final evaluation of all gathered information (Linstone & Turoff, 1975).
There are different structures within the Delphi method. Three include the Policy Delphi Model, the Trend Model (Turoff, 1970), and the Structural Model (Lendaris, 1980; Geoffrion, 1987). The Policy Delphi works toward discovering the strongest pro and con arguments about differing resolutions for a specific policy issue. It does not produce a consensus. The Trend Model first deals with a specific trend that is of concern to the group, and participants project where they believe the trend will go in the next five years. Assumptions and uncertainties are listed when making the projections. The assumptions are voted on for validity. The participants are later asked to review the valid assumptions and re-estimate their earlier trend projections. Structural Modeling allows participants individually to express independent relationships/judgments, but they are all used to produce a group or whole model or system. This is supported by Helmer (1977) who notes that the Delphi is a useful communication method among an expert panel that in turn facilitates the formation of a group judgment.
Cornish (1977) noted that the Delphi has been used widely as a way to forecast trends in technology, education, and other fields. Some studies specific to career and technical education include Wilhelm (1999) who used the Delphi technique to determine entry-level workplace skills and competencies. LaBonty and Scott (1993) used the Delphi method to investigate perceptions of business teacher educators. Bruening, et al. (2002) investigated attributes and characteristics of exemplary career and technical teacher preparation programs. Marketing education has also used the Delphi method to decipher effective instructional strategies in Tennessee marketing education (McCurry, 1996). Farmer (1998) used the Delphi technique to determine research priorities in career and technical education. Wicklein (1993) identified critical issues and problems in technology education using a modified Delphi technique.
Chandler (1996) used the Delphi as a foundation for making recommendations to the Associated Landscape Contractors of America regarding their career days program. Pullen (1996) used the Delphi to develop a profile of the duties of the secondary career and technical education supervisor predicted to be needed in the 21st century. Flanders (1988) also used the Delphi to determine curriculum content for nursery/landscape course work in vocational agriculture courses for the 21st century.
STRENGTHS AND WEAKNESSES
Stewart (2001) stated this regarding the utility of the Delphi, "Its capacity to capture those areas of collective knowledge that are held within professions but not often verbalized, makes it enormously useful in the field of professional education" (p. 922). The Delphi is exceptionally useful where the judgments of individuals are needed to "address a lack of agreement or incomplete state of knowledge . . . [thus] the Delphi is particularly valued for its ability to structure and organize group communication" (Powell, 2003, p. 376). Powell reported further that the systematic control inherent in the Delphi design increases "an air of objectivity to the outcome" (p. 377).
When the proper steps are followed in a Delphi study, including anonymity, the panel members can express their opinions freely. When the Delphi Conference is used, the participants can "meet" any time to provide their judgments/suggestions/opinions and feel as though they are a part of a group solving or discussing a particular problem. Turoff and Hiltz (1996) identify other advantages of the Delphi. These advantages include the capability of allowing individuals with the appropriate knowledge in the content area to have "differing perspectives and differing cognitive abilities" but still contribute to those parts of a complex problem that deal with their area. This provides a flexible way for individuals to approach difficult questions and solve complex problems. Turoff and Hiltz (1996) continue by stating that the most important criterion in the design of a Delphi study is enabling each individual "to choose the sequence in which to examine and contribute to the problem solving process" (p. 3).
Every type of research has its strengths and weaknesses. It is obvious that the inappropriate choice of the Delphi technique as a research tool will typically result in failure. Linstone and Turoff (1975) suggest the following as reasons for failure:
- Imposing monitor views and preconceptions of a problem upon the respondent group by overspecifying the structure of the Delphi and not allowing for the contribution of other perspectives related to the problem.
- Assuming that the Delphi can be a surrogate for all other human communications in a given situation.
- Using poor techniques of summarizing and presenting the group response and ensuring common interpretations of the evaluation scales utilized in the exercise.
- Ignoring and not exploring disagreements, so that discouraged dissenters drop out and an artificial consensus is generated.
- Underestimating the demanding nature of a Delphi and the fact that the respondents should be recognized as consultants and properly compensated for their time if the Delphi is not a part of their job function.
Another weakness may include the researcher not being able to conceptualize different ways to examine the problem being investigated. As researchers become more creative in perceiving how different individuals may view the same problem in many different ways, this becomes less of a weakness. Also, if a researcher hastily tries to complete a Delphi study, thorough time for thought may not be given to the problem and consensus may not be obtained.
Researchers must be careful to follow the steps in any Delphi study to make sure that the experts in the group have a chance to provide appropriate feedback. Time is also necessary so that the researcher appropriately restructures the questionnaire from round to round so that adequate and correct feedback is provided through the questionnaire. All of these weaknesses and limitations result in a poorly designed and executed Delphi study which will result in inadequate or incorrect data.
Linstone and Turoff (1975) further elaborate on the potential problems of Delphi studies with a checklist of eight pitfalls including: discounting the future; the prediction urge, the simplification urge, illusory expertise, sloppy execution, optimism-pessimism bias, overselling, and deception. To avoid these pitfalls, Delphi researchers must make sure that they are open-minded, not biased, and willing to put the time into the study that it requires.
USE AND STAGES OF THE DELPHI TECHNIQUE
This section provides specifics as to how to use the Delphi technique to its best advantage. Successful use of the Delphi involves a communication procedure through which information/feedback/opinion is obtained from the participants. The following summarize stages proposed by Beech (1999):
- Selection of panel (respondents) and allocation of identification numbers.
- Construction and distribution of first questionnaire (Round One). Completion and return of Round One questionnaire.
- Collation and categorization of suggestions and construction of second questionnaire (Round Two).
- Distribution of second questionnaire (Round Two). Completion and return of Round Two questionnaire.
- Collation of individual and group scores for each suggestion.
- Construction of third questionnaire (Round Three) which is similar to Round Two questionnaire but with individual and group scores for each suggestion from Round Two incorporated.
- Distribution of third questionnaire (Round Three). Completion and return of Round Three questionnaire.
- Recollation of individual and group scores for each suggestion.
- Possible further rounds of voting and possible request for rationale and comments for more extreme scores.
- Achievement of group consensus with calculation of summary statistics: maximum, minimum, and range of scores for each suggestion.
- Distribution and use of findings.
Steps may indeed vary from these suggestions; however, the procedure remains consistent. For example, step two above indicates development of a questionnaire. If the nature of the research project is to provide an answer to a question, then indeed a questionnaire would be developed (i.e., What are the primary concerns regarding classroom management of career and technical education student teachers?). However, Wilhelm (2001), stated "In some Delphis the first round is structured such that the topics are presented to the respondents in order for them to evaluate, elaborate, or otherwise comment on the topics with their individual concerns, insights, criticisms, or agreement" (p. 14). For example, round one might ask participants: Rank the following topics in order of importance in preparing future career and technical education teachers:
- Career/technical student organizations
- Philosophy of career and technical education
- Specific methods of instruction in program areas under career/technical education
- Other, please specify.
Careful selection of the panel of experts is the keystone to a successful Delphi study. Key aspects include panel selection (including experts' qualifications) and size and participant commitment. Jairath and Weinstein (1994) suggest that study participants be experts who are knowledgeable about current information and perceptions regarding the topic under investigation but are open-minded to the findings. Gibbs, Graves, and Bernas (2001) in their study of evaluation guidelines for multimedia software used these criteria to determine who should be selected and invited to participate:
- Participants had published articles in the last five years on computer-based courseware design, development, or evaluation;
- Participants had taught courses about these topics; or
- Participants' primary employment responsibilities related to these areas (p. 4).
Wicklein(1993) achieved a 100% return rate in a Delphi study used to collect information about issues and problems in technology education. This Delphi study consisted of four rounds and 25 leading experts in technology education who were carefully selected "because the success of the Delphi relies on informed opinion," not random selection (p. 1050). Recognition of being selected for a particular study in which participants are noted as experts in their field may be encouragement enough to get participants to agree to provide their opinions to complete all rounds of the study.
Thus, one might look to professional associations under the career and technical education umbrella such as members of the International Technology Education Association, the National Business Education Association, or the Association for Career and Technical Education. Powell (2003) also reported that heterogeneous rather than homogeneous groups "produce a higher proportion of high quality, highly acceptable solutions" (p. 379). To avoid homogeneity in career and technical education research, one might solicit participation from secondary educators, technical college educators, and college/university faculty.
To motivate participants to remain active and complete all the rounds in a Delphi study, it is important they understand the goal of the study and feel they are a part of the group. The participants must also believe they are able to contribute valuable judgments and help examine the problem via discussion with their peers. If the Delphi researcher can locate the expert group of peers and invite them individually to participate, the likelihood of increased participation is stronger. Turoff and Hiltz (1996) note that "blank invitations" do not provide sufficient specifics about who will participate; therefore, the potential participants do not know if it will be a group of their peers and participation is usually very low.
Anonymity is an important Delphi technique characteristic. Without the identification of those contributing information or making specific judgments, candor may be increased. Turoff and Hiltz (1996) cite these advantages to anonymity:
- If a participant provides an idea/suggestion/judgment and in the end it is deemed unsuitable, no one loses face.
- When a participant commits his/her name to an idea/suggestion/judgment, it is harder to reject/change it later.
- The idea/suggestion/judgment may be biased by who introduced it if anonymity is not used.
Keeping in mind that group communication is an essential component of the Delphi, the anonymity will remove common biases and social interaction that normally occur in faceto- face group communication.
A variety of perspectives exist regarding the number of participants needed. Many support the notion that the more the participants, the better (Murphy-Black, et al., 1998). However, Powell (2003) also stated that, "There is very little actual empirical evidence on the effect of the number of participants on the reliability or validity of consensus processes" (p. 378). Taylor-Powell (2002) stated that the number of participants in a Delphi study depends not only on the purpose of the study, but the diversity of the targeted population. Ten to 15 participants may be an adequate number for a Delphi study that is focused and where the participants do not vary a great deal.
Rounds and Determining Consensus
The Delphi "uses rounds of written questionnaires and guaranteed anonymity with summarized information and controlled feedback to produce a group consensus on an issue" (Beech, 1999, p. 283). Unlike survey research, the rounds used with the Delphi provide opportunity for initial feedback, collation of feedback, and distribution of collated feedback to participations for further review. This process requiring group communication is central to the strength of the Delphi. The Delphi method involves a set of procedures for eliciting and refining the opinions of a group (Brown, 1968; Dalkey, 1967). In rounds two and three participants are typically asked to provide a rating based on a Likert-type response (i.e., 1 is never, 5 is always, etc.). Median scores are then developed as are means and standard deviations. In rounds two or three, "it is also possible to request either a rationale or clarification from experts who score items outside a particular range, for example +/- two points of the group median score" (Beech, 1999, p. 284).
For example, if one were attempting to determine the responsibilities of a career and technical education supervisor using the Delphi technique, round one might start with questions such as,
- "What duties related to induction of new teachers, staff development activities, and teacher evaluation should be performed?" or
- "What duties relative to maintaining facilities and technology should be performed?" or
- "What duties relative to budget development and records maintenance should be performed?"
Participants would then be free to record as many responses to each question they deemed necessary. In round two, the responses to each question would be collated and/or categorized if needed. The question and responses would be developed into a second questionnaire that provided a Likert-type response for every response provided. For example, a round two question might look like this:
Duties related to the induction of new teachers, staff development activities, and teacher evaluation: 1. Establish a mentoring program for first-year teachers
and teachers new to the system.
1 2 3 4 5 2. Create staff develop activities around program area
1 2 3 4 5 3. Peruse published literature on evaluation models to
develop one appropriate for the system.
1 2 3 4 5
In round two, then, participants would be asked to rate each item using the Likert scale and key to scale numbers provided. The results, vis a vis each participant's score and the median for each item from round two would be computed. Participants would then be asked (in Round 3) to either retain their initial score or revise it up or down. An example follows.
No. Item Your
1. Establish a mentoring program for first-year teachers and teachers new to the system. 4 4 1 2 3 4 5 2. Create staff develop activities around program-area needs. 3 4 1 2 3 4 5 3. Peruse published literature on evaluation models to develop one appropriate for the system. 2 3 1 2 3 4 5
Of course participants' scores would not be revealed to the entire group, only to participant who owned the score. From this, round three feedbacks, one then is able to determine consensus.
Knowing that participants are in agreement on the questions is another key element of the Delphi. Consensus or convergence of opinion is an underlying principle of the Delphi (McCallister, 1992). Weatherman and Swenson (1974) stated that there should be a convergence of opinion as individual estimations move closer to the statistical survey of the group responses. One may decide, then, that when 60% or more of respondents give the same response on an item, consensus on that item was achieved. Also, if any item had more than 15% change in the mean score from one round to the next, the item was considered unstable and a succeeding round was needed. The median can be used to determine central tendency (Chandler, 1996).
Crews and Ray (1997) utilized a modified Delphi where a list of course topics for a telecommunications course were provided in round one and the participants could add, delete, or edit from that initial list. The participants were then provided with the new list in round two; and they could again add, delete, or edit from that initial list. Round three used the Likert scale to rate the items, and round four gave the participants an opportunity to see the group mean and change or keep their ratings on the topics. Consensus was achieved only if 100% of the participants rated an item at the highest rating in the Likert scale provided. However, if two-thirds of the participants rated the item at the highest rating, common consent was achieved. If the Delphi method is looked at as outcome-based, the outcome is an opinion. This opinion is only as valid as the opinions of those experts who serve as the group providing their judgments/suggestions/opinions (Martino, 1993).
There has to be a clear acknowledgement about when the rounds should cease. Linstone and Turoff (1975) hold the view that "when something has attained a point at which is it explicitly definable, then progress has stopped" (p. 5).
The Delphi method provides different opportunities to researchers than survey research. Essential components of the Delphi technique include the communication process, a group of experts, and essential feedback. The Delphi method has been used in a variety of ways in government, business, and education. This manuscript has presented the key principles of the Delphi technique and has provided a number of examples where it has been used in career and technical education. "The technique benefits from being a democratic and structured approach that harnesses the collective wisdom of participants" (Powell, 2003, p. 381). Its most significant strength lies in the ability to garner opinion and seek consensus among a diverse group of participants.
Adler, M., & Ziglio, E. (1996). Gazing into the Oracle: The Delphi method and its application to social policy and public health. London: Kingsley Publishers.
Beech, B. (1999). Go the extra mile—use the Delphi technique. Journal of Nursing Management (7), 281-288.
Brown, B. (1968). Delphi process: A methodology used for the elicitation of opinions of experts. Santa Monica: The RAND Corporation.
Bruening, T., Scalon, D., Hoover, T., Hodes, C., Shao, X., Dhital, P., et al. (2002). Attributes and Characteristics of Exemplary, Leading, and Innovative Career and Technical Education Teacher Preparation Programs. National Research Center for Career and Technical Education.
Council of American Survey Research Organizations (2004). Surveys and you. Retrieved May 28, 2004 from http://www.casro.org/survandyou.cfm
Chandler, S. R. (1996). The associated landscape contractors of America career days: A Delphi study with implications for strategic planning in the 21st century. Unpublished doctoral dissertation, University of Georgia, Athens.
Cornish, (1977). The study of the future. Washington, D.C.: World Future Society.
Crews, T. B., & Ray, C. M. (Fall 1998). The telecommunication course in an end-user computing support program - research-based course content recommendation. Office Systems Research Association Journal, 16 (2), 9-16.
Dalkey, N. (1967). Delphi. Santa Monica: The RAND Corporation.
Farmer, E. A. (1998). A Delphi study of research priorities in Tech-Prep. Journal of Vocational and Technical Education, 15 (1), 42-49.
Flanders, F. B. (1988). Determining curriculum content for nursery/landscape course work in vocational agriculture for the 21st century: A futures study utilizing the Delphi technique (Doctoral dissertation, University of Georgia, 1988). Dissertation Abstracts International, 49, 09.
Geoffrion, A. (1987). Introduction to structured modeling. Management Science (33) 5, 547-588.
Gibbs, W., Graves, P. R., & Bernas, R. S. (2001). Evaluation guidelines for multimedia courseware. Journal of Research on Technology in Education, 34(1), 2-17.
Helmer, O. (1977). Problems in futures research: Delphi and causal cross-impact analysis. Futures, 17-31.
Jairath, N., & Weinstein, J. (1994). The Delphi methodology: a useful administrative approach. Canadian Journal of Nursing Administration, 7, 29-42.
LaBonty, D., & Scott, J. C. (November/December 1993) Perceptions of experts about business teacher education: A Delphi Study. Journal of Education for Business, 69(2), 99-104.
Lendaris, G. (1980). Structural modeling: A tutorial guide. IEEE Transactions on Systems, Man & Cybernetics, 807-840.
Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications. London: Addison-Wesley.
Martino, J. P. (1993). Technological forecasting for decision making. New York: McGraw-Hill.
McCallister, A. D. (1992). Determining characteristics of the forest industry in the 21st century: A Delphi study with implications for agriculture education programs. Unpublished doctoral dissertation, University of Georgia, Athens, GA.
McCurry, P. S. (1996). A Delphi study for perceptual modality classification of effective instructional activities in Tennessee marketing education. ERIC No. ED393975.
Murphy-Black, T., Lamping, D., McKee, M., Sanderson, C., Askham, J., & Marteau, T. (1998). CEM and their use in clinical guideline development - Factors which influence the process and outcome of CDMs. Health Technology Assessment, 2 (3).
Powell, C. (2003). The Delphi technique: Myths and realities. Methodological Issues in Nursing Research, 41 (4), 376-382.
Pullen, D. J. (1996). Secondary vocational supervisors' duties: A futures study utilizing the Delphi technique. Unpublished doctoral dissertation, University of Georgia, Athens.
Stewart, J. (2001). Is the Delphi technique a qualitative method? Medical Education, 35, 922-923.
Taylor-Powell, E. (2002). Quick tips collecting group data: Delphi technique. University of Wisconsin, Madison, WI. Retrieved May 28, 2004 from the University of Wisconsin - Extension Web site: http://www.uwex.edu/ces/pdande/resources/pdf/Tipsheet4.pdf
Turoff, M., & Hiltz, S. R. (1996). Computer based Delphi processes. Invited book chapter in Adler, M., & Ziglio, E. (Eds.) Gazing into the oracle: The Deplhi method and its application to social policy and public health. London: Kingsley.
Turoff, M. (1970). The policy Delphi. Journal of Technological Forecasting and Social Change, (2), 149-172.
Weatherman, R., & Swenson, K. (1974). Delphi technique. In S.P. Hencley & J.R. Yates (Eds.) Futurism in education: Methodologies. (pp. 97-114). Berkeley, CA: McCutchan.
Wilhelm, B. (Spring 1999). A Delphi study of entry-level workplace skills, competencies, and proof-of-achievement products. The Delta Pi Epsilon Journal, 41(2), 105-122.
Wilhelm, W. J. ((2001). Alchemy of the Oracle: The Delphi technique. The Delta Pi Epsilon Journal, 43(1), 6-26.
Wanda L. Stitt-Gohdes, Ed.D., Professor, University of Georgia, 225 River's Crossing, 850 College Station Rd., Athens GA 30602. E-mail: email@example.com. Office Phone: Office: 706.542.4078. Facsimile: 706.542.4054.
Tena B. Crews, Ed.D., Associate Professor, University of South Carolina, College of HRSM, Carolina Coliseum # 118B, Columbia, SC 29208. E-mail: firstname.lastname@example.org. Office Phone: 803.777.5559. Facsimile: 803.777.5645.