JTE v28n2 - Important Engineering and Technology Concepts and Skills for All High School Students in the United States: Comparing Perceptions of Engineering Educators and High School Teachers

Volume 28, Number 2
Spring 2017


https://doi.org/10.21061/jte.v28i2.a.3

Important Engineering and Technology Concepts and Skills for All High School Students in the United States: Comparing Perceptions of Engineering Educators and High School Teachers

Michael Hacker and Moshe Barak

Abstract

Engineering and technology education (ETE) are receiving increased attention as components of STEM education. Curriculum development should be informed by perceptions of academic engineering educators (AEEs) and classroom technology teachers (CTTs) as both groups educate students to succeed in the technological world. The purpose of this study was to identify ETE concepts and skills needed by all high school students in the United States and to compare perceptions of AEEs and CTTs relative to their importance. This research was carried out using a modified Delphi research methodology involving three survey rounds interspersed with controlled opinion feedback.

Consensus was found on 14 of 38 survey items within five ETE domains (design, modeling, systems, resources, and human values) that are repeatedly referenced in the literature. The most important competencies for high school students to learn were to: (1) identify and discuss environmental, health, and safety issues; (2) use representational modeling to convey the essence of a design; (3) use verbal or visual means to explain why an engineering design decision was made; and (4) show evidence of considering human factors when proposing design solutions. The study established a consensus between AEEs and CTTs that contributes to the body of knowledge about what high school students should learn in ETE. Study results can inform curriculum development and revision of the Standards for Technological Literacy .

Keywords : Delphi; engineering and technology education; high school; STEM; survey

Because of the essential roles that engineering and technology play in addressing societal and environmental challenges, support for PreK–12 engineering and technology education (ETE) programs in the United States has been rapidly growing ( Katehi, Pearson, & Feder, 2009 ). There is growing recognition that school-based ETE experiences can be pedagogically valuable for all students—not only in providing an effective way to contextualize and reinforce STEM skills but also in mobilizing engineering thinking as a way for young people to approach problems of all kinds ( Brophy & Evangelou, 2007 ; Forlenza, 2010). The purpose of this study was to compare the perceptions of two constituencies whose missions focus on preparing students to succeed in our technological world through engineering and technology education: academic engineering educators (AEEs) who prepare future engineers at the university level and high school classroom technology teachers (CTTs) who teach engineering and technology courses at the secondary school level. The study established a consensus among the groups about the most important ETE concepts and skills that all students in the United States should learn by the time they graduate from high school.

Literature Review

A literature review established a basis for identifying competencies for the initial item set in the study’s survey instrument. The review also determined how to optimally use Delphi research methodology to converge expert opinion to arrive at consensus ( RAND Corporation, 2017 ) and examined differences between engineering and technology and the preparation of professionals in those fields.

Differences Between Engineering and Technology Engineering

According to the Engineers’ Council for Professional Development (ECPD), the predecessor of the Accreditation Board for Engineering and Technology (ABET):

Engineering is the profession in which a knowledge of the mathematical and natural sciences gained by study, experience and practice is applied with judgment to develop ways to utilize, economically, the materials and forces of nature for the benefit of mankind. ( ECPD, 1979 ; as cited in National Research Council [NRC], 1986, p. 74 )

An earlier (1941) definition advanced by ECPD the is that “engineering is the creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination” ( Greefhorst & Proper, 2011, p. 9 ). Bloch (1986) wrote that “ engineering is the process of investigating how to solve problems . This process leads to a body of engineering knowledge consisting of concepts, methods, data bases, and, frequently, physical expressions of results” (p. 28). Wulf and Fisher (2002) describe “what engineers do . . . [as] ‘design under constraint’” (p. 36).

Technology . The National Assessment Governing Board (2013) defines technology as “any modification of the natural world done to fulfill human needs or desires” (p. xi). According to de Vries (2005) , technology is “the human activity that transforms the natural environment to make it fit better with human needs, thereby using various kinds of information and knowledge, various kinds of natural (material, energy) and cultural resources (money, social relationships, etc.)” (p. 11). Kline (1985) suggests that technology is viewed in four ways: as an artifact, as a methodology or technique, as a system of production, and as a sociotechnical system. Swyt (1989) , at the National Institute of Science and Technology, differentiates between engineering and technology by explaining that engineering is oriented toward the solution of specific problems, whereas technology is oriented toward development of new capability.

Preparation of Academic Engineering Educators and Classroom Technology Teachers

Academic engineering educators and classroom technology teachers in the United States come from different educational traditions, although both groups advocate the importance of technological literacy for the general population. Engineering emerged as a separate subject with the founding of the first schools of engineering and professional societies in the 18th century. AEEs typically have postgraduate degrees in engineering. In the United States, technology education emerged from industrial arts, and worldwide, technology education had its roots in crafts teaching. State-certified CTTs typically have master’s degrees in technology education.

ABET Program Standards for Engineering Programs . The Criteria for Accrediting Engineering Programs by ABET’s Engineering Accreditation Commission (2012) state that engineering graduates must have the ability to “apply knowledge of mathematics, science, and engineering”; “design and conduct experiments, as well as analyze and interpret data”; “design a system, component, or process to meet desired needs”; “function on multidisciplinary teams”; “identify, formulate, and solve engineering problems”; “communicate effectively”; and “use the techniques, skills, and tools necessary for engineering practice” (p. 3). Graduates must also “understand the impact of engineering solutions in a global, economic, environmental, and societal context”; recognize “the need for, and an ability to engage in life-long learning”; and understand “contemporary issues” (p. 3). ABET requires educational programs to include a major engineering design experience that builds upon the fundamental concepts of mathematics, basic sciences, the humanities and social sciences, engineering topics, and communication skills. Engineering topics must include subjects in the engineering sciences and engineering design, which “have their roots in mathematics and basic sciences but carry knowledge further toward creative application” (p. 4).

NCATE Program Standards for Technology Education Programs . The National Council for Accreditation of Teacher Education (NCATE) is the education profession’s mechanism to help establish high-quality teacher preparation programs ( National Council for Accreditation of Teacher Education [NCATE], 2008 ). NCATE has developed program standards (e.g., International Technology Education Association [ITEA] Council on Technology Teacher Education [CTTE], 2003 ; NCATE, 2008 ) that define the criteria for accrediting technology education programs in much the same manner as ABET has defined criteria for accrediting engineering programs. NCATE standards ( ITEA, CTTE, 2003 ) state that “within the contexts of the Designed World ,” “technology teacher education program candidates [must] develop an understanding of the nature of technology” (p. 22), “of technology and society” (p. 24), “of design” (p. 26), and “of the designed world” (p. 30) as well as “develop abilities for a technological world” (p. 28). Candidates must also “design, implement, and evaluate curricula based upon Standards for Technological Literacy” (p. 32), “use a variety of effective teaching practices that enhance learning of technology” (p. 34), “design, create, and manage learning environments that promote technological literacy” (p. 36), “understand students as learners, and how commonality and diversity affect learning” (p. 38), and engage “in comprehensive and sustained professional growth” (p. 40).

In a comparison of professional competencies required by ABET for engineers and NCATE for technology teachers, Hacker (2005) found that ABET focused on technical content preparation for engineers, whereas NCATE focused on pedagogy for teachers; however, a high degree of alignment was evident with respect to other competencies. He also found that both professional groups were well prepared in areas of professional practice, design and problem solving, team functioning, ethical and professional responsibility, communication skills, social and cultural impacts, and professional growth. One clear difference between the groups was that engineers were much more rigorously prepared in mathematics and science than technology teachers.

Projects Oriented Toward Formulating an ETE Knowledge and Skill Base

Major projects that have identified student learning outcomes in ETE include the Standards for Technological Literacy (STL; International Technology Education Association [ITEA], 2007 ); the National Academy of Engineering (NAE) reports ( Katehi, Pearson, & Feder, 2009 ; National Academy of Engineering [NAE], 2010 ); the National Research Council’s (2012) Framework for K–12 Science Education and the Next Generation Science Standards ( NGSS Lead States, 2013 ), which built upon it; the Technology and Engineering Literacy Framework for the National Assessment of Educational Progress (NAEP; National Assessment Governing Board [NAGB], 2013 ); and studies conducted by Custer, Daugherty, and Meyer (2010) , Childress and Rhodes (2008) , and Rossouw, Hacker, and de Vries (2010) .

The International Technology Education Association (ITEA), now the International Technology and Engineering Educators Association (ITEEA), developed the STL to identify “what students should know and be able to do in order to be technologically literate” ( ITEA, 2007, p. vii ). The standards are divided into five knowledge categories (comprising 20 content standards and 98 benchmarks at the Grades 9–12 level): the nature of technology, technology and society, design, abilities for a technological world, and the designed world .

NAE’s Committee on Standards for K–12 Engineering Education (2010) reviewed eight prior studies “that attempt[ed] to identify of core concepts, skills, and dispositions appropriate to K–12 engineering education” (p. 24). The 16 categories that they found included: design, STEM connections, engineering and society, constraints, communication, systems, systems thinking, modeling, optimization, analysis, collaboration and teamwork, creativity, knowledge of specific technologies, nature of engineering, prototyping, and experimentation (p. 35).

The Next Generation Science Standards (NGSS) (NGSS Lead States, 2013) grew from the National Research Council’s (2012) Framework for K–12 Science Education . The NGSS integrated disciplinary core ideas, science and engineering practices , and crosscutting concepts related to technology and engineering (including design, modeling, and systems) into student performance expectations ( NGSS Lead States, 2013 ).

The National Assessment of Educational Progress (NAEP) is an assessment of “what U.S. students know and are able to do in a range of subject areas” ( NAGB, 2013, p. ix ). In 2014, the NAEP Technology and Engineering Literacy Assessment was administered to 21,500 students in Grade 8 ( The Nation’s Report Card, 2016 ). “The assessment . . . consist[s] of technological content areas . . . and technological practices that characterize the field” among which are design and systems, information and communication technology , and technology and society ( NAGB, 2013, p. A-9 ).

In a study titled “Formulating a Concept Base for Secondary Level Engineering: A Review and Synthesis,” Custer, Daugherty, and Meyer (2010) identified 13 major engineering concepts (among them design, systems, and modeling) that were drawn from a variety of sources in the literature and from three focus groups of engineering experts.

In another study, Childress and Rhodes (2008) examined what high school students “should know and be able to do prior to entry into a postsecondary engineering program” (p. 5). Categories identified included engineering design, applications of engineering design, engineering analysis, engineering and human values, engineering communication, engineering science , and emerging fields of engineering .

As a part of the Concepts and Contexts in Engineering and Technology Education (CCETE) Project, a collaboration between Delft University of Technology in the Netherlands and Hofstra University in New York State, Rossouw et al. (2010) conducted a Delphi study with 32 international experts from nine countries to identify overarching themes and contexts that could be used to develop curricula for education about engineering and technology was developed. Table 1 lists the five main themes and associated subconcepts identified in that study.

Table 1: CCETE Project Overarching Themes and Subconcepts
Themes Subconcepts
Design Optimization and trade-offs; criteria and constraints; iteration
Modeling Representational, explanatory, predictive
Systems Systems/subsystems; input-process-output; feedback and control
Resources Materials, energy, information, time, tools, humans, capital
Human values Sustainability; technological assessment; creativity/innovation; ethical decisions

In this comparison of perceptions study, we used the five themes that emerged from the CCETE Project study as organizing categories because they aligned so well with those identified by other major projects. Further details about important ETE concepts and skills within these categories were added.

Summary of the Literature Review

Through the literature review, we identified ETE knowledge and skill sets that scholars believe to be important for all high school students to learn within their fundamental education. These concepts and skills informed the set of items that comprised this study’s Round 1 survey instrument. We established the basis upon which expert panelists suggested additions, changes, or deletions to survey items in subsequent Delphi rounds.

Research Questions

The research questions for this comparison of perceptions study were:

  1. RQ1 : Where does the strongest consensus exist among the expert panelists relative to the importance of specific ETE concepts and skills that all high school students in the United States should attain as part of their fundamental education?
  2. RQ2 : Which ETE concepts and skills does the expert panel perceive to be most important for high school students to attain as part of their fundamental education?
  3. RQ3 : Where are there significant differences between academic engineering educators’ and classroom technology teachers’ perceptions of the importance of ETE concepts and skills?
  4. RQ4 : Which concepts and skills that academic engineering educators and classroom technology teachers agree are highly important are not presently addressed by the STL?

Methodology

In this study, we employed Delphi survey research methodology because it is effective in soliciting and converging experts’ opinions to obtain consensus ( Salancik, Wenger, & Helfer, 1971 ). Delphi methodology assures anonymity, provides ongoing feedback to participants, and reduces the effects of bias due to group interaction ( Dalkey, 1972 ).

The purpose of a Delphi study is “to obtain the most reliable opinion consensus of a group of experts by subjecting them to a series of intensive questionnaires in depth interspersed with controlled opinion feedback” ( Dalkey & Helmer, 1963, p. v ). Studies comparing Delphi with other methods ( Ulschak, 1983 ) confirmed its effectiveness in generating ideas and its efficient use participants’ time.

Typically, a Delphi study starts by asking participants to respond to a specific question or issue. In subsequent rounds, participants are asked to consider feedback from the previous round, and the instrument is modified to reflect experts’ opinions. “The essential feature is the use of quantitative feedback given to each participant” ( Uebersax, 2000, 4.1 The Delphi Method, para. 1 ). When respondents’ estimates for an item do not fall within the range of group responses, they are asked to reconsider their position and, when justified, change their response. Thus, an attempt is made to achieve consensus ( Wicklein & Rojewski, 1999 ).

As is often done in Delphi studies ( Chalmers, 2014 ; Greer, 2008 ; Iqbal & Pipon-Young, 2009 ; Scott, Washer, & Wright, 2006 ), we used open-ended text boxes to invite panel members to provide feedback during survey rounds and at the conclusion of the survey.

Modified Delphi Methodology

In this study, we used a modified Delphi research methodology, which “is similar to the full Delphi in terms of procedure (i.e., a series of rounds with selected experts) and intent (i.e., to predict future events and arrive at consensus)” ( Custer, Scarcella, & Stewart, 1999, p. 51 ). Modifications included: (a) “beginning the process with a set of pre-selected items” (p. 51) that were drawn from the literature review and validated by experts and (b) adding validation panel meetings. Starting with a set of preselected items “(a) typically improves the initial round response rate, and (b) provides a solid grounding in previously developed work” (p. 51). Meetings of a validation panel verified the importance and level of abstraction of initial items, vetted prospective panelists to confirm their expertise, and added structure to the survey ( Rossouw, Hacker, & de Vries, 2010 ).

In accordance with the method suggested by Fowles (1978), seven stages characterized this study’s Delphi procedure.

  • Stage 1: Define the research questions.
  • Stage 2: Assemble the panel of experts (with help from the validation panel).
  • Stage 3: Design and validate the initial set of survey items (with validation panel help).
  • Stage 4: Conduct the three-round Delphi survey.
    • Round 1 included a beginning set of concepts drawn from the literature review.
    • Round 2 reflected changes based on panel input and solicited additional suggestions.
    • Round 3 included further changes based on final panel review.
  • Stage 5: Analyze survey results.
  • Stage 6: Summarize Conclusions.
  • Stage 7: Convene validation panel to review researchers’ conclusions and reach consensus.

In the literature, three Delphi rounds have been found sufficient to arrive at consensus ( Brooks, 1979 ) because after three iterations, not enough new information is gained to warrant the cost of more administrations ( Altschuld, 1993 ). Panelists were asked to rate each concept on a 7-point Likert scale using these descriptors: strongly agree (7), 6, moderately agree (6), agree (5), indifferent (4), moderately disagree (3), disagree (2), or strongly disagree (1). Panelists were invited to suggest and justify items that should be added or deleted. Panelists were informed that items would be modified based on their suggestions, and they were invited to reconsider item ratings if theirs were at variance with whole-group median ratings.

Participant Selection and Panel Size

Because the success of the Delphi technique relies upon experts’ judgment, selection of panelists was critical, and random selection was not considered. “Large numbers of respondents generate many items and ideas making the summarizing process difficult” ( Ludwig, 1997, Participation Selection, para. 1 ). Delbecq, Van deVen, and Gustafson (1975) suggest that 10 to 15 panelists are sufficient. Dalkey (1972) reported that reliability, with a correlation coefficient approaching 0.9, was found with a panel size of 13. J. G. Wells (personal communication, March 9, 2013) suggested that in research concerned with intragroup and intergroup judgments, subgroups of 16 panelists should be recruited. To allow for attrition, we recruited 18 AEEs and 17 CTTs (35 panel members in total) for this study.

Selection Criteria

Participants were selected because they were leading authorities in their fields with (a) documented participation in initiatives linking engineering and K–12 education, (b) a minimum of 5 years of experience teaching engineering or technology education, and (c) proven ability to formulate their thinking through research or active involvement in major funded projects. Participants were identified through recommendations from professional organizations and agencies (the American Society of Engineering Education, ITEEA, NAE, the National Science Foundation, and the New York State Technology and Engineering Educators Association) and recommendations from validation panel members.

Validation Panel

The validation panel was composed of the researchers, two AEEs with over 10 years of K–12 ETE experience, and two CTTs who are professional leaders with over 10 years of K–12 ETE experience. Validation panel meetings were 3 hours in duration. A meeting was held at the onset of the study to assist us in selecting panelists and validating survey items. The second meeting was held after the study concluded to discuss results, frame conclusions, and establish a cutoff point for items to be deemed as highly important for all high school students to learn.

Instrumentation and Data Analysis Methodology

The survey was tested and conducted online using Qualtrics (2014) survey software. Data was exported to SPSS (Version 22.0) for analysis. With Likert scale data, the use of median scores is strongly favored ( Hill & Fowles, 1975 ; Eckman, 1983; Jacobs, 1996 ). Data were treated as ordinal data ( Comrey, 1973 ) and were reported using descriptive statistics: medians, frequencies, percentiles, and interquartile range (IQR) statistics. A nonparametric test (the Mann-Whitney U) was used to determine statistically significant differences between the two study groups, and p-values were reported at the α = 0.05 level. Data provided insight into the study’s research questions as follows:

Determining consensus (Research Question 1) . Data analysis determined the strength of consensus on each item by subgroup and whole group. According to Rojewski and Meers (1991) , “Consensus . . . [is] determined using the interquartile range of each research priority [or concept] statement. Interquartile Range is a descriptive statistic defined as the distance between the first and third quartiles (i.e., the middle 50% of scores)” (p. 36). Low IQRs are one measure of strong consensus on an item.

In this study, we used a 7-point scale, and whole-group IQRs ranged from 0.79 to 1.98. After an analysis of scores within each quartile for each item, the researchers and the validation panel established that an IQR of ≤ 1.61 should be considered an indicator of strong panel consensus because:

  • Sixteen of the 17 highest rated items (with median ratings of ≥ 6.00, “agree”) displayed IQRs of ≤ 1.61 (indicating whole-group agreement that those items were of high importance), and
  • Three of the four lowest rated items (medians ≤ 5) displayed IQRs of ≤ 1.61 (indicating whole-group agreement that those items were of lower importance).

As suggested by Rayens and Hahn (2000) , the IQR may be an insufficient criterion for determination of agreement. “Frequency distributions are often used to assess agreement (McKenna, 1994)” ( Na, 2006, p. 44 ), and the criterion of some percentage of panelists responding to any given response category is used to determine consensus (Loughlin & Moore, 1979, p. 103; Seagle & Iverson, 2002, p. 1; Putnam, Speigel, & Bruininks, 1995; as cited in von der Gracht, 2008, p. 53 ).

In this study, factors determining consensus included the whole-group IQR and frequency of responses at the high end of the scale (respondents choosing scale points 6–7) and at the low end of the scale (respondents choosing scale points 1–4). These “consensus factors” are shown in Table 2.

Table 2: Consensus Factors
Item importance level Determinants of consensus
Consensus that an item is of higher importance If IQR ≤ 1.61 and frequency of high scores (6–7) ≥ 80%
Consensus that an item is of lower importance If IQR ≤ 1.61 and frequency of low scores (1–4) ≥ 25%

Determining importance (Research Question 2) . To determine importance, we examined Round 3 panelists’ median ratings for each item. Whole-group and subgroup (AEE and CTT) median ratings for each survey item were determined using SPSS (Version 22.0) software. The medians were ranked using the data ranking function of Microsoft Excel. The ranking indicated which of the survey items that the subgroups and the entire panel perceived to be most important. Because median ratings for all items were quite high (ranging from 6.71 to 4.60 on a 7-point scale), the validation panel set the item cutoff point for “high importance” at median ratings of ≤ 6.0. No survey items were deemed unimportant by the validation panel.

Determining significant differences (Research Question 3) . The Mann-Whitney U nonparametric test was used to analyze if intragroup median item ratings were significantly different. Nonparametric tests compare medians rather than means, and as a result, the influence of outliers is negated ( Hayes, 1997 ). At the conclusion of the third survey round, a lack of consensus on any survey item reflected sustained differences between the groups in that perceptual differences persisted despite the use of the Delphi instrument as a means to develop consensus. An alpha level (α) = 0.05 was used for all statistical tests of significance. The null hypothesis (H o ) was: There is no significant difference between AEEs and CTTs in their perception of the importance of ETE concepts and skills. A P -value of ≤ 0.05 on any survey item led to a rejection of the null hypothesis for that item.

Gap analysis with the Standards for Technological Literacy (Research Question 4) . In this study, we identified competencies deemed important for all high school students to attain as part of their fundamental education. We did a gap analysis with the STL to compare survey items rated “important” by the Delphi panel to existing benchmarks in the high school level standards. If items were similar, rewording of the STL benchmarks based on survey item wording was suggested. The validation panel confirmed the gap analysis.

Findings

Findings indicated where consensus between the AEEs and CTTs was reached about items that were of higher or lower importance. In discussing findings, items that were rated highest by the whole group and by each subgroup are identified, significant differences between subgroups are illuminated, and potential revisions to the STL are suggested. Additionally, findings determined the internal consistency (reliability) of the survey instrument and the mean value of the participants’ responses with regard to design, modeling, systems, resources, and human values.

Initial survey items were based on the literature review and on recent projects probing the importance of ETE concepts. As a result of prelaunch trials, the Round 1 survey instrument was revised 11 times prior to first round administration as part of a continuous improvement process.

The response rate to survey Round 1 was 88.6%, and 192 comments were received from panelists. Based on panelists’ suggestions, numerous changes were made. We attempted to be responsive to all suggestions; however, comments were sometimes contradictory, and we chose to accept suggested changes in wording that improved the clarity of the item. New items were added when two or more experts suggested its inclusion. Sixteen questions were reworded, and five new questions were added for the Round 2 survey.

The number of survey items increased from 32 items in Round 1 to 37 items in Round 2. In Round 2, panelists were asked to give high scores sparingly because the study was aiming to develop a list of the most essential concepts and skills. The response rate was again 88.6%. In Round 2, the IQRs of 13 of 32 items (40%) converged, attesting to the efficacy of the Delphi method at driving consensus.

In the final round, of the 34 panelists who were sent the Round 3 survey, 34 submitted responses (a 100% response rate). Respondents included 18 AEEs (four females and 12 males) and 16 CTTs (three females and 13 males). Appendix C presents the median ratings, standard deviations, percentiles, and whole-group IQRs by item. Findings are discussed below by research question.

Research Question 1

Where does the strongest consensus exist among the expert panelists relative to the importance of specific ETE concepts and skills that all high school students in the United States should attain as part of their fundamental education? AEE and CTT consensus about high importance was reached on 14 of 38 survey items based on both consensus factors (IQR ≤ 1.61 and frequency (6–7) ≥ 80%) being satisfied. The strongest consensus that items were highly important was found on Items R7 and M1: identify and discuss environmental, health, and safety issues involved in implementing an engineering project (Item R7) and use representational modeling (e.g., a sketch, drawing, or a simulation) to convey the essence of a design (Item M1). AEE and CTT consensus about lower importance was reached on two survey items based on both consensus factors (IQR ≤ 1.61 and frequency (1–4) ≥ 25%) being satisfied. The strongest consensus that items were of lower importance was found on Items D8 and D12: provide an example and an explanation of how design solutions can integrate universal design principles to help meet the needs and wants of people of all ages and abilities (Item D8) and describe, through an example, how the reliability of a system and the risks/consequences associated with its use have or have not been adequately considered prior to its implementation (Item D12). A list of items for which consensus was reached about higher and lower importance is included in Appendix A.

Research Question 2

Which ETE concepts and skills does the expert panel perceive to be most important for high school students in the United States to attain as part of their fundamental education? The ETE concept and skills perceived by the combined group to be most important for high school students to attain were: identify and discuss environmental, health, and safety issues involved in implementing an engineering project (Item R7); use representational modeling (e.g., a sketch, drawing, or a simulation) to convey the essence of a design (Item M1); explain why a particular engineering design decision was made, using verbal and/or visual means (e.g., writing, drawing, making 3-D models, using computer simulations; Item D6); show evidence of considering human factors (ergonomics, safety, matching designs to human and environmental needs) when proposing design solutions (Item HV6); and safely and correctly use tools and machines to produce a desired product or system (Item R4). Panelists’ perceptions of the most important ETE items for high school students to learn, by whole-group median ratings and rankings, are included in Appendix B.

Research Question 3

Where are there significant differences between academic engineering educators’ and classroom technology teachers’ perceptions of the importance of ETE concepts and skills? Data analysis using the Mann-Whitney U test indicated that subgroup ratings were significantly different on four survey items at the p < 0.05 level (see Table 3). All of these items except the third (Item S5) were rated higher by AEEs than by CTTs. Not surprisingly, engineers, more than teachers, emphasized applying science and mathematics to the solution of design problems.

Table 3: Significant Differences in Median Item Ratings Between AEEs and CTTs Based on the Mann-Whitney U Test
Median
Item Survey wording of item AEEs
( n = 18) 		CTTs
( n = 16) 		Mann-
Whitney
U value 		df p -value
exact sig.
(2-tailed)
D2 Solve engineering design
problems by identifying and
applying appropriate science
concepts. 		6.35 5.80 81.00 33 .012
D11 Provide examples of how
psychological factors (e.g.,
bias, overconfidence, human
error) can impact the
engineering design process. 		5.27 4.69 91.00 33 .049
S5 Explain the difference between
an open-loop control system
and a closed-loop control
system and give an example of
each. 		5.17 5.85 88.50 33 .040
S6 Develop and conduct empirical
tests and analyze system and
analyze test data to determine
how well actual system results
compare with measurable
performance criteria. 		6.21 5.36 89.00 33 .046

Research Question 4

Which concepts and skills that academic engineering educators and classroom technology teachers agree are highly important and are not presently addressed by the STL? The validation panel suggested that survey items with median ratings of ≥ 5.70 be considered for inclusion in the next iteration of the STL. Recommendations are made that the next iteration of the STL add, substitute, or reword standards based on 16 survey items that panelists agreed are highly important for high school students to attain as part of their fundamental education but are not presently addressed by the STL. Proposed changes to the STL are included in Appendix D.

Most STL benchmarks were written in terms of what students should learn; in this study, survey items were written in terms of what students should be able to do. Survey items might thus provide additional clarity to teachers and curriculum developers relative to measurable performances that would define important student capability. As an example, the present STL Standard 2Z indicates that students should know that: “Selecting resources involves trade-offs between competing values, such as availability, cost, desirability, and waste” ( ITEA, 2007, p. 42 ). However, in the suggested additions, students should be able to:

  • Improve an engineering design by identifying, making, and evaluating tradeoffs (D4);
  • Give an example of and investigate the impact of a tradeoff a company might make between profitability and environmental, health, or safety concerns (HV4); and
  • Engage in a group problem-solving activity to creatively generate several alternative design solutions and document the iterative process that resulted in the final design (D9).

Thus, students would be demonstrating their understanding of the above benchmark.

Additional Findings

Additional findings related to psychometric properties of the survey instrument (internal consistency reliability) and to comparing mean scores for all items within each of the five domains (subscales) of design, modeling, systems, resources, and human values.

Reliability. Often, Cronbach’s alpha is used when investigating the reliability of instruments using continuous or interval data. However, because this study’s data results from panelists’ responses to items rated on a Likert scale (scale points 1–7), data is ordinal; therefore, an ordinal alpha index of reliability was used as an alternative. Reliability coefficients for each subscale were determined using statistical methods better suited to ordinal data analysis.

The SPSS Categories procedure CATPCA (a nonlinear Categorical Principal Components Analysis) uses optimal scaling to statistically transform ordinal data into a quantitative numerical variable ( Meulman, Van der Kooij, & Heiser, 2004 ). CATPCA provides an ordinal alpha reliability measure, and the reliability coefficient calculated is for the transformed variables ( IBM Support, 2013 ).

To compare and confirm reliability statistics, both Cronbach’s Alpha and CATPCA ordinal alpha analyses were conducted (using SPSS, Version 22.0), and the results are shown in Table 4. Alpha reliability coefficients normally range between 0 and 1. “A reliability coefficient of 0.70 or higher is considered ‘acceptable’ in most social science research situations” ( UCLA, Institute for Digital Research and Education, 2017 , An Example, para. 2; see also George & Mallery, 2003 ; Kline, 1999 ). It is not surprising that the values for ordinal alpha were higher (because ordinal data is being analyzed) than those for Cronbach’s alpha, which treats Likert scale data as interval data.

Mean values of responses by category. Although participants’ answers to individual survey items are on an ordinal (Likert) scale, the answers to a group of items in a category can be regarded as close to normally distributed interval data. Therefore, these data were analyzed using mean values. A comparison of the means of each subgroup by category is shown in Table 4 and is also displayed graphically in Figure 1.

Table 4: Mean Values of AEEs (n = 18) and CTTs (n = 16) Final Round Responses Related to the Five Categories in the Questionnaire (Scale Points 1–7)
Category Number of items AEEs CTTs Cronbach's Alpha CATPCA Ordinal Alpha
Mean SD Mean SD
Design 12 5.8102 .63517 5.5885 .50412 .783 .857
Modeling 6 5.5926 .98389 5.6458 .62620 .773 .877
Systems 6 5.5926 .82490 5.7083 .40597 .595 .728
Resources 7 6.1429 .64635 6.2589 .53253 .623 .810
Human values 7 5.6825 .90159 5.5357 .53579 .794 .917

Figure 1. Mean scores for all items on each subscale by subgroup.

The highest mean scores for both subgroups were obtained in the Resources category. The lowest mean score for CTTs was in the Human Values category, and the lowest mean scores for AEEs were in the Systems and Modeling categories (tied).

In summary, salient findings included:

  • Descriptive statistics including median ratings, standard deviations, and the Interquartile Range (IQR) for each item;
  • A ranked analysis of the engineering and technology concepts and skills perceived to be most important for the general education of high school students by whole-group median rating;
  • An identification of items for which differences between subgroups were statistically significant;
  • A list of concepts and skills that experts agree are highly important for high school students to attain as part of their fundamental education that are not presently addressed by the STL; and
  • Internal consistency reliability measures of the subscales.

Limitations

A limitation of the present research related to the selection of the expert panelists: there was a considerable imbalance between more experienced (presumably older) and less experience (presumably younger) panelists. Thus, perspectives of younger educators who might have reflected more contemporary views of the importance of certain ideas and skills may not have been adequately considered. Therefore, it is recommended that in selecting panelists for future studies, targeted efforts should be made to recruit younger panelists to determine if their perceptions about the importance of knowledge and skills related to contemporary technologies differ significantly from their more experienced, presumably older, colleagues.

Conclusions

Because engineering and technology education are receiving greater attention as components of STEM education, support for the establishment of PreK–12 ETE programs in the United States has been rapidly growing. Although university level academic engineering educators are an ideal professional constituency to ally with and support secondary school ETE programs, prior to this study, it was uncertain whether they held similar perceptions about the fundamental knowledge and skills that high school graduates need for life in a technological world to the classroom technology teachers who develop curriculum and deliver secondary school ETE instruction.

We have examined the alignment of the two constituencies’ perceptions about the importance of key concepts and skills that all high school students in the United States should learn as part of their fundamental education. Our findings demonstrate that there is indeed a greater degree of concordance than there are perceptual differences between the two constituencies.

From a theoretical perspective, this research contributes to the body of knowledge about the most salient ideas and skills that students need to learn and understand in five overarching domains of engineering and technology that are repeatedly referenced in the literature: design, modeling, systems, resources, and human values. Additionally, this study provides the first research-based comparison of perceptions about important ETE ideas and skills between two constituencies whose missions focus on preparing students to succeed in our technological world through engineering and technology education.

From the methodological perspective, this study illustrates how the Delphi technique can be employed within an education research study in which the emphasis is on eliciting and comparing the perceptions of different groups of experts. On one hand, the Delphi technique was utilized to identify perceptual differences between expert groups with different backgrounds; on the other hand, it was used to bridge differences in background in order to forge consensus. The Delphi research methodology used in this study was modified from the classical Delphi approach in several ways. Modifications that could be considered by other researchers include: (1) beginning the Delphi process with a set of carefully preselected items that were drawn from the literature review, (2) adding validation panel reviews and meetings to help identify panelists and initial survey items and to reach post-survey consensus, (3) establishing a set of selection criteria for choosing expert panelists, (4) including open-ended text boxes to solicit and present arguments for or against items being included in the list of “important” survey items, (5) establishing an IQR range on a Likert scale as being indicative of strong consensus, and (6) establishing frequency distribution percentage criteria for responses at both the high end and the low end of the scale.

Within the framework of this research study, a method for examining internal consistency reliability suitable to interpreting ordinal data is proposed based on Categorical Principal Components Analysis (CATPCA), as a replacement for the Cronbach’s alpha coefficient that is typically used to interpret interval data.

From a practical perspective, this research contributes to engineering and technology education by:

  • Establishing a basis for educators to develop local, state, and national ETE curriculum frameworks, instructional materials for students and teachers, and assessments of teaching and learning;
  • Informing a revision of the Standards for Technological Literacy ;
  • Establishing a basis for educators to develop local, state, and national ETE curriculum frameworks, instructional materials for students and teachers, and assessments of teaching and learning;
  • Guiding the design of proposals to foundations and government agencies to fund improvement of ETE curriculum and instruction.

References

Accreditation Board for Engineering and Technology. (2000). Criteria for accrediting engineering programs. Baltimore, MD: Author.

Accreditation Board for Engineering and Technology, Engineering Accreditation Commission. (2012). Criteria for accrediting engineering programs: Effective for reviews during the 2013-2014 accreditation cycle. Baltimore, MD: Author. Retrieved from http://www.abet.org/wp-content/uploads/2015/04/eac-criteria-2013-2014.pdf

Altschuld, J. W. (1993). Delphi technique. Lecture. Evaluation Methods: Principles of Needs Assessment II. Columbus: Department of Educational Services and Research, The Ohio State University.

Bloch, E. (1986). Science and engineering: A continuum. In National Research Council, The new engineering research centers: Purposes, goals, and expectations (p. 28–34). Washington, DC: National Academy Press. doi:10.17226/616

Brooks, K. W. (1979). Delphi technique: Expanding applications North Central Association Quarterly, 53 (3), 377–385.

Brophy, S., & Evangelou, D. (2007). Precursors to engineering thinking (PET) project: Intentional Designs with Experimental Artifacts (IDEA). In Proceedings of the 2007 Annual Conference & Exposition (pp. 12.1169.1–12.1169.11). Washington, DC: American Society of Engineering Education. Retrieved from https://peer.asee.org/3011

Chalmers, K. J., Bond, K. S., Jorm, A. F, Kelly, C. M., Kitchener, B. A., & Williams-Tchen, A. J. (2014). Providing culturally appropriate mental health first aid to an aboriginal or Torres Strait Islander adolescent: Development of expert consensus guidelines International Journal of Mental Health Systems, 8 (1). doi:10.1186/1752-4458-8-6

Comrey, A. (1973). A first course on factor analysis. London, England: Academic Press

Childress, V., & Rhodes, C. (2008). Engineering student outcomes for Grades 9–12. The Technology Teacher, 67 (7), 5–12.

Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education, 15 (2), 50–58. doi:10.21061/jcte.v15i2.702

Custer, R. L., Daugherty, J. L., & Meyer, J. P. (2010). Formulating a concept base for secondary level engineering: A review and synthesis. Journal of Technology Education, 22 (1), 4–21. doi:10.21061/jte.v22i1.a.1

Dalkey, N.C. (with Rourke, D.L., Lewis, R., & Snyder, D.). (1972). Studies in the quality of life: Delphi and decision-making. Lexington, MA: Lexington Books.

Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts (Memorandum RM-727/1-Abridged). Santa Monica, CA: RAND Corporation. Retrieved from https://www.rand.org/content/dam/rand/pubs/research_memoranda/2009/RM727.1.pdf

Delbecq, A. L., Van deVen, A. H., & Gustafson, D. H. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Glenview, IL: Scott, Foresman and Company.

de Vries, M. J. (2005). Teaching about technology: An introduction to the philosophy of technology for non-philosophers. Dordrecht, the Netherlands: Springer. doi:10.1007/1-4020-3410-5

Engineers' Council for Professional Development. (1979). 47th Annual Report 1978-79 (p. 56). New York, NY

Fowles, J. (1978). Handbook of futures research. Westport, CT: Greenwood Press.

George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Boston, MA: Allyn & Bacon.

Greefhorst, D., & Proper, E. (2011). Architecture principles: The cornerstones of enterprise architecture , (p.9). London, England: Springer Heidelberg Dordrecht. ISBN 978-3-642-20278-0. DOI 10.1007/978-3-642-20279-7

Greer, S. J. (2008). Convergence on the guidelines for designing a virtual irregular warfare internship: A Delphi study (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3320644).

Hacker, M. (2005, November). A comparison of professional competencies required by ABET for engineers and NCATE for technology teachers. Paper presented at the Mississippi Valley Technology Teacher Education Conference, St. Louis, MO.

Hayes, A. (1997). Research methods and statistics. Armidale, New South Wales, Australia: School of Psychology, University of New England. Retrieved December 26, 2012 from http://www.une.edu.au/WebStat/unit_materials/c6_common_statistical_tests/nonparametric_test.html

Hill, K. Q., & Fowles, J. (1975). The methodological worth of the Delphi forecasting technique. Technological Forecasting and Social Change, 7 (2), 179–192. doi:10.1016/0040-1625(75)90057-8

IBM Support. (2016). Does SPSS provide an internal consistency measure for ordinal variables?. Retrieved from http://www-01.ibm.com/support/docview.wss?uid=swg21477603

International Technology Education Association. (2007). Standards for technological literacy: Content for the study of technology (3rd ed.). Reston, VA: Author.

International Technology Education Association, Council on Technology Teacher Education. (2003). ITEA/CTTE/NCATE Curriculum Standards: Initial Programs in Technology Teacher Education. Reston, VA: Author. Retrieved from http://ctete.org/wp-content/uploads/2016/02/NCATEStandards10.03.pdf

Iqbal, S., & Pipon-Young, L. (2009). The Delphi method. The Psychologist, 22 (7), 598–600.

Jacobs, J. M. (1996). Essential assessment criteria for physical education teacher education programs: A Delphi study (Unpublished doctoral dissertation). West Virginia University, Morgantown, WV.

Katehi, L., Pearson, G., & Feder, M. (Eds.). (2009). Engineering in K–12 education: Understanding the status and improving the prospects. Washington, DC: National Academies Press. doi:10.17226/12635.

Kline, S. J. (1985). What is technology? Bulletin of Science, Technology & Society, 5 (3), 215–218. doi:10.1177/027046768500500301.

Kline, P. (1999). The handbook of psychological testing (2nd ed.). London, England: Routledge.

Ludwig, B. (1997). Predicting the future: Have you considered using the Delphi methodology? Journal of Extension, 35 (5). Retrieved from http://www.joe.org/joe/1997october/tt2.html

Meulman, J. J., Van Der Kooij, A. J., & Heiser, W. J. (2004). Principal components analysis with nonlinear optimal scaling transformations for ordinal and nominal data. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 49–70). Thousand Oaks, CA: Sage. doi:10.4135/9781412986311.n3

Na, S. (2006). A Delphi study to identify teaching competencies of teacher education faculty in 2015 (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3225382)

National Academy of Engineering. (2010). Standards for K–12 engineering education? Washington, DC: National Academies Press. doi:10.17226/12990

National Assessment Governing Board. (2013). Technology and engineering literacy framework for the 2014 National Assessment of Educational Progress. Washington, DC: Author. Retrieved from https://www.nagb.org/content/nagb/assets/documents/publications/frameworks/technology/2014-technology-framework.pdf

National Assessment Governing Board. (2013). Technology and engineering literacy framework for the 2014 National Assessment of Educational Progress. Washington, DC: Author. Retrieved from https://www.nagb.org/content/nagb/assets/documents/publications/frameworks/technology/2014-technology-framework.pdf

National Council for Accreditation of Teacher Education. (2008). Professional standards for the accreditation of teacher preparation institutions. Washington, DC: Author. http://www.ncate.org/documents/standards/NCATE%20Standards%202008.pdf

National Research Council. (1986). Engineering infrastructure diagramming and modeling. Washington, DC: National Academy Press. doi:10.17226/587

National Research Council. (2012). A framework for K–12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press. doi:10.17226/13165

The Nation’s Report Card. (2016). 2014 technology & engineering literacy (TEL): About the TEL assessment. Retrieved from https://www.nationsreportcard.gov/tel_2014/#about/overview

NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: National Academies Press. doi:10.17226/18290

Qualtrics (2014) [Computer software]. Provo, UT: Qualtrics Labs.

RAND Corporation. (2017). Delphi method. Retrieved from http://www.rand.org/topics/delphi-method.html

Rayens, M. K., & Hahn, E. J. (2000). Building consensus using the policy Delphi method. Policy, Politics, & Nursing Practice, 1 (4), 308–315. doi:10.1177/152715440000100409

Rojewski, J. W., & Meers, G.D. (1991). Research priorities in vocational special needs education. Journal for Vocational Special Needs Education, 13 (2), 33–38.

Rossouw, A., Hacker, M. & de Vries, M. J. (2010). Concepts and contexts in engineering and technology education: An international and interdisciplinary Delphi study. International Journal of Technology and Design Education, 21 (4), 409–424. doi:10.1007/s10798-010-9129-1

Salancik, J. R., Wenger, W., & Helfer, E. (1971). The construction of Delphi event statements, Middletown, CT: Institute for the Future.

Scott, D. G., Washer, B. A., & Wright, M. D. (2006). A Delphi study to identify recommended biotechnology competencies for first-year/initially certified technology education teachers. Journal of Technology Education, 17 (2), 43–55. doi:10.21061/jte.v17i2.a.4

Smith, R.J. (2016). Engineering. Encyclopedia Britannica. Retrieved March 21, 2017 from https://www.britannica.com/technology/engineering

SPSS (Version 22.0) [Computer software]. Armonk, New York: IBM.

Swyt, D. A. (1989). Knowledge, innovation, and the administration of publicly-funded research in technology. Unpublished manuscript, National Institute of Science and Technology, Gaithersburg, MD.

UCLA, Institute for Digital Research and Education. (2017). SPSS FAQ: What does Cronbach’s alpha mean? Retrieved from http://www.ats.ucla.edu/stat/spss/faq/alpha.html

Uebersax, J. (2000). Agreement on interval-level ratings. Retrieved from http://www.john-uebersax.com/stat/cont.htm

Ulschak, F. L. (Ed.). (1983). Human resource development: The theory and practice of need assessment. Reston, VA: Reston.

von der Gracht, H. A., (2008). The future of logistics: Scenarios for 2025. Wiesbaden, Germany: Gabler. doi:10.1007/978-3-8349-9764-7

Wicklein, R. C. & Rojewski, J. W. (1999). Toward a “unified curriculum framework” for technology education. Journal of Industrial Teacher Education, 36 (4), 38–56. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v36n4/wicklein.html

Wulf, Wm. A., & Fisher, George M. C. (2002). A makeover for engineering education. Issues in Science and Technology, 18 (3), 35–39. Retrieved from http://issues.org/18-3/p_wulf/

About the Authors

Michael Hacker ( michael.hacker@hofstra.edu ) is Co-Director of the Center for STEM Research at Hofstra University in New York.

Moshe Barak ( mbarak@bgu.ac.il ) is Professor of Science and Technology Education at Ben Gurion University of the Negev in Beersheva, Israel.

Edited