Engineering's Grand Challenges: Priorities and Integration Recommendations for Technology Education Curriculum Development

Jennifer Buelin, Aaron C. Clark, & Jeremy V. Ernst

Abstract

In this study, the 14 Grand Challenges for Engineering in the 21st Century identified by the National Academy of Engineering were examined by a panel of experts in an effort to identify prospective curricular integration opportunities in the field of technology and engineering education. The study utilized a three-round modified Delphi methodology to forecast and build consensus pertaining to the beneficial role of the Grand Challenges in education and the level in which they should enter the K–12 scope and sequence. The findings of this study indicate that experts have dissimilar opinions about the role that the Grand Challenges should play in K–12 technology and engineering curricula. Most notably, there was strong agreement among participants concerning the integration of study and application associated with making solar energy economical for the masses. Educational implications of such incorporation are identified and explored.

Keywords: engineering education; curricula development; Delphi study; grand challenges; K–12 technology and engineering education curricula

Engineering and design function as core components of contemporary technology education classes, promoting and enabling within students technological literacy, college and career readiness, creativity, and a global, empirically driven perspective. Although the development of curricula pertaining to student explorations of authentic and realistic engineering design learning experiences is clearly a point of focus for K–12 and postsecondary attention, there are few fully developed programs of study.

In 2008, the National Academy of Engineering released a list of 14 challenges deemed critical to the continued advancement of civilization and human health ( National Academy of Engineering [NAE], 2016 ). The scope of the list is broad, including topics from disciplines ranging from cyber and nuclear security to issues of sustainability and learning. The 14 Grand Challenges for Engineering in the 21st Century are listed in Table 1.

On March 23, 2015, a letter of commitment from 122 U.S. schools of engineering was presented to President Barack Obama at the White House Science Fair pledging to train and formally recognize more than 20,000 "'Grand Challenge Engineers' over the next decade" ( NAE, 2015, para. 2 ). In their letter of commitment, "U.S. Engineering School Deans' Response to President Obama on Educating Engineers to Meet the Grand Challenges," they state:

We affirm the importance of such aims as a reflection of our core values, as a source of inspiration for drawing a generation to the call of improving the human condition, as a driver for our national and world economies, and as essential to U.S. and global security, sustainability, health, and joy of living in the decades ahead. We further note that achieving these Grand Challenges requires technology and engineering, but that none can be solved by engineering alone. Hence, there is a crucial need for a new educational model that builds upon essential engineering fundamentals to develop students' broader understanding of behavior, policy, entrepreneurship, and global perspective; one that kindles the passion necessary to take on challenges at humanity's grandest scale. (2015, para. 2-3)

Table 1: Engineering's 14 Grand Challenges ( NAE, 2016 )

1. Make Solar Energy Economical

2. Provide Energy From Fusion

3. Develop Carbon Sequestration Methods

4. Manage the Nitrogen Cycle

5. Provide Access to Clean Water

6. Restore and Improve Urban Infrastructure

7. Advance Health Informatics

8. Engineer Better Machines

9. Reverse-Engineer the Brain

10. Prevent Nuclear Terror

11. Secure Cyberspace

12. Enhance Virtual Reality

13. Advance Personalized Learning

14. Engineer the Tools of Scientific Discovery

In the 7 years since the NAE initially released the 14 Grand Challenges for Engineering in the 21st Century, efforts have been made toward addressing those challenges in K–12 technology and engineering curricula; however, experts working in these fields still have widely divergent views of what and how students should learn. The engineering schools' recent letter of commitment includes a list of key elements that each institution will address. Among them are "a creative learning experience connected to the Grand Challenges such as research or design projects," "authentic experiential learning," and "entrepreneurship and innovation experience" (U.S. Engineering School Deans' Response to President Obama on Educating Engineers to Meet the Grand Challenges, 2015, para. 4). These objectives bear remarkable similarities to those found in the Standards for Technological Literacy ( International Technology Education Association [ITEA], 2007 ).

In recent years, the field of technology education has strategically aligned initiatives and efforts with those of engineering education. In 2010, the International Technology Education Association (ITEA) fundamentally embraced engineering, becoming the International Technology and Engineering Educators Association (ITEEA), representing a curricular shift toward the active and holistic integration of STEM's T and E . Alongside this shift came ITEEA's support of Engineering byDesign (EbD™), a "standards-based national model for Grades K-12 that delivers technological literacy in a STEM context" ( International Technology and Engineering Educators Association [ITEEA], 2016, para. 1 ).

Since 2003, ITEEA's curricular framework has been defined by the Standards for Technological Literacy: Content for the Study of Technology ( ITEA, 2007 ). This document identifies 20 standards outlining "what students should know and understand about technology, and what they should be able to do . . . in order to be technologically literate" (p. 14). "The organization stated that the content contained within the STL standards was the foundation for students to develop 21st Century STEM literacy—the very core of abilities needed for students to become advanced problem solvers, innovators, technologists, engineers, and knowledgeable citizens" ( Asunda, 2012, p. 50 ). Contention exists, however, as to whether this is sufficient. Gattie and Wicklein (2007) proposed "adjusting the focus of Technology Education to a defined emphasis on engineering design and the general process by which technology is developed" (p. 6). Rose (2010) noted the direct relationship between human technological innovation and the origins of many challenges that today's engineers face, giving particular attention to human threats to ecosystems and public health.

In addition to EbD™, at least two large-scale instructional materials providers have begun to offer pre-engineering content at the K– 12 level: Engineering is Elementary® (EiE®), a product of the National Center for Technological Literacy at the Museum of Science, Boston, for Grades 1 through 8, and Project Lead the Way (PLTW), a nonprofit organization started in 1986 by a high school pre-engineering teacher and developed with support from the Charitable Leadership Foundation ( Project Lead the Way [PLTW], 2014 ). The EiE® website states that they address "America's pressing need for effective STEM education in three ways: curriculum development and dissemination, professional development (PD), and educational research" ( Engineering is Elementary [EiE], 2016a, para. 2 ). Similarly, PLTW claims to have "grown to become the nation's leading provider of science, technology, engineering, and mathematics (STEM) programs for students in grades K– 12" by preparing "students to be the next generation of problem solvers, critical thinkers, and innovators for the global economy" (2014, para. 1).

"More than 6,500 schools in all 50 states and the District of Columbia offer PLTW courses to their students" ( PLTW, 2014, para. 4 ). Although PLTW's program offerings in engineering, biomedical science, and computer science offer potential learning contexts for alignment with the Grand Challenges, documentation of programmatic alignment is unavailable at present. EiE® similarly offers potential learning contexts for alignment with the Grand Challenges, and EiE® curriculum units such as "To Get to the Other Side: Designing Bridges" and Now "You're Cooking: Designing Solar Ovens" provide curricular entry points to the basic concepts underlying the Grand Challenges ( EiE, 2016b ). Although EiE® has published alignment guides demonstrating standards alignment with the Standards for Technological Literacy, Next Generation Science Standards , Georgia Performance Standards for K–5 Science, and Common Core State Standards, no such guide is available for alignment with the Grand Challenges at present.

Engineering by Design (EbD) is offered to K–12 students in more than 20 states and Engineering is Elementary (EiE) is a Grade 1–8 curriculum that is offered in schools in all 50 states and the District of Columbia. EbD and ITEEA published a matrix ( ITEEA, 2012 ) of course alignment with the Grand Challenges, including a scaled indicator of the level of detailed content integration and assessment ranging from 1 ( Topics and lessons refer to previous knowledge ) to 4 ( Content is directly integrated into lessons in detail and assessed ).

Examination of the matrix ( ITEEA, 2012 ) reveals room for additional curriculum development designed to specifically address the content of the Grand Challenges. The number of Grand Challenges referenced at any level ranges from one to eight across a semester. Of the 15 EbD courses mapped on the matrix, the mean number of Grand Challenges referenced at any level is 5.07. Although Restore Urban Infrastructure is reported to be aligned with 11 EbD courses at either the 3 or 4 level, Advance Personalized Learning is addressed only in a single EbD course, Foundations of Technology 3. At present, documentation more extensively reporting on the nature of EbD course alignment with the Grand Challenges has not been published.

In recent years, supplementary teaching and learning materials have been published, some at no cost, for teachers who wish to address the Grand Challenges at some level alongside current curricula. The NAE Grand Challenge K12 Partners Program (2016) outlines the "5-Part Make It Happen Plans," structured on the steps: "Learn It," "Do It," "Share It," "Create It," and "Teach It." The Partners Program website functions as a repository for 5-part plans created by a community of stakeholders, including those interested in becoming partners. These plans offer blueprints of a sort for the development of curricula, but they are not fully developed lessons.

Further compounding the national dialogue on the direction of pre-engineering education in the United States, in spring 2013, the Next Generation Science Standards (NGSS) were released, stressing the interdependence of science, engineering, and technology and the influence of science, engineering, and technology on society and the natural world (NGSS Lead States, 2013). Although the Framework for K–12 Science Education document, which "provided the foundation for" the NGSS, took "a big step toward widespread inclusion of engineering at the K-12 level, this document does not articulate a complete set of core ideas in engineering appropriate for K-12 students as the 2010 NRC report recommends" ( Moore et al., 2014, p. 3 ). Furthermore, the "limited treatment of engineering" in the NGSS (p. 3) is directly addressed in the standards document itself. Appendix I states: "It is important to point out that the NGSS do not put forward a full set of standards for engineering education, but rather include only practices and ideas about engineering design that are considered necessary for literate citizens" ( NGSS Lead States, 2013, p. 104 ).

In spite of the availability of materials from providers such as EbD, PLTW, and EiE as well as supplementary content from providers such as the NAE Grand Challenge K12 Partners Program, there appears to be a need for the development of additional content for K–12 education that not only introduces the basic concepts underlying the Grand Challenges but also supports substantive and authentic inquiry into concepts critical to the pursuit of solutions to those challenges. The purpose of this study was to explore that need using a panel of experts selected from technology and engineering education.

Research Questions

A study was proposed and initiated to investigate elements of the Grand Challenges that are currently deemed relevant and accessible to the field of technology and engineering education. The central goal for this study was to arrive at a concise listing of engineering's Grand Challenges that should actively be pursued in the development of curriculum materials. The following research questions guided this investigation:

1. Which of the Grand Challenges identified by the National Academy of Engineering lend themselves to curricular integration in the field of K–12 technology and engineering education?
2. Are there additional challenges that should be added to the list for purposes of curriculum development that are appropriate and beneficial to the study of technology and engineering education?
3. How important are individual challenges to the K–12 technology and engineering curriculum?
4. At what point (elementary, middle, or high school) is it appropriate to introduce each challenge to students?

Rationale and Background

The researchers selected a modified Delphi technique for attaining consensus from experts in fields related to engineering and technology education as the most appropriate design for this study. It allows for participants to be anonymous during the process; only the researchers know who is responding, and no one individual can dominate the discussion ( Wright & Geroy, 1988 ). A number of individuals have used or suggested the use of this method for curriculum development, including many researchers in fields related to technology and engineering education (e.g., Dalkey, 1972 ; Paige, Dugger, & Wolansky, 1996 ; Volk, 1993 ; Zargari, Campbell, & Savage, 1995 ). The conventional Delphi methodology consists of three to four rounds of instruments designed to achieve consensus among experts ( Meyer & Booker, 1990 ). The overall procedures for conducting this type of research came from sources that recommend the use of a panel of experts to compile a list of program characteristics (e.g., Delbecq, Van de Ven, & Gustafson, 1986 ; Linstone & Turoff, 1975 ; Meyer & Booker, 1990 ). By adopting this method, researchers were assured that the participants in the study represented a variety of different professional backgrounds (i.e., experts) in fields related to engineering and technology education. By using email for communication and the Internet for sharing resources, the study could be completed in an economical and efficient manner ( Clark & Scales, 2003 ).

The Delphi technique was developed by the RAND Corporation in the 1950s as a forecasting tool for the U.S. military for the purpose of soliciting reliable responses from a group of experts ( Stitt-Gohdes & Crews, 2004 ). The method begins with an open-ended solicitation for expert opinions on a topic or problem, followed by additional rounds of inquiry in which participants are charged with rating the importance of items and fine-tuning the phrasing and substance of item content ( Custer, Scarcella, & Stewart, 1999 ). The modified approach—specifically, the choice to begin with the NAE's list of Grand Challenges as well as an open-ended write-in option—was deemed preferable to the traditional approach because of the pre-existing support in the research literature of the value of the Grand Challenges to future technology curriculum development.

The Delphi method has a history of implementation in technology education research ( Clark & Wenig, 1999 ; Custer, Scarcella, & Stewart, 1999 ; Scott, Washer, & Wright, 2006 ). Wicklein (1993) used a modified Delphi to "determine the present and future critical issues and problems facing the technology education profession" (p. 56). Katsioloudis (2010) used the method to identify quality visual-based learning materials [such as tables and photographs] for technology education. Clark and Wenig (1999) used the Delphi for identifying "quality indicators for technology education programs" (p. 23).

Further influencing the selection of the modified Delphi procedure for obtaining consensus from a nationally dispersed panel of experts in technology and engineering education were the numerous simultaneous influences on the trajectory of technology and engineering curriculum development described above. Stitt-Gohdes and Crews (2004) identified common reasons for choosing the Delphi over alternative methodologies:

• The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis.
• More individuals are needed than can effectively interact in a face-to-face exchange.
• Time and cost make frequent group meetings infeasible.
• Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured.
• The heterogeneity of the participants must be preserved to assure validity of the results, i.e., avoidance of domination by quantity or my strength of personality ( Linstone & Turoff, 1975 ). (p. 56-57)

Methods

Three rounds of surveys were conducted digitally using the modified Delphi technique in order to identify the Grand Challenges as well as related challenges that experts in the field believe to be valuable content for curriculum development in technology and engineering education.

The Panel

The panel of experts ( n = 27) was randomly selected for participation from each of the following strata: ITEEA's published list of Distinguished Technology and Engineering Professionals (DTEs) ( n = 6); ITEEA's Teacher Excellence Award recipients at the elementary ( n = 1), middle ( n = 6) and high school ( n = 4) levels for 2009–2013; self and peer nominees from the 2013 active membership of the ITEEA Council for Supervision and Leadership ( n = 5); and the American Society for Engineering Education K–12 Division 2013 active membership ( n = 5). Participants averaged 17.43 years of professional experience in technology and engineering education. The panel consisted of 14 females and 13 males. Participants represented all regions of the United States: South ( n = 13), Northeast ( n = 7), Midwest ( n = 6), and West ( n = 1).

Round 1

Round 1 provided participants with a brief overview of and links for additional information about the National Academy of Engineering's Grand Challenges, ITEEA, and NGSS. In Round 1, participants were asked the following questions.

1. "Which of the Grand Challenges identified by the National Academy of Engineering lend themselves to curricular integration in the field of K–12 technology and engineering education?" The response options for this question were keep or reject .
2. "Are there additional challenges that should be added to the list for purposes of curriculum development that are appropriate and beneficial to the study of technology and engineering education?" This question allowed for write-in responses.

Descriptive statistics were used to determine the challenges that would be eliminated from further inquiry. Commonalities among written responses led to the addition of three additional challenges to be presented in Round 2. Notification of Round 1 results and a period of participant feedback led to the acceptance of the eliminations and additions prior to the release of Round 2 surveys.

Round 2

In Round 2, participants were asked to do the following.

1. "Rank the 13 challenges (the remaining 10 Grand Challenges, along with the three new challenges) on the importance of their role in the K–12 technology and engineering curriculum," using both a Likert-type ranking ranging from is not needed to must be included as well as a drag-and-drop rank ordering feature; and
2. "Indicate the appropriate point (elementary, middle, or high school) at which the topic should be introduced to students," a multiple choice prompt.

Statistical analysis led to the elimination of three additional challenges from further inquiry as well as the categorization of the remaining challenges into elementary, middle, and high school groupings.

Round 3

A final list of challenges, including the suggested additions, was presented to participants in Round 3 for acceptance or rejection for each level (elementary, middle, and high school) and for rank ordering in terms of importance at each level.

Findings

The first Delphi round eliminated the following Grand Challenges, which fell into the upper quartile (most often rejected), from further consideration in this study: Provide Energy from Fusion, Develop Carbon Sequestration Methods, Manage the Nitrogen Cycle , and Reverse-Engineer the Brain . Round 1 results suggested the need to identify a developmentally appropriate point of introduction to the curriculum (elementary, middle, or high school) for each challenge. Commonalities among responses to the write-in question about additional relevant challenges led to the addition of three new challenges to be presented in Round 2: Provide Access to Adequate Nutrition, Develop Assistive Technologies for the Disabled , and Advance Nanotechnology .

In Round 2, the Grand Challenges were ranked based on the importance of their role in the K–12 technology and engineering curriculum, ranging from "is not needed" to "must be included." Prevent Nuclear Terror, the single challenge receiving a mean score less than 3.0, was eliminated from further investigation. Challenges were rank ordered using the Qualtrics drag-and-drop feature, and the three challenges falling into the upper quartile (ranked as least important)—Advance Health Informatics, Enhance Virtual Reality, and Prevent Nuclear Terror—were eliminated from further investigation (Figure 1) ¹ .

Figure 1. Round 2 mean rankings of challenges: importance to K-12 technology and engineering curriculum.

Figure 2. Round 2 mean rankings of challenges:relative importance for 10 accepted challenges and three suggested challenges.

Based on Round 2 results, challenges were categorized by recommended point of introduction: elementary, middle, or high school (see Table 2). For two challenges— Make Solar Energy Economical and Advance Personalized Learning —recommendations were evenly dispersed among elementary, middle, and high school; therefore, inquiry moved forward, keeping those challenges in all three categories.

Table 2: Round 2: Points of Introduction for Retained Challenges

Challenges	Elementary	Middle School	High School
Provide access to clean water	19 (73.1%)	4 (15.4%)	2 (7.7%)
Provide access to adequate nutrition	18 (69.2%)	8 (30.8%)	0 (0%)
Develop assistive technologies for the disabled	13 (50.0%)	6 (23.1%)	7 (26.9%)
Engineer the tools of scientific discovery	13 (50.0%)	9 (34.6%)	4 (15.4%)
Make solar energy economical a	9 (34.6%)	9 (34.6%)	8 (30.8%)
Advance personalized learning a	9 (34.6%)	9 (34.6%)	8 (30.8%)
Secure cyberspace	8 (30.8%)	10 (38.5%)	7 (26.9%)
Engineer better medicines	4 (15.4%)	13 (50.0%)	9 (34.6%)
Restore and improve urban infrastructure	7 (26.9%)	11 (42.3%)	8 (30.8%)
Advance nanotechnology	1 (3.8%)	9 (34.6%)	16 (61.5%)

Note. N = 26 panelists. ^a Recommendations were evenly dispersed for these two challenges, so they were kept at all three levels.

Validation of the categorization of challenges at developmental levels was pursued via a third Delphi round in which participants were asked to accept or reject each challenge at each level. Within developmental-level categories, participants were asked to rank order challenges from most to least important. Table 3 indicates rankings and acceptance rates for each developmental level.

Table 3: Round 3 Results: Ranks and Acceptance Rates of Retained Challenges for Elementary, Middle, and High School Levels

Challenges	Rank	M	SD	IQR	Mdn	Acceptance Rate
Elementary Level
Provide Access to Clean Water	1	1.4	0.63	1	1	100.00%
Engineer the Tools of Scientific Discovery	2	3.27	1.58	2.5	3	73.00%
Develop Assistive Technologies for the Disabled	3	3.53	1.46	2.75	3	100.00%
Provide Access to Adequate Nutrition	4	3.47	1.77	3	3	67.00%
Make Solar Energy Economical	5	4.07	1.03	1.75	4	73.00%
Advance Personalized Learning	6	5.27	1.03	1.75	6	53.00%
Middle School Level
Restore and Improve Urban Infrastructure	1	2.27	1.62	2	2	85.71%
Make Solar Energy Economical	2	2.73	1.44	1.75	2	100.00%
Secure Cyberspace	3	3.67	1.68	2.75	4	50%
Engineer Better Medicines	4	4	1.65	1.75	4	42.86%
Advance Personalized Learning	5	5.07	1.16	2	6	57.14%
High School Level
Advance Nanotechnology	1	2	1.25	2	1	85.71%
Make Solar Energy Economical	2	2.4	1.12	1.75	2	100.00%
Advance Personalized Learning	3	3.6	1.35	2	3	71.43%

Discussion and Conclusions

Participants in this study indicated a familiarity with and understanding of the Grand Challenges and were provided with links to relevant sites for reading prior to participation in Round 1. However, upon examination of the write-in responses, it became evident that an opportunity exists for deeper inquiry into, and likely professional development for, methodical learning about the Grand Challenges. One commonality among participants was the belief that technology education offers a suitable environment for the study of solar energy, and that it is accessible to students at all levels, K–12. However, the Grand Challenge associated with that concept is more specific: Make Solar Energy Economical . Claims of curricular alignment with this challenge do not consistently address the concept of solar energy from the perspective of economics and affordability. Therefore, in order to substantively address this challenge, technology lessons must be expanded to assume a broader and more global perspective. Opportunities for integrative STEM lessons are abundant.

Security and defense themed challenges, Prevent Nuclear Terror and Secure Cyberspace , were not deemed as important to K–12 technology education as other challenges were. One explanation for that might be that as issues of cybersecurity constantly evolve. Study participants may not have a deep understanding of how these concepts manifest in and affect society or how they could be broken down into manageable, developmentally appropriate, up-to-date lessons. Alternately, participants may not perceive cybersecurity or the prevention of nuclear terror as relevant to the pursuit of technological literacy. Concerns about the introduction of frightening content at an inappropriately young age may have also driven decision making. The rationale for such choices warrants further investigation. Technology educators and curriculum developers might look to programs already in place such as CyberPatriot, the National Youth Cyber Education Program originally conceived by the Air Force Association and presented by the Northrop Grumman Foundation "to inspire students toward careers in cybersecurity or other science, technology, engineering, and mathematics (STEM) disciplines critical to our nation's future" ( Air Force Association, 2015, para. 1 ). CyberPatriot currently serves middle school and high school students and is expanding its reach to serve elementary school children in the near future.

The challenges Provide Energy from Fusion, Develop Carbon Sequestration Methods, Manage the Nitrogen Cycle , and Reverse-Engineer the Brain were eliminated as appropriate for the technology curriculum at any level during the first round of this study. This could potentially be due to the perceived learner-level appropriateness or understanding of these challenge areas in the context of core literacies within technology and engineering education curricula. That said, Reverse-Engineer the Brain is reported to already be aligned with one EbD course at the 4 level, "Content is directly integrated into lessons in detail and assessed"; Manage the Nitrogen Cycle is aligned with three courses; Provide Energy from Fusion is aligned with two courses; and Develop Carbon Sequestration Methods is aligned with four courses ( ITEEA, 2012 ).

Engineering design experiences, and attendant curricula for technology education, reinforce core ideas and enact a process for the attainment of set benchmarks in a meaningful societal STEM context. Given the commitment that the field of technology education has made to integrate engineering at all levels, given the NDSS' emphasis on the interdependence of science, engineering, and technology, and given the recent large-scale display of support from schools of engineering for using the Grand Challenges to guide the design of their programs, it stands to reason that the field of technology and engineering education would serve students well through significant and explicit focus on the Grand Challenges in its work toward preparing technologically literate citizens.

---

Jennifer Buelin ( jkbuelin@iteea.org ) is Director of Curriculum, Professional Development, and Digital Initiatives for the ITEEA/STEM Center for Teaching and Learning; Aaron C. Clark ( aclark@ncsu.edu ) is Professor and Director of Graduate Programs for the Department of STEM Education at North Carolina State University; Jeremy V. Ernst ( jvernst@vt.edu ) is Associate Professor and Associate Director for the Office of Educational Research and Outreach at Virginia Tech.

¹ The Grand Challenge Prevent Nuclear Terror was eliminated in both question types during Round 2.

References

Air Force Association. (2015). What is CyberPatriot? Retrieved from http://www.uscyberpatriot.org/Pages/About/What-is-CyberPatriot.aspx.

Asunda, P. A. (2012). Standards for technological literacy and STEM education delivery through career and technical education programs. Journal of Technology Education, 23 (2),44–60. doi:10.21061/jte.v23i2.a.3

Clark, A. C., & Scales, A. Y. (2003). A Delphi study of assessment practices in engineering education. Engineering Design Graphics Journal, 67 (2), 21–32.

Clark, A. C., & Wenig, R. E. (1999). Identification of quality characteristics for technology education programs: A North Carolina case study. Journal of Technology Education, 11 (1), 16–26. doi:10.21061/jte.v11i1.a.2

Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique—A rotational modification. Journal of Vocational and Technical Education, 15 (2), 50–58. doi:10.21061/jcte.v15i2.702

Dalkey, N. C. (with Rourke, D. L., Lewis, R., & Snyder, D.). (1972). Studies in the quality of life: Delphi and decision-making . Lexington, MA: Lexington Books.

Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1986). Group techniques for program planning: A guide to nominal group and Delphi processes . Middleton, WI: Green Briar Press.

Engineering is Elementary. (2016a). About Engineering is Elementary. Retrieved from http://www.eie.org/about-engineering-is-elementary-EiE

Engineering is Elementary. (2016b). EiE curriculum units. Retrieved from http://www.eie.org/eie-curriculum/curriculum-units

Gattie, D. K., & Wicklein, R. C. (2007). Curricular value and instructional needs for infusing engineering design into K-12 technology education. Journal of Technology Education, 19 (1), 6–18. doi:10.21061/jte.v19i1.a.1

International Technology Education Association. (2007). Standards for technological literacy: Content for the study of technology (3rd ed.). Reston, VA: Author.

International Technology and Engineering Educators Association. (2012). EbD™ alignment with the grand challenges for engineering. Retrieved from http://www.iteea.org/EbD/Resources/GrandChallengesMatrix.pdf

International Technology and Engineering Educators Association. (2016). Engineering byDesign. Retrieved from http://www.iteea.org/EbD/ebd.htm

Katsioloudis, P. (2010). Identification of quality visual-based learning material for technology education. Journal of Industrial Teacher Education, 47 (1), 70–98. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v47n1/pdf/katsioloudis.pdf

Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications . Reading, MA: Addison-Wesley.

Meyer, M. A., & Booker, J. M. (1990). Eliciting and analyzing expert judgment: A practical guide . Washington, DC: Division of Systems Research, Office of Nuclear Regulatory Research, U.S. Nuclear Regulatory Commission. doi:10.1137/1.9780898718485

Moore, T. J., Glancy, A. W., Tank, K. M., Kersten, J. A., Smith, K. A., & Stohlmann, M. S. (2014). A framework for quality K-12 engineering education: Research and development. Journal of Pre-College Engineering Education Research, 4 (1), 1–13. doi:10.7771/2157-9288.1069

National Academy of Engineering. (2015, April 17). U.S. engineering schools to educate 20,000 students to meet grand challenges. Retrieved from http://www.engineeringchallenges.org/14373/15549/15678.aspx

National Academy of Engineering. (2016). 14 grand challenges for engineering in the 21st century. Retrieved from http://www.engineeringchallenges.org

National Academy of Engineering Grand Challenge K12 Partners Program. (2016). 5-part make it happen plans for the grand challenges. Retrieved from http://www.grandchallengek12.org/node/10

NGSS Lead States. (2013). Next generation science standards: For states, by states . Washington, DC: National Academies Press. doi:10.17226/18290

Paige, W. D., Dugger, J. C., & Wolansky, W. D. (1996). Essential components of doctoral programs for industrial technology education. Journal of Technological Studies, 22 (2), 15–20. doi:10.21061/jots.v22i2.a.3

Project Lead the Way. (2014). Project Lead the Way: History. Retrieved from https://www.pltw.org/about-us/history

Rose, M. A. (2010). EnviroTech: Enhancing environmental literacy and technology assessment skills. Journal of Technology Education, 22 (1), 43–57. doi:10.21061/jte.v22i1.a.3

Scott, D. G., Washer, B. A., & Wright, M. D. (2006). A Delphi study to identify recommended biotechnology competencies for first-year/initially certified technology education teachers. Journal of Technology Education, 17 (2), 43–55. doi:10.21061/jte.v17i2.a.4

Stitt-Gohdes, W. L., & Crews, T. B. (2004). The Delphi technique: A research strategy for career and technical education. Journal of Career and Technical Education, 20 (2), 55–67. doi:10.21061/jcte.v20i2.636

U.S. engineering school deans' response to President Obama on educating engineers to meet the grand challenges [Letter of commitment]. (2015, March 23). Retrieved from http://www.engineeringchallenges.org/File.aspx?id=15680&v=c29105cb

Volk, K. S. (1993). Curriculum development uses the Delphi technique. The Technology Teacher, 52 (4), 35–36

Wicklein, R.C. (1993). Identifying critical issues and problems in technology education using a modified-Delphi technique. Journal of Technology Education, 5 (1), 54–71. doi:10.21061/jte.v5i1.a.5

Wright, P. C., & Geroy, G. D. (1988). A review of qualitative data gathering methods and their applications to support organizational strategic planning processes (Study No. 6). University Park, PA: The Pennsylvania State University, Institute for Research in Training and Development.

Zargari, A., Campbell, M., & Savage, E. (1995). Determination of curriculum content and requirements for a Doctor of Philosophy degree program in industrial technology. Journal of Industrial Teacher Education, 32 (4), 57–73. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v32n4/zargari.html