Identifying Indicators Related to Constructs for Engineering Design Outcome

This study ranked constructs articulated by Childress and Rhodes (2008) and identified the key indicators for each construct as a starting point to explore what should be included on an instrument to measure the engineering design process and outcomes of students in high schools that use the PLTW and EbDTM curricula in Idaho. A case-study design was used. Data were collected in two stages. In the first stage, a content analysis was conducted for PLTW and EbDTM curricula to identify the indicators that are associated with the six constructs articulated by Childress and Rhodes (2008). In the second stage, the constructs and key indicators or concepts were placed on a survey and sent to experts for them to rate their importance for assessment and their difficulty to assess. Main findings included engineering and human values and the application of engineering design being ranked as first and second, respectively, for inclusion on an instrument to measure the engineering design process and outcomes. In addition, a total of 141 indicators were identified for all constructs. The indicators identified provide a useful list of measures that can be used by technology and engineering teachers. Selected indicators can be used by math, science, technology, and engineering education teachers as they coordinate in the teaching of STEM concepts and collaborate in the designing of project-based activities that they engage students in solving.


Introduction
Problem-based learning (PBL) promotes deep thinking and problemsolving skills (Woods, 1996).It has proven to be an effective way to learn subject knowledge, and in most PBL programs, "the goal is to empower the students with the task of creating the learning objectives that are important to them" (Woods, 2000, p. 2).Students are confronted with a scenario constructed around real-life problems, which by their nature are ill-structured, open-ended, and ambiguous, that launch students' inquiry as they collaborate to find solutions (Banks & Barlex, 2014;Woolfolk, 2013).Project-based learning is commonly used in technology and engineering education.Because project-based learning shares many of the instructional, multidisciplinary traits as PBL, the terms are often confused or used interchangeably (Honey, Pearson, & Schweingruber, 2014).
According to Banks and Barlex (2014), the difference between projectbased learning and PBL is that: PBL has tended to be a way of configuring the curriculum and relating what the students know to actual, real-world problems which in turn leads them to find out new knowledge and skills to bring to bear on the problem.Rather, project-based learning has been more about a pupil choosing an extended activity that they are interested in and using it as a vehicle for demonstrating their current capabilities, but also including demonstrating their abilities in researching and investigating new knowledge and acquiring skills as required.(p.141) Project-based learning can also be built on authentic, real-world situations or problems (Edström, Soderholm, & Knutson Wedel, 2007).As they work in groups, students are not restricted on where they may look for answers.In a review of the research on project-based learning, Thomas (2000) articulated five criteria that characterized projects: 1. "Projects are central, not peripheral to the curriculum," 2. "focused on questions or problems that 'drive' students to encounter (and struggle with) the central concepts and principles of a discipline," 3. "involve students in a constructive investigation" (p.3), 4. "student-driven to some significant degree," and 5. "realistic, not school-like" (p.4)."PBL incorporates real-life challenges where the focus is on authentic (not simulated) problems or questions and where solutions have the potential to be implemented (Gordon, 1998;as cited in Thomas, 2000, p. 4).
Assessment refers to the process of determining the extent to which students are achieving the intended learning outcomes (Gronlund, 1998).In PBL and project-based learning, assessment should emphasize problem solving, critical thinking, and reasoning skills.Creating problems that are similar to tasks accomplished in real life-authentic tasks-is a key principle for assessment used in both paradigms of instruction.However, assessment techniques that are repeatable are very challenging because of the subjective nature of PBL and project-based learning.In fact, McCracken and Waters (1997) believed that the requirement to have authentic tasks in problem solving conflicts with the requirement for assessments to be repeatable, because authentic tasks are themselves ill-structured and difficult to assess objectively.

Assessments in Technology and Engineering Education
The inclusion of engineering design as a part of the Standards for Technological Literacy has resulted in more curricula promoting engineering design activities.Engineering design is a systematic and often iterative approach to designing objects, processes, and systems to meet human needs and wants (National Research Council, 2012).The Standards for Technological Literacy define engineering design as "the systematic and creative application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems" (International Technology Education Association, 2007, p. 238).Engineering design problems are project-based activities during which students use the engineering design process to solve the problem while working in groups.
Traditional engineering education programs at the college level use a variety of methods to collect evidence that students are achieving intended learning outcomes.These include written and oral questions, performance ratings, product reviews, journals, portfolios, and other self-reports such as inventories and questionnaires.Written assessments include multiple-choice and other closed items, calculations, and open-ended questions.Oral questions, on the other hand, enable teachers to uncover students' misconceptions.They require students to think on their feet and speak coherently.Journals and portfolios provide records of students' individual and collaborative efforts in design projects."They reveal students' critical thinking and reasoning skills, and record the steps students followed in an engineering [design] process" (Gray, 2007, p. 161).Performance rating can be used to assess students' process and products in engineering design.Rating scales that define the degrees of quality along with rubrics (which are a list of the quality of a performance, process, or product) are used to assess the student.Self-report measures allow "students to reflect on their learning experiences" and help "them to see more clearly the connections among the concepts they have learned, as well as the applications of these concepts to new situations" (Gray, 2007, p. 161).
Addressing the infusing of engineering design at the K-12 level, some researchers have indicated that there are still areas in assessment that are open issues (Lewis, 2005;Kelley, 2008;Wicklein, 2005).Technology educators face these issues or challenges when they seek to implement engineering design into their curriculum.For example, the past few years have seen more school districts in Idaho adopting either the Engineering by Design (EbD TM ) or the Project Lead the Way curricula in their technology and engineering education programs.Some teachers and administrators, including the program director for the Technology and Engineering program in the State of Idaho, have expressed the need to explore having some assessment tool to measure engineering design outcomes that is repeatable, irrespective of whether the school uses the Engineering by Design (EbD TM ) curriculum or the Project Lead the Way curriculum (PLTW).Other teachers, including science and math teachers in some smaller school districts who also teach technology education, think that it would be helpful to have some instrument that guides them in their assessment.As a starting point to explore this issue, the authors decided to identify indicators to measure the constructs associated with engineering design process and outcomes particularly for Grades 9-12 level-the level at which the EbD TM and the PLTW curricula are primarily used in Idaho.

The Framework
The conceptual framework for this study drew on the work of Childress and Rhodes (2008) in which they examined engineering design content that should be taught in high school curricula.They articulated a framework to define the engineering design curriculum content.The seven categories were identified through a modified Delphi approach.They are: 1 In this research study, we used six of the seven categories.The seventh category, emerging fields of engineering, was not used because it related mainly to nanotechnology, which is not covered in the high school curriculum.

Purpose of the Study
The purpose of this study was to identify indicators for each of the constructs identified by Childress and Rhodes (2008) that can be used by STEM teachers in Idaho as a guide when they are assessing design outcomes of students in high schools, irrespective of whether the curriculum in use is EbD TM or PLTW.In addition, these indicators can provide researchers in STEM with items that can be used in the development of an instrument for assessment in engineering design at the high school level.The research questions that guided this study are: 1. How are the constructs identified by Childress and Rhodes (2008) ranked by professional engineers and educators in terms of criticality for inclusion on an instrument to measure engineering design outcomes in high schools in Idaho?
2. What are the key indicators associated with each of the constructs identified by Childress and Rhodes (2008) to measure engineering design outcomes in high schools in Idaho?
Research Design A case-study design was used.Case-study research involves the study of a case within a real-life, contemporary context or setting.It is a qualitative approach in which the investigator explores a real-life bounded case over time, using detailed data collection (Yin, 2009).A letter was sent to the program manager for the Technology and Engineering Education program requesting permission for the two schools' participation.

The Cases
Two cases were examined.One school in northern Idaho that uses the EbD TM curriculum and another school in southern Idaho that uses the PLTW curriculum.The school in northern Idaho had its own unique way of organizing and supplementing the EbD TM course material.The Fundamentals of Technology course is taught in Grade 9 and is only offered one semester.The technological design curriculum is taught in Grade 10 and covers topics such as career search, sketching, toy design (which the instructor uses for teaching shop safety, power tools, and finishing), Logo design concepts, mouse-trap cars, SolidWorks™ for bridge building, Co 2 cars, and an additional design problem.The curriculum emphasizes the engineering team concept and encourages creative design for all students.The Advanced Design Applications Class, taught in Grade 12, uses a material science curriculum developed by Energy Concepts Inc. that includes solid materials, metals, polymers, ceramics, and composites.The emphasis is on the importance of materials engineering to the manufacturing process.The engineering design courses included more SolidWorks™, robotics, and the VEX curriculum as well as total quality management.Each course requires the students to complete a project.
The school in southern Idaho uses the PLTW curriculum.Introduction to Engineering is taught in Grade 9 and focuses on the design process and its application.Principles of Engineering is taught in Grade 10 and introduces major concepts that students encounter in postsecondary engineering courses, such as mechanisms, statics, materials and kinematics.There are five specialization courses within PLTW: Aerospace Engineering (AE), Biotechnical Engineering (BE), Civil Engineering and Architecture (CEA), Computer Integrated Manufacturing (CIM), and Digital Electronics (DE).Digital Electronics and Aerospace Engineering are taught in Grade 11.Engineering Design and Development (EDD) is taught in Grade 12.This is the capstone course in which students work in teams to design and develop solutions to a problem by applying the engineering design process.

Procedure Data Collection
Data were collected in two stages.In the first stage, a content analysis was conducted for PLTW and EbD TM curricula to identify the indicators that are associated with the six constructs identified by Childress and Rhodes (2008).In the second stage, the constructs and key indicators or concepts were placed on a survey form and sent to experts for them to rate their importance for assessment and their difficulty to assess.

Content analysis
A qualitative content analysis of selected courses from the EbD TM and PLTW curricula used by each school was conducted to identify concepts of engineering design that were associated with the constructs identified by Childress and Rhodes (2008).These concepts are referred to as indicators in this study.Content analysis is a research tool in which researchers quantify and analyze the meanings and relationships of words and concepts within a text (Busch et al., 2012;Krippendorff, 2004).).Content analysis enables researchers to sift through large volumes of data in a systematic fashion with relative ease.It also allows inferences to be made that can then be corroborated using other methods of data collection.The curriculum materials that were analyzed from the PLTW and EbD curricula are displayed in Table 1 (below and continued on next page).Coding.Two coders assigned codes to the six constructs.The curricula were then examined to identify engineering design concepts and then categorized each of these concept under one or more of the constructs of Childress and Rhodes (2008).To ensure intercoder reliability, each coder was given a copy of Grade 10 curriculum materials for both PLTW and EbD TM .The researcher provided instructions to the coders prior to the coding process.The coders independently highlighted words and phrases relating to engineering design concepts that were in the Grade10 curriculum materials of both PLTW and EbD TM .The coders met to review and discuss their findings.Discrepancies were discussed and resolved.The process was repeated until an inter-coder reliability of 87% was obtained.Krippendorff (2004) indicated that in order to assure the data under consideration are at least similarly interpreted by two or more coders it is customary to require an intercoder reliability of 80% or more; therefore, the intercoder reliability for this study was well within acceptable levels.The coders then proceeded to perform a content analysis of the remaining sample of curriculum materials for both PLTW and EbD TM .

Construct Code
Engineering design that emphasizes the importance of creativity in designing engineered solutions to problems, as well as design iterations and tradeoffs Application of engineering design that included outcomes relating to design activities, experimentation, prototyping and reverse engineering Engineering analysis that includes mathematics in optimizing solutions and the use of both science and math in the engineering design process Engineering and human values that consists of the interactions between engineering design and society such as safety and the environment versus costs and ethics Engineering communication that included all sorts of communications important to the engineering design process Engineering science that includes the traditional sciences such as statics and dynamics as well as material properties, energy, power, etc.

ED-CIT ED-EPR ED-MSO ED-HV ED-C ED-ESD
After words relating to concepts of engineering design were identified, similar concepts were grouped together.The total number\ of words relating to engineering concepts that were identified by the coders amounted to 711, 618 of which were common to both coders.Some of the words were a derivative of the same word, so they were reduced into a final manageable, qualitative descriptive frequency list.This process was done by including the highest frequency word found within a group of similar words.For example, a group of words found by the coders were: communicate, communication, and communications.The final word selected was communication because it had the highest frequency.A part of the final frequency list of words is shown in Table 3.After the final frequency list was identified, the curriculum material was again examined by the coders to better understand the context in which the words were used and determine which of the constructs they were related to (Busch et al., 2012).Words that appropriately related to a construct were coded using the codes identified in Table 2. So, the constructs served as categories.Brief statements containing a verb, object, and sometimes a modifier were finally used to better capture the meaning of the concept or context in which it was used.These were indicators.For example, for the word communication, which was coded as ED-C, an examination of the meaning and context produced the statement Communicating knowledge professionally.

Survey
The survey instrument used was a modification of the Task Verification instruments used by Norton (1999).In the instrument Norton used, duty statements of a job or occupation are stated, and the task statements relating to each duty were listed below the duty statement.Expert workers were asked: (a) do they perform the task; (b) rate the importance of the task on a Likert scale of 0-5, with 0 meaning No Importance and 5 meaning Great Importance; and (c) rate the difficulty of a task on a Likert scale from 0-5, with 0 being Extremely Easy and 5 being Extremely Difficult.The criticality of each task was determined by multiplying the importance index by the difficulty index.
The instrument developed by the researchers replaced the duty statements with the six constructs of Childress and Rhodes (2008): • Engineering design, • Application of engineering design, • Engineering analysis, • Engineering communication, • Engineering and human values, and • Engineering science.
The indicator statements replaced the task statements on Norton's (1999) task verification instruments.The instrument asked expert participants to examine the indicators for each construct and rate each indicator on a 5-point Likert scale with 1 representing Strongly Disagree and 5 representing Strongly Agree for their (a) importance for assessment and (b) difficulty to assess engineering design process and outcomes.The criticality index for each indicator was determined by multiplying its importance score and its difficulty score.The criticality index for each construct was determined by multiplying the averaged importance score and the averaged difficulty score for the key indicators of that construct.
The survey was pilot tested by sending it to two teachers to fill out.Simple grammatical errors were corrected, and then it was sent to six experts.The experts were chosen for their experience in teaching engineering education in high school and at the college level and for practicing engineering in industry.The expert team consisted of two technology and engineering education teachers from two high schools in Idaho with combined years of teaching of over 30 years, two engineers from industry in Idaho with a combined working experience of 45 years, and two engineering education faculty from two universities with a combined experience of over 15 years in teaching and research.

Results
The results obtained from an analysis of the data are presented in respect to the two research questions posed at the beginning of the study.The first question was: How are the constructs identified by Childress and Rhodes (2008)

ranked by professional engineers and educators in terms of criticality for inclusion on an instrument to measure engineering design outcomes in high schools in
Idaho? Childress and Rhodes (2008) framework consisted of seven constructs, six of which were used for this study.The criticality index for each construct was derived by multiplying the indicators' average importance index by the average difficulty index.The constructs were then rank ordered from the highest criticality index to the lowest criticality index.As indicated in Table 4, engineering and human values had the highest criticality index and so was ranked one, and engineering science had the lowest criticality index and was ranked six.The second research questions was: What are the key indicators associated with each of the constructs identified by Childress and Rhodes (2008) to measure engineering design outcomes in high schools in Idaho?The category engineering and human values had six indicators (see Table 5).Five of the six indicators were rated high in importance, receiving scores ranging from 4.0 to 4.8.Three of the indicators were perceived to be difficult to assess.For the construct application of engineering design, 12 indicators were identified (see Table 6).Eleven of the 12 indicators were rated high in importance, receiving scores ranging from 4.0 to 4.8.The construct engineering communication had 20 indicators (see Table 7).Fifteen of the 20 indicators had importance ratings at 4.0 and above.Interestingly, the indicator utilizing brainstorming methods was scored 4.5 for importance but received 4.2 for difficulty to assess, the highest difficulty score for this construct.For the construct engineering design concepts, 16 indicators were identified from the content analysis (see Table 8).Eleven of the indicators had scores ranging from 4.0 to 4.8.Each of these 11 indicators had difficulty to assess, with scores ranging from 2.3 to 3.3, indicating they are not difficult to assess in class.The construct engineering analysis had 30 indicators (see Table 9).Thirteen of these indicators had scores ranging from 4.0 to 5.0.The indicator utilizing mathematics to solve problems received the highest importance score.The last category or construct on the instrument, engineering science, had 61 indicators (see Table 10, below and continued on next page).The importance rating data indicated that 20% of the indicators ranged at 4.0 or above, which means twelve of the 61 key indicators were rated high in importance for inclusion in an engineering design assessment tool.Six of the indicators were ranked below 3.0 for importance.Only three indicators were rated at 3.0 or above in their difficulty to assess.The two indicators that were scored as least difficult to assess were differentiating and calculating velocity and differentiate digital and analog systems.Assessments that are guided by indicators related to all six constructs should give technology and engineering educators that use the EbD TM and PLTW curricula in high school in Idaho a more holistic representation of students' performance in engineering design.Importantly, the list of indicators relating to each construct can also help to reinforce to math and science teachers the depth of students' immersive STEM experiences when their schools use the EbD TM and the PLTW curricula.This might motivate more collaboration across these disciplines.These indicators can also provide technology, math, and science education teachers with a list of items that can be included on performance rating forms to assess students' process and products in engineering design.Rating scales along with rubrics are used to assess students.Selected indicators can also be included on self-report measures that allow students to reflect on their learning experience and help them see the connections among the concepts that they learned as well as the applications of these concepts in new situations (Gray, 2007).It must be mentioned that many of these indicators can be broken down further into discrete actions that can provide useful measures of student's competency in a particular designing activity.In fact, the indicators that were viewed as difficult to assess (such as develop solutions to problems, understand attributes of a design process, and utilize brainstorming methods) may need to be broken down into more discrete action statements to provide clarity for assessment.

Conclusion
This study explored ranking engineering design constructs identified by Childress and Rhodes (2008) and identifying their indicators.The results represent preliminary work in addressing assessment of engineering design outcomes in schools in Idaho, irrespective of the curriculum in use.Admittedly, more questions still need to be answered.For example, can an instrument be developed from the indicators that validly and reliably assesses students' outcomes in design?What indicators should be included on such an instrument?More study needs to be done to answer these questions.The indicators identified for each construct in this study, however, provide a useful list of measures that can be used by technology and engineering teachers.Selected indicators can be identified by math, science, technology, and engineering education teachers as they coordinate in the teaching of STEM concepts and collaborate in the designing of project-based activities that they will engage students in solving.But, at present, the list provides a menu that teachers can choose from that relates to their instructional objectives, which they can use to assess students learning outcomes.

Table 1
Curriculum Materials Analyzed for PLTW and EbD

Table 3
Part of Final Frequency List

Table 4
Criticality Ranking of the Six Constructs

Table 5
Key Indicator Results for Engineering & Human Values

Table 6
Key Indicator Results for Application of Engineering Design

Table 7
Key Indicator Results for Engineering Communication

Table 8
Key Indicator Results for Engineering Design Concepts

Table 9
Key Indicator Results for Engineering Analysis

Table 10
Key Indicator Results for Engineering Science