Who Is Doing the Engineering, the Student or the Teacher? The Development and Use of a Rubric to Categorize Level of Design for the Elementary Classroom

Science, technology, engineering, and mathematics (STEM) professional development for K–5 teachers often includes engineering design as a focus. Because engineering applications provide perspective to both teachers and their students in terms of how mathematic and scientific principles are employed to solve real-world problems ( Baine, 2004 ; Roden, 1997 ), there is great interest in using engineering as a context for studying STEM education. Engineering as a context for learning mathematics and science is documented in the National Research Council’s review of K–12 engineering curricula ( National Research Council, 2010 ). Further, engineering has become integrated into the Next Generation Science Standards ( NGSS Lead States, 2013 ), which provide a mandate for the formal integration of engineering into the K–5 curriculum.

Although it may suggest fluidity in curriculum and instruction among the four disciplines, “the STEM acronym is more often used as shorthand for science and mathematics education” ( Katehi, Pearson, & Feder, 2009 , p. 12). The increased attention to engineering in elementary curriculum (e.g., the Next Generation Science Standards ), teacher preparation, and professional development provided the motivation for our research. Our project provided teachers with professional development opportunities designed to enhance their knowledge and preparation for teaching using engineering design. Following the professional development course, we observed how the teachers implemented engineering design lessons with students in their classrooms.

In recognition of the limited preparation of elementary level teachers to teach engineering content and pedagogy, we created and implemented a professional development opportunity for grade K–5 teachers to enhance their knowledge of engineering and the design process. Specifically, our collaboration sought to enhance the participating teachers’ understanding of the work of engineers. We also explored the procedures for engineering design as approaches to solving problems and conducting research while recognizing the developmentally appropriate application and use of these approaches for teaching STEM to elementary level learners.

Our STEM education intervention consisted of a three-day summer institute that combined presentations, workshops, hands-on activities, and curriculum planning and development. Our research was based on the anticipated influence of engineering-focused professional development on teacher practice and the subsequent increase in student engagement in engineering design-based learning activities ( Fox-Turnbull, 2006 ). Specifically, we examined the elements of the design process that teachers emphasized in their instruction and the student-generated artifacts inspired by the lessons, and we classified the design assignments by the extent of responsibility taken by the teacher and student in terms of the structure of the elements in the design process. We gathered empirical data detailing teacher knowledge of the design process, their instructional use of engineering design, and student response to design assignments at the elementary, K–5 level. Our report presents a new level of design classification rubric developed for categorizing levels of responsibility of the students and teachers in the design process. The rubric was designed to classify design lessons based on the student and teacher (or instructional resource) responsibility for decision making in determining the structure of the elements of design.

Engineering Design in Elementary Level Education

Engineering design is becoming more popular to include as a K–5 instructional approach, usually in conjunction with engineering, technology, and related contexts ( Davies, 1996 ; Lachapelle & Cunningham, 2007 ). Using design as a focus for instruction and learning, curricular programs such as Project Lead the Way ( Bottoms & Anthony, 2005 ) and Engineering is Elementary ( Cunningham & Hester, 2007 ) provide structure and materials for teaching engineering content, concepts, and processes. Additional outreach programs and a growing number of teacher professional development offerings (e.g., Nadelson, Seifert, & Moll, 2011 ) have been implemented to prepare elementary teachers to teach using design and engineering as contexts for teaching a range of STEM content. Nadelson and colleagues ( Nadelson et al., 2010 ; Nadelson et al., 2011 ) have established that such professional development offerings can increase teacher content knowledge and comfort with teaching engineering related content, concepts, and the design process.

The increased interest in supporting engineering in elementary teacher preparation and subsequent professional development suggests that there may be multiple justifications for providing continuing education opportunities designed to enhance teacher knowledge of engineering design ( Felder, Brent, & Prince, 2011 ; Guzey, Tank, Wang, Roehrig, & Moore, 2014 ; Lewis, 2006 ). We speculate that the increase in teacher knowledge of how to engage their students in the engineering design process happens most effectively by engaging teachers in engineering design projects that adhere to student-centered instructional practices.

The majority of the research on K–12 teacher professional development in engineering design has been directed toward secondary teachers ( Burghardt & Hacker, 2007 ; Fontenot, Talkmitt, Morse, Marcy, Chandler, & Stennett, 2009 ; Tufenkjian & Lipton, 2007 ). We argue that the increased awareness and consideration of engineering and design in the curriculum necessitates a greater understanding of how to prepare elementary teachers to effectively teach engineering design. Recognition of the potential for engineering design to provide learning contexts that are rich with opportunities to engage students in STEM habits of mind (e.g. problem solving, critical thinking, evidence based decision making—also see Berland, 2013) suggests that there is benefit to continued exploration of how design is and can be effectively taught in the K–5 curriculum. Thus, there are a number of anticipated benefits to preparing K–5 teachers to teach using design as well as a need to document how teachers are engaging their students in engineering design ( Lewis, 2006 ).

We recognize the need to increase teacher content and pedagogical knowledge associated with engineering design to enhance their capacity to effectively influence their students’ learning ( Darling-Hammond & Bransford, 2005 ). We also recognize the need to support the development of teacher pedagogical content knowledge ( Niess, 2005 ; Shulman, 1987 ), particularly in the teaching of engineering (Fransson, & Holmberg, 2012), and teacher creativity in their STEM teaching practice. Creative expression has been deemed necessary for teachers to generate the mental dexterity associated with being flexible, adaptable, and original in their practice ( Dobbins, 2009 ), which is likely critical when exploring new curricular areas such as teaching engineering design. Kampylis Berki, and Saariluoma (2009) argue that teachers influence their students’ development of creativity through modeling, which provides justification for enhancing and encouraging teachers to express their creativity in their teaching practice. Thus, a key element in providing teachers with experiences in design and engineering is encouraging them to think creatively and to reflect deeply on student-centered lessons. We assert that K–5 engineering professional development offerings need to expose teachers to design as a creative endeavor. This exposure is likely to foster teachers’ creative expression, increasing their instructional use of design and engaging students in a wide range of novel, student-centered engineering challenges

Classifying the Level of Instructional Use of Engineering Design

One of the challenges with researching the instructional use of engineering design is the wide range of possible implementation configurations, from very teacher centered to very student centered. In order to investigate this, we needed to develop a tool that would allow us to classify engineering lessons with respect to levels of responsibility for teacher and student. A similar situation with scientific inquiry motivated Schwab (1962) to develop a rubric to classify the level of inquiry with respect to teacher and student responsibility. Schwab’s rubric provides a means of classifying inquiry lessons based on the level to which the teacher or instructional resources are responsible for the structure of inquiry and the level to which students are responsible for the structure of the inquiry. In Schwab’s rubric, for a Level 0 inquiry assignment, teachers or instructional resources provide the structure for all elements of inquiry, and the students simply follow instructions as they engage in the inquiry process. Level 0 inquiry is essentially an exercise in confirming the results from prior research and established outcomes. In contrast, a Level 3 inquiry (based on Schwab’s rubric) engages students in investigations in which they are responsible for generating and responding to all aspects of inquiry, a situation equivalent to full discovery learning.

We were not able to locate a similar rubric or classification scheme that has been specifically developed to evaluate the responsibility level of students and teachers in engineering design instruction. We addressed this gap and developed and validated a rubric which can be used to classify the level of design used in engineering design instruction. In the development of our Level of Design Rubric we consulted several models of engineering design that are being promoted in the elementary engineering curriculum ( Cunningham, 2009 ; National Aeronautics and Space Administration [NASA], 2008 ; The Works Museum, 2011 ). Although each of these models was unique in the representation of engineering design, the models share some common elements or processes. Common to the models were identifying problems, exploring ideas, brainstorming, building products, gathering data, and evaluating the results. Further, the models of engineering design associated with elementary education also recognized the iterative nature of engineering. We combined the common elements presented in these models with our knowledge of the engineering design process, adding identification and listing of constraints and criteria to create a framework for the critical elements of engineering design that should be included in K–5 lessons (see Table 1). It is important to note that we present the essential elements here and rely on the details of the various other models explained in the literature for the finer gain details ( Cunningham, 2009 ; NASA, 2008 ; The Works Museum, 2011 ).

We used the elements in our engineering design model to guide the development of our Level of Design Rubric. Throughout the development process, we made decisions to collapse some design elements from the model to form the principal categories of our Level of Design Rubric. For example, we combined the engineering design processes of generating ideas and select a solution into one element, and we also combined the two processes of present results and evaluate outcomes into a single element. The essential elements contained within our engineering design instructional model and a brief description of the processes within each of the elements are presented in Table 1.

Table 1
Essential Elements of the Design Process Used in Instruction and the Associated Processes

	Problem Statement	Criteria and Constraints	Generate Ideas and Select Solution	Process Used to Build the Product	Present Results and Evaluate
Description of the Associated Process(s)	The problem to be solved is identified and explained	Criteria to which the solution must conform, and the specifications for the product are listed The constraints, limitations, or bounds for the product are recognized	Brainstorming about possible solutions to the problem Identifying what seems to be the best solution Justification and assurance that the preferred solution conforms to criteria and constraints	The solution is prototyped A solution is selected A working solution is created The solution is tested, data are gathered	The final solution is presented to others The solution is evaluated for conformity to criteria and constraints and effectiveness in solving the problem Evaluation is used to plan for the next generation of the solution

Similar to Schwab’s (1962) rubric for classifying the level of inquiry used in instruction, our rubric classifies the level of engineering design used in instruction by the level of responsibility assumed by students and the teacher for the structure of the engineering design elements. In the use of the Level of Design Rubric, each of the five design elements are considered and scored such that if a teacher (or the teacher-provided resources) provides all of the structure of the design element, the element would be scored as a 0. By contrast, if the student is responsible for all of the structure of the element, the element would be scored as a 1. If an element in our rubric is not present in a design assignment, such as presenting products and evaluating results” the element should be scored 0 because it is assumed that the teacher (or resource developer) made the decision not to include the design element process and, therefore, took full responsibility for that element of the design activity. We anticipate that students and the teacher (or resources) will share the responsibility for the structure of the design process elements, so fractional scores of 1 are encouraged. For example, an equal sharing of the responsibility for identifying and developing a problem statement by the teacher (or teacher-provided resources) and the student may result in a score of .5 for the problem statement element on the rubric. After each element has been scored with a value from 0 to1, the element scores are summed, and an overall Level of Design score is established. Thus, the final score for a level of design used for instruction evaluation and research on how elementary teachers (and other teachers) are structuring their engineering design lessons would be a value between 0 and 5.

Structure Responsibility Score 0 to 1 [If teacher or resources solely responsible—Score 0] [If student is solely responsible— Score 1]	Design Element					Level of Design Sum of Element Scores (From 0–5)
	Problem Statement	Criteria and Constraints	Generate Ideas and Select Solution	Process Used to Build the Product	Present Results and Evalute
Responsibility for Element Structure Score (from 0 - 1)

Figure 1. The Level of Design Rubric.

As with Schwab’s (1962) rubric, because level of design is somewhat subjective, we constructed our level of engineering design classification scheme to be used and interpreted on an ordinal scale based on what the observer focuses on during a lesson observation. The outcome from our rubric should be considered as a general indicator of the level of design. For example, a sum of scores of 2.75 may be rounded to “level 3” engineering design lesson which would indicate that the student assumed slightly more responsibility for the elements in the engineering design assignment than the teacher (or instructional resources). Thus, similar to those using Schwab’s rubric to classify instructional level of inquiry it is up to the discretion of the users of our Level of Design Rubric to apply their understanding of engineering education. The conditions of a design lesson that is under evaluation should be taken into consideration when interpreting the significance of the level of design which is observed.

To illustrate what different levels of the instructional implementation of design rankings might represent, we provide the following examples of design assignments that would be ranked from Level 0 to Level 5. The examples we provide detail the level of responsibility of the student, the teacher, and expected design assignment outcomes.

A Level 0 design would be equated with situations in which the student (through direct instruction by the teacher or instructional resources) is provided with all design element structure for all aspects of the design activity. Thus, a Level 0 design assignment would have a highly prescriptive structure with students following provided directions to replicate the construction of an existing solution or structure. In a Level 0 design assignment, the teacher (or instructional resources) is responsible for the structure of all design elements. A Level 1 design assignment would be structured such that students have some (but rather little) responsibility for the design elements structure and thus would be engaging in a design process that is likely to permit them to make slight modifications to an expected outcome or process. In a Level 1 design, the deviations the students may be permitted to take would be slight, and expectations would be for the students to essentially follow a detailed process that allows minimal decision making. At Level 2, the students assume some responsibility for the design process elements, which may allow them to make significant alternations toward the development of an expected outcome or product. A Level 2 design assignment would allow students to take responsibility for the design product as long as the spirit and outcome of the assignment design product is maintained. At Level 3, the students are given more responsibility for finding a solution to a problem and are afforded the opportunity to add new features and alternative solutions to a predetermined outcome. Thus, at Level 3, students have some freedom and responsibility to explore possibilities and seek unique solutions within the bounds of a proposed problem. At Level 4, the students assume a great deal of responsibility for the design process and develop new ideas and designs to solve a problem they have identified that is likely to lack a solution or model. At Level 4, students are approaching the work of professional engineers, but they are provided some structure and direction for their work. A Level 5 design assignment would place the full responsibility for all of the design elements on the students with minimal to no structure provided by the teacher (or instructional resources). Hence, in a Level 5 design assignment, the teacher’s role is almost exclusively shifted to that of a resource guide or consultant while the students take responsibility for seeking and identifying problems, designing and building unique solutions to meet the criteria and constraints that they identified, and evaluating, scrutinizing, and making modifications to their products to optimize their designs, as a professional engineer might do.

We do not advocate for a specific level of design but rather the right level of design for the corresponding level of instruction to meet student learning needs and developmental capacity. Similar to Schwab’s (1962) level of inquiry rubric, we envision our Level of Design Rubric as a way of determining the extent to which engineering design lessons are student centered or teacher centered. The degree of responsibility for a lesson interpreted in the context of a level provides a means of determining the appropriate structure of an engineering design lesson and the possible needs for curriculum and teacher professional development to assure an appropriate engineering design lesson structure. Further, the Level of Design Rubric provides a means of documenting how engineering design assignments evolve over time as students become more familiar with the elements of design.

Learning and the Level of Design

Some may argue that students are likely to learn more at higher levels of design because of the necessity for a greater degree of engagement due to higher levels of responsibility. However, we maintain that the instructional success of a design lesson is directly associated with student learning capacity; therefore, success is dependent on the experience of the students and the teacher with design, the context of the design assignment, and the prior content knowledge necessary to effectively engage in finding a solution to a problem. Further, we assert that there is no guarantee that student engagement in an assignment and associated learning will be positively correlated with the level of design; students may become highly engaged in lower level design or become disengaged with higher levels. The key to student engagement and learning may be the alignment between the level of design and the capacity and knowledge of students to effectively complete an engineering design challenge. Our Level of Design Rubric could provide a means of documenting factors associated with the alignment between level of responsibility and students engagement in an engineering design assignment.

We think it is wise for teachers to implement lower level design lessons when initially exposing their students to engineering design because students need to learn the processes. To help students learn the process, it is most effective if the teacher assumes a greater level of responsibility for the structure of the design and models or scaffolds the design elements for the students ( Bruning, Schraw, Norby, & Ronning, 2004 ). Thus, a lower level design may be necessary to help student learn the steps of the engineering design process. Similarly, we suggest that when teachers use engineering design to introduce new concepts or content, they should consider using lower level teacher-centered engineering design lessons. Removing the responsibility from the students for providing the structure for the engineering design elements lowers the cognitive demand on the students so that they may attend to learning the new concepts or content without also having to attend to the design process, which may reduce their capacity for learning new content ( Bruning et al., 2004 ).

As students gain a deeper understanding of engineering design and are given assignments in which they can elaborate on their prior knowledge, they are more likely to be able to take increased responsibility for design elements. Thus, we maintain that engineering design as an instructional approach is likely to be most effective and result in the greatest learning success when teachers scaffold and adjust the level of student responsibility for elements based on the students’ content knowledge and design experience. Our Level of Design Rubric could be used to document the evolution of lessons and determine how lessons are being structured in conjunction with other tools that document student learning or performance.

Methods

Research Questions

The goals of our research project were to document increases in teachers’ knowledge of engineering design and how teachers use the engineering design process to teach STEM content. Of particular interest to us were the engineering design elements that the teachers selected, how they structured the lessons they developed, and how the lessons were taught in terms of levels of responsibility. Thus, we recognized the need to develop and use a tool to document the level of design of the lessons that the teachers created and taught. We formed the following questions to guide our research:

• Did our participating teachers’ knowledge of engineering design increase due to participation in our summer institute, and if so, was the shift sustained over time?
• What was the challenge focus of the engineering design lessons that the teachers taught?
• What elements of engineering design did the teachers emphasize in their lessons?
• What was the level of engineering design of the observed lessons?
• How did the students engage in the engineering design lessons?

We speculated that our participating teachers would experience a sustained increase in their knowledge of engineering design due to their participation in our summer professional development institute and the follow-up support for teaching engineering design that they received during the school year. Further, we anticipated that our participating teachers would develop a diversity of creative engineering design lessons that utilized all the basic design elements. We predicted that we would find that the teachers had implemented engineering design lessons in a range of levels of design. Based on the structure and context of engineering design lessons, we speculated that the students would be highly engaged in the design lessons.

Participants. Our data were drawn from observations of the 142 K–5 elementary teachers who voluntarily participated in our STEM-focused professional development project. The participants all worked in the same school district in one of six partnering elementary schools. The mean age for teachers was 40.7 years old (S = 10.2) with 10.5 years of teaching experience (S = 7.4). Ninety percent were female. The participants had completed an average of 3.6 college level mathematics classes and 3.2 college level science classes. Eighty-four percent declared a major endorsement in elementary education with other relevant major endorsements including biology or life science (3%), physical science (1%), and mathematics (1%). The participants’ teaching assignments were nearly equally distributed among Kindergarten to Grade 5.

During the three years of our professional development program, the participants voluntarily engaged in a free, three-day intensive summer institute for which they received continuing education credit. For the small minority of cases in which teachers did not attend the professional development program, we provided an equivalent professional development opportunity for them at their schools. In addition, the participants received mentoring and follow-up professional development during the academic year. Because our summer institute was not focused specifically on engineering and design but rather on STEM in general, not all of our summer institute participants are included in this report of our research project. Thus, our current report of design lesson structure and content is based on observation data collected on our project participants who were teaching engineering design lessons on the days we conducted classroom observations.

STEM Professional Development Summer Institute

Our three-day professional development summer institute was designed to enhance the participating K–5 teachers’ knowledge of STEM content as well as their use of scientific inquiry and engineering design as instructional approaches. The institute theme and content were focused on exploring the processes used by STEM professionals in their work as contexts for teaching and learning in K–5 education. A portion of the institute was dedicated to exploring the use of plastic brick manipulatives (Lego®-like bricks marketed as PCS BrickLab ® by PCS Edventures!) for teaching STEM. Our primary goal was to provide a curriculum to our participants that allowed them to gain a deeper understanding of the best practices used to teach STEM and, in particular, to increase their knowledge and capacity to teach engineering design.

During the summer institute, we engaged the teacher participants in a number of engineering design activities, including making the most efficient paper helicopter, building the tallest structure possible on an inclined plane, and creating a “lander” that, when released from a third floor balcony, was to slowly descend to a target on the ground floor ( Carpinelli, Kimmel, & Rockland, 2014 ). These and other activities provided explicit instruction and models of engineering design and provided many opportunities for the participants to think about and plan for how to teach engineering design. Further, to enhance participants’ STEM pedagogical content knowledge, we explicitly compared and contrasted the goals and processes of inquiry and design and encouraged the teachers to think about creative and unique solutions and possible opportunities for implementation of engineering design within their curriculum.

Following the summer institute, the participants were expected to develop and implement four lessons that utilized the plastic brick manipulatives to teach lessons from each of the four STEM domains. Teachers were encouraged to get creative, to adapt and adopt extant lessons, and to develop interesting and engaging STEM learning opportunities for their students. We observed the participants at least twice during the academic year while they were teaching a STEM lesson (approximately 30–60 minutes in length) to their students using the plastic brick manipulatives. We provided the teachers with observation data and feedback, extending beyond the summer institute, in order to situate the professional development in the classroom.

Data Collection

Design Process Knowledge. To assess our participants’ knowledge of engineering design, we adapted and adopted items from an extant instrument, the Design Process Knowledge Test, which had been validated for undergraduate engineering majors ( Sims-Knight, Upchurch, & Fortier, 2006 ). The original instrument used a selected response format to assess engineering design knowledge across a range of related concepts. Because of the difference in our study population (K–5 teachers) as compared with the original instrument targeted population (undergraduate engineering majors), we determined it necessary to screen the items in the instrument and select only those that were aligned with general knowledge of engineering design and remove items associated with idiosyncratic engineering definitions, engineering coursework, or engineering degree program structures or activities. The resulting instrument contained 18 items such as, “Which of these is the best definition of engineering design?” and “Which is not a benefit of preliminary design or prototype?” and alternatives that represented a range of possible views from naïve or misconceived to informed. Our version of the instrument also included several items such as, “Successful design involves breaking a problem down into smaller problems” which required responses along a Likert-type scale ranging from 1 ( almost always true ) to 5( almost always false ), including I don’t know . In the original study, Sims-Knight, Upchurch, and Fortier (2006) report a Cronbach’s alpha of .84, indicating a good level of instrument reliability. We gave participants our modified version of the knowledge of design instrument as a pre and posttest.

Classroom Observations. Following the summer institute, we conducted classroom observations of the participants to determine the extent to which they utilized the summer institute concepts to teach STEM content. We used an observation rubric that we successfully used in prior research ( Nadelson et al., 2010 ) to document our observations. The observation rubric was structured to assure that data collection was consistent in that the same kinds of data were collected during each observation. The observation rubric was also structured to be flexible enough to document variations in the nature of the learning and instruction that took place that we could associate with our summer institute content. Each lesson was observed in its entirety, most of which were about 50 minutes in duration. It was our goal to observe each participant teach two STEM lessons, but it was up to the teacher to select what area of STEM lesson was taught and the STEM pedagogy that was used in the lesson. Thus, the STEM content area under observation was not restricted to an engineering design lesson but could be from any of the STEM domains.

During the observations, our field researchers did not participate in the instruction, but they may have interacted with the students by asking the children to explain what they were doing, which allowed for the accurate completion of the observation rubric. We also utilized Livescribe Smartpens® to audio record the observed lessons to provide an additional means of accurately documenting the lessons for the completion of the observation rubric. The audio recordings were not retained once the observation rubrics were completed.

In some circumstances, video recordings were made of the students’ interactions while building and explaining their products. We were authorized to gather the video as long as student names, faces, or other information was not gathered that would allow the students to be readily identifiable. These video recordings were also useful for completing the observation rubrics.

Classification of the Level of Design. We utilized our Level of Design Rubric to document the engineering design elements that the teachers emphasized in their lessons and to quantify the level of engineering design. Thus, we used our Level of Design Rubric to document the level of responsibility that the teachers (or teacher-provided instructional resources) and students took for the elements associated with an observed engineering design lesson. Again, the Level of Design Rubric score is interpreted as a basis for documenting the source of the responsibility for element structure in a design activity. If an overall level of design score is near 0, indicating a low level of design, it is because most (or all) of the responsibility for the design structure was provided by the teacher (or instructional resources). In contrast, if a level of design score is near 5, indicating a high level of design, a major amount (or all) of the responsibility for the structure for the design activity was assumed by the student.

To establish the inter-rater reliability of the instrument we had two researchers independently score the lessons. The level of agreement was approximately 85%, with differences resolved though conversation. Thus, we were confidence that the 85% level of inter-rater reliability was acceptable and that our data could be examined without concerns of consistency or bias.

Results

Teacher Knowledge of Design

Our first research question asked: Did our participating teachers’ knowledge of design increase due to participation in our summer institute and if so was the shift sustained over time? Before we conducted our analysis, we calculated the reliability of our modified design process instrument which we found to have a Cronbach’s Alpha of .78, indicating that we had an acceptable level of instrument reliability and could proceed with our analysis with the assumption of consistent measures of our participants’ knowledge of design. To answer our research question, we conducted a paired samples t-test using our participants’ pretest, immediate posttest, and delayed posttest scores. Our paired samples t-test analysis revealed a significant increase in design knowledge t (46) = 4.94, p < .01, with the pre-institute composite scores 10.17 (S = 3.60) shifting upward post-institute to 12.91 (S = 2.07), a .35 partial eta squared effect size. Our analysis did not reveal a significant change in design knowledge from posttest to delayed posttest. Our results indicate that our participants experienced significant and sustained gains in their knowledge of the engineering design process. This finding led us to consider how the participating teachers transferred their knowledge of and experience with the design process into their instruction and interactions with their students.

Lesson Content

Our second research question asked: What was the challenge focus of the engineering lessons that the teachers taught? To answer this question, we conducted a content analysis of our 169 STEM lesson observations, 36 of which were categorized as engineering design lessons that included a design challenge. The balance of the lessons focused on other content areas such as mathematics or science. Our content analysis of the observed lessons revealed the teachers implemented an array of design challenges, representing a diversity of creative expression and focusing on a range of topics. For example, in one lesson, students were challenged to design a container to hold the largest number of Silly Bandz given certain budget constraints. Our analysis exposed a number of design lessons in which students were presented with challenges of building model structures such as houses and bridges. For example, in one lesson, the students were challenged to construct a bridge that could withstand a simulated lateral motion earthquake. In another bridge activity, the students were instructed to build a bridge that could span two desks. Thus, our observation data analysis revealed that the teachers utilized a range of topics and ideas to engage the students in design challenges and were creative in their selection and organization of their lessons.

Our analysis also revealed that several teachers implemented lessons focused on developing models of existing objects or structures. For example, we observed several lessons in which the teacher instructed the students to “design” and build an element from nature, such as a flower, which we interpreted as the process of constructing a model of a flower. Similarly, in another lesson, students were instructed to design a model of the life stages of a pumpkin. The modeling lessons raised an unanticipated situation, provoking questions as to how to classify the lessons and whether it was necessary to distinguish between engineering design lessons and modeling lessons in our analysis. The overlap between the problem-solving strategies and the stages of development (elements) that the teachers or students used in the model construction and analysis and the engineering design process provided us with justification for grouping the modeling lessons with the engineering design lessons. The grouping of the modeling and design lessons is particularly defensible when examining the structure of the lessons. The observed instructional steps that teachers implemented in their modeling lessons were essentially the same as those that we observed in the teachers implementing design lessons. However, we also explicitly recognized that the teachers in our project mixed the processes of model building and engineering design and creatively utilized the elements of design as pedagogical approaches for a variety of lesson orientations.

Emphasized Elements

Our third research question asked: What elements of engineering design did the teachers emphasize in their lessons? To answer this question, we did a content analysis of our classroom observations, seeking evidence of the elements of the design process that were emphasized in the lessons. Our analysis revealed that all of the lessons included the problem statement, generate ideas and select solution , and process used to build product elements . The establishing criteria and/or constraints element and the present results and evaluate were present in only approximately half of the observed lessons, yet, our analysis of the lessons by design and modeling did not reveal significant shifts in the presence of the elements. Thus, our analysis indicates that teachers consistently placed emphasis on some design elements but were less uniform in their implementation of other design elements.

Level of Design

Our fourth research question asked: What was the level of design of the observed lessons? To answer this question, we used our Level of Design Rubric to analyze the observations and determine the level of design used on the lessons. We analyzed all 36 of the observed design lessons and scored the observations (see Table 2). We used the criteria that we established in our introduction of the rubric, using 0 for instances when the teacher (or instructional resources) provided the element structure and 1 when the students were responsible for all the structure with values between 0 and 1 corresponding to mixed levels of responsibility. We then summed the outcomes of the analysis of the individual lessons and calculated the average to produce an overall mean (and standard deviation) for each of the elements. In addition, we identified the maximum and minimum values observed.

Table 2
Average Level of Design Scoring of Our 36 Design Lessons

Structure Responsibility Score 0 to 1 [If teacher or resources solely responsible—Score 0] [If student is solely responsible— Score 1]	Design Element					Level of Design Sum of Element Scores (From 0–5)
	Problem Statement	Criteria and Constraints	Generate Ideas and Select Solution	Process Used to Build the Product	Present Results and Evalute
Responsibility for Element Structure Score (from 0 - 1)	.69	.53	.27	.99	.21	2.68
Average Element Score (SD) [min, max]	.69 (.14) [.3, 1]	.53 (.21) [0, 1]	.27 (.27) [0, .8]	.99 (.08) [.5, 1]	.21 (.30) [0, 1]	2,68 (.64) [1.5, 3.7]

Our level of design analysis revealed that, on average, there was a distributed range of level of element structure responsibility. For the problem statement , the teachers (or instructional resources), on average, took less responsibility, while the students assumed more responsibility. Our analysis indicated that there was a nearly equally shared responsibility of the structure of the constraints and criteria element. In terms of the level of the generate idea and select solution element, our analysis revealed that, on average, the teachers (or instructional resources) assumed a large part of the responsibility for providing the structure of the element. We found a similar outcome for the present results and evaluate element. For the process used to build the product, our analysis revealed that the students almost exclusively assumed all responsibility for this element in the design processes.

Overall, our analysis revealed that, on average, the level of design (M = 2.68, S = .64) documented in the observed lessons indicated a nearly equally shared responsibility by the teachers and students for establishing the structure of the design activities. The range of level of design structure of the lessons fell between a low of 1.5 and a high of 3.7. The range of design structure indicates that shared responsibility was present in all the observed lessons. Overall, the observed lessons extended beyond a comprehensively teacher (or resource) structured level of design. Similarly, there were no instances in which the structure for design was comprehensively the responsibility of the students. Thus, our observations revealed that, on average, the teachers implemented design lessons sharing the responsibility for the structure of the engineering design elements with their students as the students engaged to design challenges.

Student Engagement

Our final research question asked: How did the students engage in the design lessons? To answer this question, we again did a content analysis of our classroom observation data and reviewed the video recordings taken during the implementation of the design lessons. Our analysis revealed high levels of student motivation and engagement. Beyond their excitement about engaging in the building of solutions, students were also eager to develop and refine their products. Many of the observations recorded eagerness by the students to explore possibilities and make modifications to enhance or optimize their products, a critical aspect of design that was not necessarily an explicit instructional aspect of the lessons. Our data also revealed high levels of pride and willingness to share and explain their products. In some situations, the students did wander off task and decided to engage in their own activities, which interestingly was more common in the lower level design lessons in which students were not required to provide the structure for the design elements.

We have provided several video clips that typify the high levels of student engagement in design lessons at the end of our article. These clips provide insight into how the students explained and evaluated their products and, in some instances, suggested or made modifications to optimize their designs.

Discussion and Implications

With the increased emphasis on engineering as part of the K–5 STEM curriculum, as promoted in the Next Generation Science Standards ( NRC, 2013 ), there is a need to enhance teacher capacity to teach engineering design based lessons ( Lewis, 2006 ; Wilson, 2013 ). Our results indicate that a rather brief professional development intervention with follow-up classroom support can significantly influence teacher knowledge of the engineering design process. Our results are promising because they suggest that teachers can gain lasting knowledge of engineering design with relatively brief but appropriately structured interventions. We attribute the increase in our participants’ knowledge to the format of our summer institute or the comparable in-school professional development and subsequent follow-up sessions, which placed the teachers in situations where they actively interacted in design challenges in the context of the classroom. The design activities conducted during the summer institute provided the teachers with both knowledge of the design process and an instructional model for implementing an engineering design lesson. We contend that the active engagement in design activities or equivalent is instrumental to increasing teachers’ understanding of design while enhancing their pedagogical knowledge of how to use design in teaching. Thus, modeling and engaging teachers in design activities appears to be a very effective way to increase both their procedural and content knowledge of engineering design and, therefore, their preparation to teach engineering design lessons.

The wide range of lessons that we observed suggests that our participating teachers were willing to seek out new ideas and creatively implement engineering design lessons. We find their willingness encouraging because it suggests that there is potentially a wide range of teacher initiated, highly engaging design lessons that will be developmentally, cognitively, and instructionally aligned for K–5 students. Our exposure of a possible conflation of modeling and engineering design suggests that if we want teachers to understand the distinction between modeling and design, we may need to be more explicit about the difference in the preparation of teachers to teach using design. It may also be possible that the teachers are creatively adopting and adapting design lessons and are not concerned with adhering to the engineering professionals’ conception of design but rather engaging their students in active learning. Teacher feedback about using design and their conceptions of design as compared to modeling is an excellent direction for future research.

Our analysis of the level of emphasis that teachers placed on the design elements revealed a range of attention to the processes. As we reviewed the observations, it became apparent there was consistent emphasis placed on the problem statement, generate ideas and select solution, and process used to build the product elements. We speculate that these elements were consistently emphasized because they are closely aligned to project-based learning instruction, a pedagogical practice that most teachers are familiar with or use in their practice. We also exposed a reduced level of emphasis on the criteria and constraints and present results and evaluate elements. The lack of emphasis on these components may be due to the lack of experience of the teachers with using design in instruction and the importance of criteria, constraints, and evaluation to the design process. Perhaps additional reinforcement of these elements in the professional development program would result in a greater emphasis in the instruction. It may also be that these are the most nebulous and cognitively demanding elements of design and that teachers do not feel the students are developmentally prepared to address these as issues in their learning. Further, our field observers noted that the impact of time constraints of the classroom often seems to be involved in teacher determination of lesson flow and structure. Thus, the time intensive nature of these elements may deter teachers from implementing these aspects of design. We speculate that teachers may have chosen to focus on certain aspects of the design process that allow students to develop STEM habits of mind and were not necessarily concerned with students’ mastery of the design process as a whole. Again, why teachers emphasize certain elements and how the students’ abilities may influence their decisions for implementing design lessons is an excellent direction for future research.

As we applied our rubric to classify the level of design of the observed lessons, we found that, on average, the students and teachers (or instructional resources) equally shared responsibility for the structure of the activity. We speculate that the teachers structured their design assignments to engage students in the process while maintaining significant responsibility for the structure to assure that the students were successful at completing the task. Thus, it is likely that the teachers did not think that their students were ready to engage in the highest levels of design because of lack of experience with the process. It may also be possible that the teachers’ lack of experience in teaching using design constrained their ability to create design assignments that could give students most or all of the responsibility for the structure of the design activity. Regardless, we conjecture that as teachers’ experience in teaching using design increases, they will shift to implementing higher level design activities. It is important to note that our classroom observers and the teachers they observed did not use or even have knowledge of the design elements in the rubric; the data was extracted post-classroom observation. Using the rubric during observations may enhance our ability to expose conditions that could be useful in explaining why teachers choose the structure level for their students. Further, the data may also provide insight into what conditions might be in place that prompts teachers to develop higher level design assignments.

The students’ enthusiasm for engaging in the design activities suggests that engineering design is an effective instructional format for motivating students to be actively involved in their learning. In our observations, it became apparent that design activities provide an excellent mechanism for creating rich learning context for engaging students in problem solving and thinking. Further, it appears that well-organized lessons are very effective in engaging students in the design activity and focusing their attention. Perhaps it is due to the students need for order, but student focus and attention to task was not correlated with the level of design or grade level. Thus, when using design as a context for instruction, it is critical that the activities be orchestrated, paced, and guided in an organized manner to maximize their potential to influence learning.

In our review of learner engagement data we uncovered what appeared to be an innate desire of the students to refine and modify their builds to optimize their products. The motivation and interest of the students is very encouraging and should be capitalized upon when considering approaches to instruction that most effectively enhance student learning. We assert that students’ innate desire to modify their builds and optimize their products indicates that students are eager and capable of taking the responsibility for structuring the present results and evaluate design element. Ironically, our data also indicated that the teacher (or resources) provided the majority of the structure for the present results and evaluate design element. Thus, modification to design instruction to enhance student development of problem-solving and critical-thinking skills may be easily achieved by shifting a greater level of responsibility for the structure of the present results and evaluate design element by allowing students to critically evaluate their work. The outcome of a shift to a greater level of student responsibility for the structure of the present results and evaluate design element is an excellent direction for future investigation.

We also speculate that student engagement in modifications to their builds can be attributed to the plastic brick manipulatives (BrickLabs®) that they were using in the lesson which allow for a low “cost of change,” meaning that the nature of the bricks encourages students to add to, subtract from, or change the shape of their builds, making bricks an ideal platform for teaching design. Further, the bricks can be assembled into an array of configurations making them attractive as an instructional tool to address a wide range of curriculum. The presence of the bricks in the classrooms may also encourage teachers to be more exploratory and creative with design curriculum and instruction and encourage their students to consider alternatives in their builds. How student engagement is linked to the nature of the instructional materials such as BrickLabs® and how the materials influence student desires to evaluate and modify their builds is an excellent direction for future research. Further exploration to expose additional manipulatives that encourage similar levels of engagement in design would also be another fruitful avenue of investigation.

Limitations

Although we observed each of our participating teachers twice during the academic year, we had no specific criteria for the STEM lessons that they were to teach during the observations. As we pointed out, there may be a conflation of teacher understanding of modeling and design. Yet, our data do not necessarily expose the relationship between teachers’ focus on modeling as a design activity and their understanding and perceptions of engineering design. Additional observations along with interviews of teachers teaching modeling lesson may resolve this limitation and lead to a greater understanding of the teachers’ choices of lessons and understanding of design and modeling.

Similarly, only about a quarter of our STEM lesson observations were engineering design lessons. Thus, the other three quarters of the participants may have taught design lessons from a different STEM perspective. Yet, the 36 lessons we observed were not specifically selected because they were engineering design lessons but occurred in an arbitrary fashion such that we predict the observed lessons were likely representative of the study population. Additional observations and interviews to confirm our prediction would be an excellent direction for future research.

As with all observation protocols, our rubric has limitations. Although we have refined our observation protocol over a three -year period it is still limited in terms of the data it is designed to collect. Thus, there may be other activities and nuanced variations in our participants’ design lesson implementation that we are not explicitly capturing. However, we did use an observation tool with established consistency and inter-rater reliability to collect our observational data, which should have minimized possible variations in the perceptions of research personnel. Perhaps video recording the lessons and then reviewing them several times may expose subtle variations in observation data, which is an excellent direction for future research. The challenge will be gaining access to public school classrooms to gather video data for such research.

Finally, as previously mentioned, we applied our Level of Design Rubric to the observation data that was gathered using a different protocol and instrument. Although the observers were able to substantiate our scoring and assessment of the observation data based on their experience, the data were not collected using the Level of Design Rubric. We were also using our general observation rubric to capture data from an array of STEM lessons because the STEM content of the observed lessons was not known prior to the observations. The application of our elements table and Level of Design Rubric in observations of design lessons is one of the goals of our continued research.

Conclusions

As engineering design grows as an instructional approach for teaching a range of STEM content it is critical that we develop a framework for preparing teachers to teach the content and the tools to examine the implementations. In our research project, we explored the effectiveness of a teacher professional development offering intended to enhance teacher capacity to teach engineering design. We developed and used some tools for classifying the elements of design and the level of the design in terms of teacher and student level of responsibility for the structure of the design lessons. We hope others will find our work useful as they plan professional development offerings for K–5 teachers and then study teacher implementation of engineering design in the curriculum and student engagement in the design activities. In addition, we hope that those who use our Level of Design Rubric will provide us with feedback as we seek to refine and increase the accuracy of our tool for evaluating the instructional use of design.

Acknowledgements

The Idaho SySTEMic Solution is supported by a grant from the U.S. Department of Education Fund for Improvement of Education Program, Award # U215100034. However, the contents of this paper do not necessarily represent the views of the Department of Education. The authors also recognize and appreciate the enthusiasm and support of the teachers and administrators in the Meridian Joint School District in Meridian, Idaho.

---

Louis S. Nadelson ( louis.nadelson@usu.edu ) is Associate Professor of Education at Utah State University. Joshua Pfiester ( jpfiester@daltonstate.edu ) is Assistant Professor of Education at Dalton State College. Janet Callahan ( janetcallahan@boisestate.edu ) is Professor of Materials Science and Engineering and Associate Dean at Boise State University. Patricia Pyke is Director of Research Development at Boise State University.

References

Baine, C. (2004). Is there an engineer inside you? A Comprehensive guide to career decisions in engineering (2nd ed.). Belmont, CA: Professional Publications.

Berland, L. K. (2013). Designing for STEM integration. Journal of Pre-College Engineering Education Research (J-PEER), 3 (1), 22–31. Retrieved from http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1078&context=jpeer

Bottoms, G., & Anthony, K. (2005). Project lead the way: A preengineering curriculum that works (Research Brief). Atlanta, GA: Southern Regional Education Board. Retrieved from http://publications.sreb.org/2005/05V08_Research_PLTW.pdf

Bruning, R. H., Schraw, G. J., Norby, M. M., & Ronning, R. R. (2004). Cognitive psychology and instruction (4th ed.). Upper Saddle River, NJ: Prentice Hall.

Burghardt, D., & Hacker, M. (2007). Engineering professional development . Paper presented at the National Symposium to Explore Effective Practices for Professional Development of K–12 Engineering and Technology Teachers, Dallas, TX. Retrieved from http://hub.mspnet.org/media/data/BurghardtHacker.pdf?media_000000006150.pdf

Carpinelli, J., Kimmel, H., & Rockland, R. (2014). Paper drop design competition . Denver, CO: Regents of the University of Colorado. Retrieved from https://www.teachengineering.org/view_activity.php?url=collection/cub_/activities/cub_paper/cub_paper_activity1.xml

Cunningham, C. (2009). Engineering is elementary . The Bridge, 39(3), 11–17. Retrieved from https://www.nae.edu/Publications/Bridge/16145/16170.aspx

Cunningham, C. M., & Hester, K. (2007). Engineering is elementary: An engineering and technology curriculum for children . Paper presented at the American Society for Engineering Education Annual Conference and Exposition, Honolulu, HI. Retrieved from http://eie.org/sites/default/files/research_article/research_file/ac2007full8.pdf

Darling-Hammond, L., & Bransford, J. (Eds.). (2005). Preparing teachers for a changing world: What teachers should learn and be able to do . San Francisco, CA: Jossey-Bass.

Davies, D. (1996). Professional design and primary children. International Journal of Technology and Design Education, 6 (1), 45–59. doi:10.1007/BF00571072

Dobbins, K. (2009). Teacher creativity within the current education system: A case study of the perceptions of primary teachers. Education 3-13, 37 (2), 95–104. doi:10.1080/03004270802012632

English, L. D., & Mousoulides, N. (2009). Integrating engineering education within the elementary and middle school mathematics curriculum. In B. Sriraman, V. Freiman, & N. Lirette-Pitre (Eds.), Interdisciplinarity, creativity, and learning: Mathematics with literature, paradoxes, history, technology, and modeling (pp. 165–176). Charlotte, NC: Information Age Publishing.

Felder, R. M., Brent, R., & Prince, M. J. (2011). Engineering instructional development: Programs, best practices, and recommendations. Journal of Engineering Education, 100 (1), 89–122. doi:10.1002/j.2168-9830.2011.tb00005.x

Fontenot, D., Talkmitt, S., Morse, A., Marcy, B., Chandler, J., & Stennett, B. (2009). Providing an engineering design model for secondary teachers . Paper presented at the Frontiers in Education Conference, San Antonio, TX. doi:10.1109/FIE.2009.5350738

Fox-Turnbull, W. (2006). The influences of teacher knowledge and authentic formative assessment on student learning in technology education. International Journal of Technology and Design Education, 16 (1), 53–77. doi:10.1007/s10798-005-2109-1

Guzey, S. S., Tank, K., Wang, H.-H., Roehrig, G., & Moore, T. (2014). A High‐Quality Professional Development for Teachers of Grades 3–6 for Implementing Engineering into Classrooms. School Science and Mathematics, 114 (3), 139–149. doi:10.1111/ssm.12061

Kampylis, P., Berki, E., & Saariluoma, P. (2009). In-service and prospective teachers’ conceptions of creativity. Thinking Skills and Creativity, 4 (1), 15–29. doi:10.1016/j.tsc.2008.10.001

Katehi, L., Pearson, G., & Feder, M. (Eds.). (2009). Engineering in K–12 education: Understanding the status and improving the prospects . Washington, DC: National Academies Press.

Lachapelle, C. P., & Cunningham, C. M. (2007). Engineering is elementary: Children's changing understandings of science and engineering . Paper presented at the American Society for Engineering Education Annual Conference and Exposition, Honolulu, HI. Retrieved from http://eie.org/sites/default/files/research_article/research_file/ac2007full9.pdf

Lewis, T. (2006). Design and inquiry: Bases for an accommodation between science and technology education in the curriculum? Journal of Research in Science Teaching, 43 (3), 255–281. doi:10.1002/tea.20111

Nadelson, L., Callahan, J., Pyke, P., Hay, A., & Schrader, C. (2009). A SySTEMic solution: Elementary teacher preparation in STEM expertise and engineering awareness . Paper presented at the American Society for Engineering Education Annual Conference and Exhibition, Austin, TX. Retrieved from http://search.asee.org/search/fetch?url=file%3A%2F%2Flocalhost%2FE%3A%2Fsearch%2Fconference%2F19%2FAC%25202009Full939.pdf&index=conference_papers&space=129746797203605791716676178&type=application%2Fpdf&charset=

Nadelson, L., Callahan, J, Pyke, P., Hay, A., & Schrader, C. (2010). Teaching inquiry-based STEM in the elementary grades using manipulatives: A SySTEMic solution report . Paper presented at the American Society for Engineering Education Annual Conference & Exhibition, Louisville, KY. Retrieved from http://search.asee.org/search/fetch?url=file%3A%2F%2Flocalhost%2FE%3A%2Fsearch%2Fconference%2F32%2FAC%25202010Full1218.pdf&index=conference_papers&space=129746797203605791716676178&type=application%2Fpdf&charset=

Nadelson, L. S., Pyke, P., Callahan, J., Hay, A., Pfiester, J., & Emmet, M. A. (2011). Connecting science with engineering: Using inquiry and design in a teacher professional development course . Paper presented at the American Society of Engineering Education Annual Conference, Vancouver, BC, Canada. Retrieved from http://www.asee.org/public/conferences/1/papers/441/view

Nadelson, L. S., Moll, A. J., & Seifert, A. L. (2011). Living in a materials world: Materials science engineering professional development for K-12 educators . Paper presented at the American Society of Engineering Education Annual Conference, Vancouver, BC, Canada. Retrieved from http://www.asee.org/public/conferences/1/papers/2432/view

National Aeronautics and Space Administration. (2008) Elementary school standards-based engineering design process . Retrieved from http://www.nasa.gov/audience/foreducators/plantgrowth/reference/Eng_Design_K4.html

National Research Council. (2010). Rising above the gathering storm, revisited: Rapidly approaching category 5 . Washington, DC: National Academies Press.

NGSS Lead States. (2013). Next generation science standards: For states,by states . Washington, DC: National Academies Press.

Niess, M. L. (2005). Preparing teachers to teach science and mathematics with technology: Developing a technology pedagogical content knowledge. Teaching and Teacher Education, 21 (5), 509–523. doi:10.1016/j.tate.2005.03.006

Roden, C. (1997). Young children's problem-solving in design and technology: Towards a taxonomy of strategies. Journal of Design & Technology Education, 2 (1), 14–19. Retrieved from http://ojs.lboro.ac.uk/ojs/index.php/JDTE/article/view/375

Schwab, J. J. (1962). The teaching of science as inquiry. In J. J. Schwab & P. F. Brandwein (Eds.), The teaching of science (pp. 3–103). Cambridge, MA: Harvard University Press.

Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard educational review, 57 (1), 1–23.

Sims-Knight, J., Upchurch, R., & Fortier, P. (2006). Developing Assessments That Help Students Learn to Develop Open-ended Designs. In D. Deeds & B. Callen (Eds.), Proceedings of the National STEM Assessment Conference (pp. 274–289), Washington, DC. Springfield, MO: Drury University. Retrieved from http://openwatermedia.com/downloads/STEM%28for-posting%29.pdf

The Works Museum. (2011). Engineering design process . Retrieved from https://theworks.org/wordpress/wp-content/uploads/2013/03/Engineering-Design-Process.pdf

Tufenkjian, M,. & Lipton, E. (2007). A professional development model to infuse engineering design content into the high school curriculum . Paper presented at the American Society for Engineering Education Annual Conference, Honolulu, HI. Retrieved from http://search.asee.org/search/fetch?url=file%3A%2F%2Flocalhost%2FE%3A%2Fsearch%2Fconference%2F14%2FAC%25202007Full2518.pdf&index=conference_papers&space=129746797203605791716676178&type=application%2Fpdf&charset=

Wilson, S. M. (2013, April 19). Professional development for science teachers. Science, 340 (6130), 310–313. doi:10.1126/science.1230725