Two Approaches to Engineering Design:
Observations in sTEm Education

Todd R. Kelley, Ph. D
Purdue University

Mr. Daniel C. Brenner
Bartlett High School

Mr. Jon T. Pieper
Purdue University

Abstract

A comparative study was conducted to compare two approaches to engineering design curriculum across different schools (inter-school) and across two curricula Project Lead the Way and Engineering Projects in Community Service (inter-curricula). The researchers collected curricula material including handouts, lesson plans, guides, presentation files, design descriptions, problem statements, and support guides. The researchers conducted observations in the classrooms to collect qualitative indicators of engineering/technology reasoning, collect data on the nature of students’ questions, how students define problems, and operate within the constraints of a design problem.

Observational protocol studies were conducted on students participating in Project Lead the Way curriculum programs and with students participating in Engineering Projects in Community Service (EPICS). Students were asked to work through an ill-defined problem, in this case the problem of creating a new playground for an elementary school. The data from these protocols were analyzed using a coding process; a list of universal technical mental processes ( Halfin,1973 ) and a computer program OPTEMP, ( Hill,1997 ) to accurately record frequency and time of each mental process employed by the students. The data from the protocol results were used to identify common cognitive strategies employed by the students to determine where these students placed greatest emphasis throughout the observational protocol.

General findings indicated that participants in the EPICS-High program were more solution-driven problem solvers, where the Project Lead the Way participants were generally problem-driven as defined by Kruger & Cross (2006) . Additionally, the participants in both groups had completed advanced courses in mathematics, very little mathematics was employed (less than 3%) to describe constraints of the problem or predict results of proposed solutions. Over half of the students became fixated at some point on the provided picture. ( Smith; Ward; & Schumacher, 1993 ). This study provides important insight about how students solve ill-defined problems, providing vital information for technology education as it seeks to implement engineering design.

Introduction

The purpose of this study was to better understand the current status of engineering-focused curriculum programs at the high school level and its greater impact on student learning. Specifically, this study examined the implementation of Project Lead The Way (PLTW) and EPICS High, engineering-focused curriculum and its impact on students’ abilities to employ the engineering design processes using mathematical and scientific reasoning to solve engineering design problems; a construct of interest to K-12 engineering education and the greater STEM community.

The rise in engagement of secondary engineering education ( Douglas, Iversen, & Kalyandurg, 2004 ) and the increase in development of engineering-focused curriculum for grades 9-12 ( Dearing & Daugherty, 2004 ) provide strong rationale for examining pre-existing secondary engineering-focused programs to evaluate the impact on student learning.

Researchers in technology education have investigated the effects of technology education instruction on performance in mathematics and science for over a decade ( Childress, 1996 ; Merrill, 2001 ). Findings indicated that engineering drawing and design courses have had a significant impact on student achievement on mathematics standardized end-of-year tests ( Dyer, Reed, & Berry, 2006 ). Another study discovered that engineering design instruction at the middle school level reduced the performance gaps among certain ethnic groups and increased student’s conceptual knowledge of science, while simultaneously generating higher order thinking skills such as analysis and synthesis ( Cantrell, Pekcan, Itani, &Velasquez-Bryant, 2006 ). These studies indicate that there is a growing interest in investigating how engineering-focused instruction impacts student’s STEM learning.

In addition, several initiatives have started in the last decade to introduce engineering principles and engineering reasoning into grade 9-12 classrooms ( Brophy, Klein, Portsmore, & Rogers, 2008 ). While PLTW and other curriculum programs like EPICS-High focus on different aspects, from engineering problem solving to service learning; each of these new curriculums share the lack of a cohesive body of systematic and rigorous research studies investigating the effectiveness of their programmatic efforts. Specifically, there is a lack of investigation on higher order thinking, open-ended problem solving and the development of design and engineering knowledge/skills ( Chaker, 2008 ).

Dyer, Reed, and Berry (2006) recommended more research on the instruction of engineering design to accurately determine what elements of instruction are most beneficial to student’s learning of math and science. Similarly, Sheppard, Pellegrino, & Olds (2008) and Brophy et al. (2008) call for more research on engineering efforts in high schools. This becomes especially pressing, since individual states develop specific engineering standards (i.e. Massachusetts) and specific teacher licensures will likely follow. As these programs continue to grow, there is a need to build a strong base of rigorous research to provide educated and specific feedback on how to improve existing curricula and build a cohesive research agenda on engineering reasoning development in the K-12 grade spectrum.

Participants

Project Lead The Way (PLTW) is a national pre-engineering program that has been implemented into a number of high school and middle schools. The Project Lead The Way program launched in 1997 based on previous work in the 1980s by Richard Blais at Shenendehowa Central School ( Blais & Adelson, 1998 ). Since 1997, Project Lead The Way has grown with over 1300 schools and 175,000 students ( Bottoms & Anthony, 2005 ). Currently, there are over 280 high schools in the state of Indiana alone implementing curricula modules of Project Lead The Way. Two PLTW schools were selected in northwestern Indiana, these schools were selected because they had successfully implemented PLTW at the high school level and had similar student demographics.

Engineering Projects in Community Service (EPICS) is a national engineering-centered, academic service-learning program initiated at Purdue University ( Coyle, Jamieson, & Oakes, 2005 ). The main tenets of service learning are 1) curricular connections; 2) reflection; 3) community partnerships; 4) authentic, complex and ill-structured problems; 5) addressing real needs; and 6) performance-based assessment ( Honnet & Poulsen, 1989 ). Multidisciplinary team partnerships are formed with local not-for-profit organizations to define, design, build, test, deploy, and support engineering-centered projects that significantly improve the organization’s ability to serve the community. EPICS-High (for grades 9-12) began in 2007 bringing the engineering design concept into the high school environment. The program has quickly grown to 32 active high schools across the nation with over 650 students served, 50% female students, 48% minority students, and over 50% in free and reduced lunch programs. Two EPIC High programs were selected in northwestern Indiana and were chosen by their record of success in implementing the EPICS High program. Every effort was made to select these participating schools that align with the student body demographics.

Table 1. Participating School General Student Demographics

	PLTW 1	PLTW 2	EPICS 1	EPICS 2
Enrollment	883	1606	1833	883
Graduation Rate	75.7%	92.2%	83.8%	75.7%
White	72%	88%	86%	72%
Hispanic	25%	5%	7%	25%
Multicultural	2%	3%	2%	2%
Asian	1%	2%	1%	1%
Black	1%	2%	3%	1%
Native American	1%	0%	1%	1%
Surrounding Area	Rural	Urban Fringe	Urban Fringe	Rural

Research Questions

This research was guided by the following research questions. Since the nature of our research is comparative, the result to each of the questions will be compared across different schools (inter-school) and across curricula (inter-curricula):

1. What are the most common elements within student dialogues as they define engineering, engage in student collaboration and class discussions when seeking to solve engineering design problems? Which attributes or elements of engineering are missing or strongly represented?
2. What are common cognitive and meta-cognitive strategies employed by high school students as they work to solve an engineering design focused problem?
3. What elements in the engineering design problems (or in the curriculum unit) encourage teachers to engage mathematical and science curricula elements and support mathematical reasoning or scientific reasoning of their students?

Methodology

The research team collected curricula material from all participating classrooms to analyze. This included: textbooks, handouts, lesson plans, PowerPoint files, design descriptions, problem statements, and support guides. The researchers conducted observations in the classrooms to collect qualitative indicators of engineering/technology reasoning, collect data on the nature of students’ questions, how students define problems, and how they operate within the constraints of an engineering design problem.

Next, a protocol analysis session was conducted with a group of three student volunteers from each site. Each volunteer was given the same design problem (the transfer problem), see Appendix A. Each student was asked how they would proceed from the given problem statement in order to improve the current condition described in the statement. The students were asked to define the problem, list all constraints that they impose on this problem, and describe how he or she would proceed to solve the problem. The participants were asked to verbalize their thoughts as they worked through the ill-defined problem (verbal protocol methodology). The researchers prompted participants to keep talking through the problem. The testing sessions were limited to thirty minutes, most lasted around ten to fifteen minutes. Certainly, in the time constraint of a testing session, a student would be unlikely to reach the final stages of the design process. However, one of the most important stages of the engineering design process occurs at the onset of a technical problem: ‘framing the problem’ is this important stage. Experts in the field of design identify that framing the problem is a critical step to the design process which occurs as soon as the designer is presented with a technical problem ( Dym, Agogino, Eris, Frey, & Leifer, 2005 ; SchÖn, 1983 ). The transfer problem was developed to share characteristics of an ill-defined, complex, and dynamic design problem ( Jonassen, 2000 ).

Analysis

The project involves two separate analyses of artifacts, (1) curricula materials, and (2) results from selected students transfer problem analysis. Both forms of data were analyzed with an analytical induction framework ( Bogdan & Biklen, 1992 ) that similarly to the constant comparative method of grounded theory ( Strauss & Corbin, 1990 ), combines data analysis, further literature review, and theory-building in a cyclical manner. In the analytical induction approach, data built the basis for further descriptions and interpretations, and in contrast to grounded theory is informed by prior research.

As students progressed through the transfer problem session, the students’ cognitive processes were identified and coded from a list of 17 universal mental processes ( Halfin (1973) . A computer analysis tool called the Observation Procedure for Technology Education Mental Processes (OPTEMP) ( Hill, 1997 ) was utilized to capture, record, and organize the codes from each transfer problem session. The researchers coded the actions and cognitive processes used by each student participant as he or she worked through the engineering design problem.

Findings

Classroom Observations:

A total of three class observations were conducted at each PLTW and EPICS-High site. Each PLTW and EPICS High class session was 90 minutes for a total of 270 minutes of observation at each site.

Project Lead the Way Observations

The course observed was PLTW Principles of Engineering . The PLTW students were working on marble sorting activity during the observation sessions, the activity required student teams to design and build a device to sort marbles by color using Fishertechnik parts. The nature of much of the classroom dialogs were regarding Fishertechnik parts (motors, sensors, structures, etc) and how the students were planning to connect the various parts to make the device function properly. However, as the student teams’ device began to be tested, malfunctions often occurred. As a result of these device malfunctions; students worked as cohesive teams to troubleshoot various problems encountered. Student dialogs were healthy problem solving conversations and represented cooperative team efforts. However, students did not appear to be using a systematic approach to solve problems. The students approach to solving problems was reactive as they employed trial and error approaches, which might also be described as “tinkering”. Students often were observed reconfiguring the device parts such as wires, motors, and sensors.

EPICS High Observations

Both EPICS High participating schools were observed for a total of three class observations. One important observation about both EPICS High classes was the use of an engineer’s notebook. Both EPICS High programs required students to obtain and keep an engineer’s notebook for the course. The EPICS High School #1 class was discussing “great teams” during one of the classroom observations and “community” during another class observation. The community discussion was an EPICS High created lesson titled Your community…My community. The instructor led a class discussion by asking questions like “How big is your community? Where does it begin and end?” The instructor encouraged students to think about how they individually define community, what did community mean to them. Later in the class, the discussion led to the brainstorming of community service projects to be explored by the class. One researcher had a conversation with the EPICS #1 instructor who shared that his school started the EPICS High program to purposefully target the middle level academically performing students instead of targeting the high performing students. The course gave students an alternative to PLTW which the school also offers. The Purdue EPICS program provided a $48,000 grant to provide the EPICS school #1 with new laptops for all the students to use in the class. The grant also allowed the school to travel to Aurora, IL to the Walter Payton Brew House roundhouse. Last year’s EPICS High project was a feasibility study to restore the train roundhouse that is located in the community. EPICS High school #1 class is 12 weeks long.

One classroom observation at EPICS High #2 school found small groups of students (two groups of two students) discussing with the instructor their EPICS High community projects. The instructor reminded the students that they needed to be recording their design thinking in the design notebooks and how the information was to be recorded. The instructor asked the students to present their design project ideas informally to the group. One group selected a community project to design and build a walking path for students on school grounds, the path would need to be wheelchair accessible. The design team was talking about making a pathway using some sort of gravel. The design discussion lead to brainstorming the different material they could use so the wheel chairs won’t get caught. The other design team selected a project to create a storage container that would allow teachers to use a magnetic key fob to lock and unlock their laptop computers in a storage unit. Many teachers at this school have complained about having to carry their laptops everywhere they go for security reasons instead of being able to just leave it in their classrooms or one central location. The EPICS High #2 class was small and as a result was a very informal classroom structure, however there was good classroom dialogue regarding the design process. The instructor emphasized the importance of creating a clear and concise problem statement and recording design work in the engineer’s notebook. The EPICS High #2 instructor was new to the EPICS High program and had just completed EPICS training in the summer of 2009. The EPICS High #2 school program runs the EPICS High course as a club and not an official class; students do not get course credit and meet after school hours.

Curriculum Documents

A review of PLTW curriculum documents for the course Principles of Engineering revealed a focus on teaching students about systems, subsystems, open and closed loop systems, basic computer programming, and troubleshooting problems within a system. Later in the semester, students also learned about basic statics and dynamics. Furthermore, students were taught how to use free body diagrams and conduct basic vector analysis. This knowledge would have been helpful for students who participated in the transfer problem. However, this instruction came after the transfer problem testing session. Upon review of the Principles of Engineering curriculum, the researchers identify that the students are exposed to a limited scope of the engineering design process. PLTW curriculum program consists of a series of courses in pre-engineering; therefore it is unfair to fully assess students design capabilities upon their completion of just one course. Assessment of students who have experienced all the PLTW courses would be ideal such as in the PLTW capstone course: Engineering Design and Development . This course was studied in a similar protocol analysis study ( Kelley, 2008 ). However, the schools in this study did not offer this course. Furthermore, most of the students used in this study had taken Introduction to Engineering Design that provided a broader overview of the engineering design process so students should have been prepared to employ the engineering design process to solve the transfer problem.

The EPIC High curriculum documents that were studied were lesson plan documents taken from EPICS High Curriculum Module 1- Design, week 3. The activities in this module focus on design notebooks. One activity required students to assess their current note-taking skills by examining their notebooks they keep for other classes. The activity asks students to brainstorm in a small group how the design notebook differs from traditional class notebooks.

Other similar class activities included making use of other people’s design notes, building note taking skills, and building organizing skills for design notebooks. Lessons for later in the course focused on the topic of solving engineering problems in teams. These lessons required students get into small or large design teams and work to solve some ill-defined problems. One problem was as follows:

“In your group, solve the following design problem:

• Landfill space is rapidly running out. Develop a plan to eliminate your city’s dependence on the local landfill. Imagine, for now, that the city’s population is 100,000.
• As you solve this problem, each member should take notes according to your design notebook guidelines.”

(EPICS High curriculum Module 1- Design lesson plans, p.29).

When students were finished exploring possible solutions, they were required to compile a design record providing a design rationale including experimental measurements, sketches, safety precautions, design criterion rationale. Students were also asked to consider assessing how they functioned as a design team during the assignment. This exercise in open-ending problem solving would provide ideal experiences for students and should prepare students for the transfer problem, see Appendix A.

Ill-defined Transfer Problem Session:

Table 2. Transfer Problem Participants’ Coursework

Participant	Grade level	Math Courses	Science Courses	Technology Courses
Math Courses Key: A1- Algebra; A2- Algebra II; G- Geometry; S- Statistics; T- Trigonometry; PC- Pre-Calculus; C- calculus. Science Course Key: Ana- anatomy; Bio- Biology; Chem- Chemistry; ICP- Integrated chemistry and Physics; Phy- Physics. PLTW Course Key- IED- Introduction to Engineering Design; POE- Principles of Engineering
PLTW 1	10	A1, G	ICP	AutoCAD, POE
PLTW 2	10	A1, A2	Biology, Physics	IED, POE
PLTW 3	10	A1, A2, G	Bio, Chem	IED, POE
PLTW 4	10	G, A-AP		IED, POE
PLTW 5	12	G, A2, T, PC, C		IED, POE, Bio-Tech
PLTW 6	11	G 1, G2, A3-A4, T, PC	Bio, Chem, Phys	IED, Design
EPICS 1	10	A1, G	Bio, ICP	EPICS, IED
EPICS 2	12	A1	Bio, Zoology, ICP	EPICS, Manufacturing, Communications, Construction, Robotics, Machining
EPICS 3	12	A2, G, PC, S	Bio, Chem	IED, EPICS
EPICS 4	12	A1, A2, G, PC, C	Bio, Chem, Phy, AP Phy	IED, POE, EPICS
EPICS 5	12	A1, A2, G, PC, C	Bio, Chem, Phy, Chem2	Manufacturing, Transportation
EPICS 6	12	A1, A2, G, PC,C	Bio, Chem, Phy, Chem2	EPICS

Project Lead The Way Transfer Problem Results

There were several similarities in the Project Lead The Way participant groups when comparing their employed cognitive processes during the protocol session. Out of the seventeen cognitive processes that were first identified by Halfin (1973) , the participants only used eight of the processes, see Table 3. Due to testing session constraints such as time limits, location, and available resources, it was expected that cognitive processes such as models/prototypes, measures , and testing as well as other cognitive processes would not be used when developing a design solution. On average the PLTW students used the analysis (AN) cognitive process 37.6 % of the time with a range of low 18% to a high of 61%. The design (DE) cognitive process was used 36.5% of the time with a low 21% and a high of 53% of the time. These were the two most employed cognitive processes by the PLTW participants. Besides the quantitative data that was collected, there were several similarities in the ways that the students went about solving the problem.

Of the six PLTW students that were given this ill-defined problem, there was distinctive common pattern when comparing the two female participant results, see Figure 1.1 & 1.2 and Table 4 & 5, participants #2 & #5 were females. Although the research design did not call for comparison of participants’ design thinking by gender, the frequency of the cognitive processes of the female students were similar, where as the male students varied. Although these results are interesting, the researchers acknowledge that a greater sample size (n) would be necessary to generalize the finding.

Table 4. Project Lead The Way School #1 Transfer Problem Results

	PLTW #1		PLTW #2		PLTW #3		PLTW School 1 Mean Scores
Halfin Code	Freq.	Time	Freq.	Time	Freq.	Time	Freq.	Time
AN	10	01:38.4	13	05:48.5	11	07:40.3	11.3	05:02.4
CO	0	00:00.0	1	00:08.4	0	00:00.0	0.3	00:02.8
DE	8	03:00.7	8	03:14.2	8	02:39.7	8.0	02:58.2
DF	6	02:00.0	1	00:44.1	1	01:02.8	2.7	01:15.6
ID	1	00:03.8	0	00:00.0	2	00:11.6	1.0	00:05.1
MA	0	00:00.0	1	00:02.8	2	00:03.4	1.0	00:02.1
MO	8	02:12.4	1	00:13.6	1	00:31.1	3.3	00:59.0
PR	2	00:09.5	3	00:25.1	4	00:21.4	3.0	00:18.7
QH	0	00:00.0	2	00:15.0	1	00:01.5	1.0	00:05.5

Table 5. Project Lead The Way School #2 Transfer Problem Results

	PLTW #4		PLTW #5		PLTW #6		PLTW School 2 Mean Scores
Halfin Code	Freq.	Time	Freq.	Time	Freq.	Time	Freq.	Time
AN	15	02:56.2	34	11:17.1	11	04:19.3	20.0	06:10.9
CO	3	00:27.9	2	00:09.6	0	00:00.0	1.7	00:12.5
DE	17	06:47.2	32	08:06.5	21	09:07.5	23.3	08:00.4
DF	2	01:21.1	3	00:59.2	2	00:58.6	2.3	01:06.3
ID	5	00:38.6	8	00:51.8	0	00:00.0	4.3	00:30.1
MA	2	00:09.0	3	00:21.8	1	00:12.7	2.0	00:14.5
MO	8	02:08.7	6	00:50.6	14	02:00.3	9.3	01:39.9
PR	2	00:05.6	8	00:38.9	1	00:03.9	3.7	00:16.1
QH	0	00:00.0	0	00:00.0	2	00:32.6	0.7	00:10.9

EPICS High Transfer Problem Results

With the exception of EPICS High participant #3, all other EPICS High students spent 1/3 or more of their time designing solutions with a high of 46 % time on (DE) Designing for EPICS participant # 6 to a low of 20% EPICS participant #3. On average, each EPICS High participant used seven of the nine Halfin coded cognitive strategies that were employed. In general the EPICS High students not only employed multiple cognitive strategies but also spent an ample percentage of time employing a variety of strategies. EPICS #1 and EPICS # 6 are the two female participants. See figures 1.3 & 1.4 and Tables 6 & 7.

Table 6. EPICS High School #1 Transfer Problem Results

	EPICS #1		EPICS #2		EPICS #3		EPICS School 1 Mean Scores
Halfin Code	Freq.	Time	Freq.	Time	Freq.	Time	Freq.	Time
AN	4	02:36.4	9	01:52.1	14	04:08.8	9.0	02:52.4
CO	0	00:00.0	1	00:04.5	1	00:18.9	0.7	00:07.8
DE	5	02:26.6	14	03:31.8	14	02:53.0	11.0	02:57.1
DF	1	01:01.2	4	01:39.1	7	01:48.5	4.0	01:29.6
ID	0	00:00.0	2	00:11.4	6	00:57.9	2.7	00:23.1
MA	1	00:04.5	2	00:05.9	1	00:08.9	1.3	00:06.4
MO	3	00:44.4	3	00:32.8	12	03:40.0	6.0	01:39.1
PR	0	00:00.0	4	00:49.9	1	00:12.7	1.7	00:20.9
QH	4	00:49.4	1	00:22.3	0	00:00.0	1.7	00:23.9

Table 7. EPICS High School #2 Transfer Problem Results

	EPICS #4		EPICS #5		EPICS #6		EPICS School 2 Mean Scores
Halfin Code	Freq.	Time	Freq.	Time	Freq.	Time	Freq.	Time
AN	6	00:40.9	5	00:45.4	5	01:21.6	5.3	00:56.0
CO	1	00:03.5	1	00:04.4	0	00:00.0	0.7	00:02.6
DE	10	02:34.4	7	01:57.8	8	02:40.8	8.3	02:24.3
DF	3	01:20.3	1	00:49.4	2	01:07.1	2.0	01:05.6
ID	0	00:00.0	0	00.00.0	0	00:00.0	0.0	00:00.0
MA	1	00:04.7	1	00:13.3	0	00:00.0	0.7	00:06.0
MO	4	00:32.8	3	00:27.0	6	01:13.7	2.7	00:44.5
PR	3	00:15.1	3	00:13.0	4	00:06.0	3.3	00:11.4
QH	0	00:00.0	2	00:13.8	1	00:36.4	1.0	00:16.7

PLTW results compared with EPICS High results

The results of the EPICS High transfer problem observational protocol sessions reveal that in general the EPIC High students used a greater variety of cognitive capabilities when designing a solution to the transfer problem compared with PLTW transfer problem results. Clearly, DE (Designing) and AN (Analysis) were the most often employed cognitive strategies for both groups, EPICS High participants employed a variety of other cognitive strategies for an ample amount of time, possibly indicating that the EPICS High students engaged in a more well-rounded approach to design by employing multiple cognitive strategies. Both groups with the exception of one participant in each program (PLTW #3 and EPICS #3) employed 1/3 or more percentage of their time on the transfer problem in DE (Designing). This is a very promising result because it indicates that the students were successfully able to move from the problem space to the solution space. However, 4 out of the 6 PLTW participants focused 40% or more of their time on (AN) Analyzing. One EPICS High student spent 34% of time on (AN) Analyzing, all other EPICS High participants dedicated less than 30% of their time using Analyzing as a cognitive strategy. It is important to note that (AN) Analysis Halfin code was recorded by the researchers when participants were observed breaking down the problem and identifying constraints and criteria. (AN) Analysis along with (DF) Defining Problems are cognitive strategies that represent the ‘problem space’ of design thinking. Lawson (1979), identified problem solving strategies as either ‘problem focused’ or ‘solution focused’, and claimed the ‘solution focused’ strategies were more representative of a design-based problem solving strategy.

Only PLTW participants #2 and #3 appear to be stuck in the problem space, a common pattern for novice designers that limits their ability to generate solutions ( Cross & Dorst, 1999 ; Dorst & Cross, 2001 ). The PLTW participants that place emphasis on (AN) Analysis may be performing this way because the PLTW students have had experience in identifying constraints and criteria in their PLTW course work and classroom activities. For example, the PLTW Principles of Engineering project, the marble sorter activity (see description above in classroom observation section of the findings) require the student teams to troubleshoot problems that occur in their design solutions by identifying what constraints and criteria are limiting the success of the device. This result of this study reveals that these PLTW students are possibly transferring their knowledge and experience from the Principles of Engineering class to the transfer problem sessions. Although these results are promising to illustrate a curriculum program having impact on students ability to transfer their learning to an open-ended transfer problem, too much emphasis of (AN) Analysis can limit students’ abilities to move from problem space into solution space ( Cross & Dorst, 1999 ; Dorst & Cross, 2001 ).

Limited Mathematical Thinking

Both EPICS High and PLTW student participating in the transfer problem observational protocol sessions had limited employment of (CO) Computing. The Halfin code computing is defined as “The process of selecting and applying mathematical symbols, operations, and processes to describe, estimate, calculate, quantity, relate, and/or evaluate in the real or abstract numerical sense”. The researchers did not expect that participants were going to be generating multiple mathematical models to develop solutions to this transfer problem in a 15 -30 minute test session; however, the transfer problem was created purposely with numbers embedded within the problem to see if students would attempt to quantify, estimate, calculate, or describe design solutions using the numerical information provided within the open-ended problem. None of the six participants spent more than 3% of their design thinking time using the (CO) Computing strategy. PLTW #4 spent the greatest amount of time computing, dedicating 3% of time on this strategy, five other participants (PLTW #1, 3, 6 and EPICS #1, 6) never employed computing once during the testing session. Like the results of a similar study, Kelley (2008) , found that often the participants with the least amount of math instruction employed the most mathematical thinking. PLTW # 4 had only two math courses (Geometry and AP Algebra) but applied the most mathematical thinking for 3% of time compared with PLTW 5, PLTW 6, EPICS 4, EPIC 5, and EPIC 6 that had five or more math courses but employed (CO) Computing for 1 or less % of time. In fact, PLTW 6 had taken six math courses and EPICS 6 had taken five courses and neither participant employed any mathematical thinking during the protocol session. Some leaders in engineering design based instruction at the secondary education level suggest that mathematical modeling and mathematical analysis are key missing pieces in the technological design process ( Hailey, et al., 2005 ; Wicklein, 2006 ). Although all student participants have taken advanced coursework in mathematics ranging from Algebra 1 to AP Calculas, the participants’ employment of mathematical strategies was minimal. This is an extremely important finding that could limit supporters of PLTW and EPICS High to promote these curriculum projects as effective strategies for improving STEM education.

Design Example Fixation

As stated above, the students were given an ill-defined problem presented on a one page handout. On the problem statement handout, there was a stock photo of a school under construction and its surrounding areas; see appendix A. The photo was used to provide a general context to the problem statement. All of the participants at some time in the testing session referenced the photo and sometimes analyzed the photo even though the photo was never referenced in the problem statement. Each participant at some point in the testing session used the photo like a template for the location of their playground. This discovery was unexpected; however the phenomenon has been discovered in other design studies ( Smith, Ward, & Schumacher, 1993 ). Smith, et al. (1993) have discovered that providing a designer, both novice and expert with a design sample can cause the designer to become fixated on the design example, thus, limit and influence the designer. Kelley (2008) also found that participants in a similar protocol study were often fixated on the picture that was included in problem statement. Some participants in that study even asked questions about the picture and carefully studied the picture’s URL citation.

In this study, students were potentially limited in their creative thinking to the context of the photo of a school under construction. This can be an extremely important discovery for K-12 engineering educators because it illustrates the potential negative impact of providing an existing design example to students before they generate their own design ideas or even to provide images or illustrations representing the problem. Furthermore, this result of the study may indicate that students often believe that solutions to assigned problems are contained within the student handout.

Inter-rater Reliability Results

Two researchers were used to code the transfer problem observational protocol sessions. The two researchers were trained by the PI of this study on how to interpret the Halfin code and how to record the codes using the OPTEMP software. The researchers practiced coding protocol sessions using sample videotape sessions as a part of their data analysis training. Both research assistants were present for all transfer problem sessions and each researcher coded independently using the OPTEM software. The researchers hope for correlation close to 1, indicating that both raters agree and affirm the reliability of these results. The lowest Pearson correlations were .76 and .75 for PLTW #1 and PLTW #2, these were the first two transfer problem sessions that were coded by the research assistants, see Table 8. It is logical that the researchers were refining their coding abilities during these first two sessions. However all other correlations were .93 or higher indicating a strong correlation and reliable coding of the transfer problem sessions.

Table 8. Inter-rater Reliability- Pearson R results

PLTW participants	Pearson R	EPICS High Participants	Pearson R
PLTW #1	0.76	EPICS High #1	0.98
PLTW #2	0.75	EPICS High #2	0.99
PLTW #3	0.92	EPICS High #3	0.99
PLTW #4	0.93	EPICS High #4	0.94
PLTW #5	0.99	EPICS High #5	0.94
PLTW #6	0.93	EPICS High #6	0.95

Limitations

The researchers acknowledge the study has a small n of approximately 60 students observed in PLTW and EPICS High classes and an n of 12 participants for the observational protocol; therefore, the result of the study are not generalizable to all secondary engineering design programs.

Other possible limitations include a potential bias regarding PLTW and EPICS High. According to Merriam (2001) researchers must acknowledge potential biases within the final report of the study. The researchers acknowledge that a potential bias could exist regarding both PLTW and EPICS High due to the fact that the researchers work within an engineering/ technology teacher program that provides undergraduate students with certification in PLTW courses and the fact that EPICS High was created by Purdue engineering education faculty. To help ensure that such biases did not taint research findings, the researchers implemented methods such as Merriam’s observational elements guide (2001). The observational elements guide provided researchers with guiding questions that required reflection on a variety of factors in order to maintain an objective perspective as the researcher collected observational data. The researchers also use triangulation methods to provide multiple perspectives and multiple forms of data. Finally, researchers used multiple raters on the observational protocol transfer problem session to maintain consistent and reliable data collection. These are sound research methodologies that can help to ensure an objective data collection process.

Conclusions / Implications

The researchers acknowledge that classroom observations were limited to approximately 60 students observed in PLTW and EPICS High classes and a sample of 12 participants for the observational protocol; therefore, the result of the study are not generalizable to all secondary engineering design programs. However, all members of the STEM education community should carefully consider the results of the findings from this research. The results of this study indicate that students from both engineering design curriculum programs successfully develop problem solving abilities to move from ‘problem space’ to ‘solution space’ as they worked through an open-ended ill-defined problem. Furthermore, classroom observation revealed that student design teams had healthy design discussions and worked cooperatively to solve technical problems.

However, students were limited in their ability to use mathematical thinking as a design tool to help create design solutions. Although most of the students had advanced math classes with over 50% having 3 or more classes (four participants had a total of five math classes), students spent 3% or less time using mathematics in their design thinking. This finding confirms prior findings in similar research. Kelley (2008) , a study of two approaches to design instruction also used a protocol session with a transfer problem found that mathematical thinking was a limited cognitive strategy employed by the protocol session participants. Moreover, Kelley and Wicklein in a national descriptive study of engineering design curriculum content found a low emphasis on mathematics and engineering sciences in current technology education curriculum (2009a). Technology education teachers also place a low emphasis on mathematics in student project assessment; the individual item using mathematics to optimize and predict design results yielded the lowest percentage of assessment time ( Kelley & Wicklein, 2009b ). It appears that technology education teachers are not placing great emphasis on using mathematical models to predict design results. Technology education teachers appear to recognize this limitation and have identified it as a teacher challenge. Kelley and Wicklein (in-press) in the national status study of technology education regarding engineering design also found technology education teachers identified: integrating the appropriate levels of mathematics and science into instructional content (mean of 2.49 on a four-point Likert) as a major challenge implementing engineering design content. This was the highest mean score for an individual teacher challenge item, thus indicating this is major concern for technology education teachers. More efforts need to be placed on identifying appropriate mathematical models and science inquiry examples that teachers can use to integrate STEM concepts in engineering design curriculum content. EPICS High teacher #2 provided some insight on why students had limited use of mathematics in the transfer problem in response to the question:

“Do you believe that your students will employ mathematical thinking (when appropriate) as they work to solve the transfer problem?”

EPICS High teacher #2’s response:

“Yes, I have given similar design problems that required students to use math to solve the problem. However, if students can see that they don’t need the math to begin designing then I think they won’t use math….if they can find a way to solve the problem without math…they will.”

These research findings should be considered by educators and curriculum developers of pre-engineering or secondary engineering design curriculum, of which many also promote these programs as ideal platforms for STEM education. These findings indicate that currently the transfer of STEM learning through the engineering design process on ill-defined problem solving is limited. Engineering design curriculum developers must be more purposeful in creating learning experiences that embed mathematical problems and science inquiry activities into the engineering design process. More efforts need to take place to locate appropriate ill-defined problems that can be explored through the engineering design process that will authentically engage students in the analysis and optimization stages of the engineering design process. Furthermore, greater efforts must take place across the nation to provide professional development opportunities to assist math, science, and technology education teachers to locate appropriate levels of STEM subjects for delivery in preengineering or secondary engineering design curriculum.

Future Studies

The researchers suggest the following studies for the future that can extend the results of this research:

This research study has provided great insights into several approaches to teaching engineering design content at the secondary level. From this study, other practitioners in the field of technology education will better understand what is taking place in secondary education classrooms regarding the teaching of engineering design. However, more information is needed to help properly inform the field about this construct. Consequently, the following recommendations are suggested for further research to inform the field of technology education:

1. Larger n for observational protocol studies

   Great insight has been obtained from conducting a study to extend the results of prior work ( Kelley, 2008 ). The sample size of 12 students for the transfer problem nearly doubled the prior work ( Kelley, 2008 ); however a sample size of 12 is too small to generalize to the population. More observational protocol studies need to be conducted on students engaged in design activities that have authentic levels of math and science inquiry embedded into the design problems. This type of study can help locate an appropriate balance of cognitive capabilities that will have the greatest impact on STEM abilities and produce effective designers and technical problem solvers.

2. Expert designer studies

   The field of K-12 engineering education can learn from expert engineers and expert designers through observational protocol studies of these experts working through a transfer problem. A study of this type will help educational leaders in K-12 engineering education determine possible appropriate levels of various cogitative capabilities employed at an expert level. Currently, the researchers can only indicate based upon personal opinion that less than 3% of a student’s engineering design thinking dedicated to mathematical thinking (CO) Computing is not at an acceptable level. Expert designer observational studies can provide great insight into what percentage of time on various cognitive capabilities indicate an individual is functioning as an expert designer.

3. Revisit Halfin’s study

   Harold Halfin (1973) conducted his Delphi study of 10 expert technical problem solvers back in 1973, others have revised the study before Wicklein and Rojewski, 1999 and Hill, 1997 but now is an appropriate time to revisit this research and locate expert technical problem solvers that represent the type of designers that employ the engineering design process but also exemplify the type of professionals who would have credentials to function effectively in today’s global society, see Friedman,2005 ; Pink, 2005 . Results from a study using Halfin’s study as a model could help leaders in K-12 engineering education identify appropriate competencies for secondary engineering education.

---

Todd R. Kelley, Ph.D. is an assistant professor in the Department of Industrial Technology at Purdue University. He can be reached at trkelley@purdue.edu .

Mr. Daniel C. Brenner is a high school teacher at Bartlett High School. He can be reached at brenner_daniel@asdk12.org .

Mr. Jon T. Pieper is a graduate student in the Engineering/Technology Education program at Purdue University. He can be reached at jpieper@purdue.edu .

References

Appendix A. Transfer Problem

Problem

A new K-5 elementary school has been constructed in the local area and new playground still needs to be designed. You obtained the following specifications: The school is expected to have around 500 students. The area of the playground has not been determined and space is limited. Safety of both students and school properties needs to be considered. For example, if students are playing softball too close to a building, they would risk breaking windows. At minimum, one entire grade will use the playground space at once. The playground needs to be attractive and fun for all students.

Your Task

Describe how you would design the playground for the school in the problem statement. Please describe all assumptions, information you need to obtain, and justification of use of space as you “think aloud” your strategies for developing the solution.