JTE v25n2 - Team Based Engineering Design Thinking
Team Based Engineering Design Thinking
The objective of this research was to explore design thinking among teams of high school students. This objective was encompassed in the research question driving the inquiry: How do teams of high school students allocate time across stages of design? Design thinking on the professional level typically occurs in a team environment. Many individuals contribute in a variety of ways to facilitate the successful development of a solution to a problem. Teachers often require students to work in teams, but little is known about how the team functions in the context of design and the potential interaction between team performance and authentic design challenges. Few research results are available to guide high school teachers in developing successful design teams and how to encourage teams in their efforts.
Problem Statement
The National Center for Engineering and Technology Education has formalized a focus on infusing engineering design into secondary technology education ( National Center for Engineering and Technology Education, 2013 ). Results of the Center’s work have included engaging students in engineering design challenges using a problem based learning framework. While extensive efforts have be afforded to developing learning experiences and professional development activities for teachers, little attention has been given to how teams of high school students allocate their design time. The endeavor to model design problem solving satisfactorily has engaged scholars across domains ( Hayes, 1989 ; Newell & Simon, 1972 ; Polya, 1945 ; Rubenzer, 1979 ). Understanding how students use their time provides educators with opportunities to improve areas where students spend little effort. This study investigates how teams of students allocate time in the design process and draws comparisons between two different design problems administered. Comparisons are provided between teams in this study and previous studies on individual experts and individual students.
The discrepancy between our society’s dependence upon technology and our ability to understand various technological issues has emerged as a serious concern for educators. “Technology is the outcome of engineering; it is rare that science translates directly into technology, just as it is not true that engineering is just applied science” ( National Academy of Engineering, 2004 , p. 7). Specifically, “Americans are poorly equipped to recognize, let alone ponder or address, the challenges technology poses or the problems it could solve” ( Pearson & Young, 2002 , pp. 1-2). The relationship between understanding engineering and technological literacy is of special urgency during the high school years, since “technologically literate people should also know something about the engineering design process” ( Pearson & Young, 2002 , p. 18).
Design thinking is fundamental to understanding the technologically dependent nature of our society. A need for a technologically literate populace, therefore, includes an understanding of the engineering design process. The design process links technology and engineering, two elements of STEM education. “Design is the central component of the practice of engineering and a key element in technology education” ( Pearson & Young, 2002 , p. 58). While design thinking is an elusive and difficult construct to define, measurements for this study included a pertinent subset of measurements consistent with previous literature, much of which was generated through work of the Center for Engineering Learning and Teaching ( Atman, Chimka, Bursic, & Nachtmann, 1999 ; Atman, Kilgore, & McKenna, 2008 ; Morozov, Yasuhara, Kilgore, & Atman, 2008 ; Mosborg et al., 2005 ; Mosborg et al., 2006 ). This paper reports on measurements including time allocated across essential elements of the design process and disaggregates the data by problem type and team gender composition.
Methodology
A descriptive study was conducted spanning multiple high schools in urban, rural and suburban environments. This study identified quality high school technology and engineering learning and teaching environments in a criterion based sampling strategy. Criterion for selective schools was aligned with the vision of Pearson and Young, where “technology teachers with a good understanding of science and the interactions between technology, science, and society will be well prepared to work with other teachers to integrate technology with other subjects” (p. 108). Teachers at the target schools permitted advertising to recruit their students for participation in the study. Students in this study were considered to be representative of experienced students who had taken most or all engineering related courses at their high school. Students were recruited who were actively engaged in the study of engineering design through a criterion sampling strategy ( Creswell, 1998 ) using the following criteria:
- The high schools had an established program of study which employs a focus on engineering in a sequence of courses developed in association with an engineering outreach effort as part of a university program.
- In these courses, students participated in design activities which engage their critical thinking and problem solving skills within the framework of the engineering design process.
Sample. The quantitative research method design leveraged the use of data from 17 design teams comprised of 2-4 students each for a total of 47 students. The 17 teams were composed of eight male only teams, four female only teams, and five mixed gender teams. The teachers grouped the students into teams according to their personal schedules, and the teams were assigned the design challenge. Some team members were friends while other teams were comprised of students who did not know each other well. All members of a team were from the same school, but the 17 teams spanned four schools in two states. Each of the schools selected to participate had a recognized engineering program associated with an outreach effort by a university engineering program. Curricular offerings at the high schools included Project Lead the Way (PLTW), Engineering Projects in Community Service (EPICS) High, First Robotics, and locally developed engineering and/or technology courses supported by their regional University.
Demographics . Students were selected who represent diverse backgrounds and have chosen to enroll in this sequence of courses. Most students in this study were seniors who had taken multiple engineering courses during their high school experience. Typically participants were enrolled in a senior level capstone design course such as the Project Lead the Way, Engineering Design and Development course. Approximately one-third of the students were female and most reported their ethnicity as White or Asian. About one-half of the students responded to a question about their future career choice and nearly three-quarters of those indicated engineering. Refer to table 1 for demographics.
| Student Response | |
|---|---|
| Mean Number of Engineering Courses | 3.6 courses |
| Expressed interest in Engineering | 68% |
| Females | 34% |
| Seniors | 74% |
| Underrepresented in Engineering | 17% |
Engineering Design Problems . According to the National Center for Engineering and Technology Education Caucus Report of 2012, “There is a need for more definitive guidance about what makes quality design challenges and how they can be implemented well in existing courses” (p. 2). Two different design problems were administered in this study which were open-ended, realistic, accessible, and complex ( Mosborg, et al., 2006 ). The “playground problem” was provided to permit comparisons between team and individual performance while the “street crossing problem” variation facilitated comparisons among types of problem structures. The “playground problem” was comparable to the design problem used in previous studies with individual high school students ( Mentzer, Becker, & Park, 2011 ) and previous work with college students, and experts ( Atman, et al., 1999 ; Mosborg, et al., 2005 ; Mosborg, et al., 2006 ). The street crossing problem was less structured, could be readily adapted in order to be locally relevant, and was potentially more authentic for the participants because it was situated in a local context experienced daily by the students. In addition, students engaged in the pedestrian flow problem had an opportunity to specify constraints and criteria as none were presented. The “pedestrian flow problem”, administered to approximately half of the teams, was a variation of the “street crossing problem” adapted from previous literature ( Cardella, Atman, Turns, & Adams, 2008 ; Carie Mullins, Atman, & Shuman, 1999 ).
Administration of the Design Challenges . Eight of the teams received a playground design problem and nine received a locally relevant school hallway traffic flow problem. Teams were expected to develop a solution in 2 hours. The interactions of group members were video and audio recorded while they were developing the design solution. Data included video and audio recordings of the design sessions. Video cameras were small, mounted on miniature tripods to minimize their intrusion. All students were wired with a lavaliere microphone to ensure high quality audio feeds. Wires were run under the team workspace to prevent tangling, however, the wires limited student mobility. Students generated documents and other artifacts with traditional office supplies provided. Artifacts typically included sketches, notes, and formal drawings.
The Playground Problem has been used in multiple studies and can be traced to Dally and Zang (1993) . The original need for project driven approaches in the freshman engineering design course was to increase student performance and retention and to situate student learning of abstract concepts through real world applications in an experiential activity. In the original activity, students designed a swing set with slides and seesaw. Atman et al. (1999) revised the foundational work of Dally and Zang to create a playground design problem. In their challenge, engineering students were presented with a brief playground design task and access to background information upon request. Participants were provided with a maximum of three hours to develop a solution to the problem while thinking aloud. Mosborg et al. (2005) applied the playground design challenge using the “think aloud” research protocol with 19 practicing engineers who were identified as experts in the field. Mosborg et al. (2006) compared groups of freshman and senior engineering students with practicing engineers using data their research team previously collected on the playground design challenge. Atman et al. (2008) analyzed data from previous studies with a focus on the language of design, its relationship to design thinking as a mediator, and relationships between the internalization of design thinking and language acquisition.
Consistent with previous studies on college and high students, participants were given a one page design brief of the playground problem. The participating teams, acting as engineers, were assigned to design a playground on a donated city block. The constraints included limited budget, child safety, and compliance with zoning regulations and applicable laws. Participants were able to query the research administrator for additional specific information on the lot layout, cost of materials, neighborhood demographics, or other information. There was a two-hour time limit for completion of the design proposal, which was a modification of the original three hour limit in previous studies ( Becker, Mentzer, & Park, 2012 ). This modification was made because the average design time in the previous study of high school students was about 90 minutes. The two-hour limit provided more time than the average individual needed, yet it reduced the resources needed for data collection. The participants presented a written proposal describing their design. This activity engaged the participants in problem framing and the development of an initial solution. Limitations of this design task included the lack of opportunity for participants to investigate the need for a solution, since the problem was simply assigned to them. Students did not have an opportunity to construct physical models or prototypes. Participants were aware that implementation of the design project would not occur and that their designs would not be realized.
The Street Crossing Design Problem was adapted from previous research ( Cardella, et al., 2008 ; Carie Mullins, et al., 1999 ). The National Center for Engineering and Technology Education assembled a Caucus in August 2011 to identify characteristics of engineering design challenges ( Householder, 2011 ). Results of the discussions by this group of experts indicated that excellent design challenges should incorporate the following characteristics:
- Authenticity
- Have personal and social relevance
- Require analytical thinking
- Involve group efforts
- Require hands-on participation
- Are clearly structured but open-ended
- Foster creative solutions
- Consider ethical issues
- Meet applicable constraints
- Provide opportunities for modeling with replication
- Consider systems implications
- Are well documented
- Are self-assessed and independently evaluated
- Enable communication among team members
The Street Crossing Design Problem was potentially more authentic and more closely aligned with the National Center’s Caucus suggestions than the playground problem. “Authentic problems currently affect real-life situations encountered by the learners, their families, and their communities – and they do not have a generally recognized “right answer.” ( National Center for Engineering and Technology Education, 2012 , p. 22). The problem was modified slightly from its original administration to more closely exemplify these characteristics. The street crossing problem was discussed with teachers at the schools involved. Teachers were asked to think about an intersection fitting characteristics of the original problem but located at or near the school where students would immediately recognize the problem as they personally experienced it daily. After negotiating with the teachers, it was discovered that car/pedestrian traffic flow was an issue, but a more relevant and pervasive similar issue was pedestrian (student) flow in school hallways. Each school had one or more significant blockages that caused congestion, frustration, and delay at passing times between classes. The design problem was modified to focus on student hallway flow rather than car traffic flow, making the problem more relevant and personal, as most students experience the congestion several times per day.
Each pedestrian flow problem was presented in a similar format: the school floor plan (“map”) was provided to participants along with a very brief narrative stating that the student team was a team of engineers contacted by the school district. The narrative introduced an area that the students immediately recognized as a congested area and requested that the team present a proposal for resolving the issue. The constraints and criteria were not specified; leaving student design teams the opportunity to discuss and negotiate their specific problem definition and determine the most appropriate solution proposal. An example pedestrian flow problem looked like this, though details varied across schools to situate the problem in the local context:
You are a team of engineer’s contacted by [your school name here] School District. Often hallways are congested at passing time between classes. Hallway one, which is between the new and old portions of the school, is difficult to navigate. [your school name here] School District would like your team to develop and propose a solution.
This less structured problem is consistent with the National Center’s suggestion that “Engineering design challenges are ill-structured problems that may be approached and resolved using strategies and approaches commonly considered to be engineering practices” (2012, p. 2). Typical office supplies were provided for the participants, a condition similar to those in the playground problem. However, participants were given access to the Internet in lieu of printed sheets of relevant data. The decision to provide Internet access was made to increase the sense of authenticity and relevance as students are familiar with and accustomed to having Internet access. The notion of having predetermined what information is needed for their solutions may unintentionally guide student design decisions to those based upon a finite resource pool.
Data Analysis . Time is a limited resource and the ways designers allocate their time among the areas of the design process has been a focus of previous work. The coding scheme was congruent with the approach used in earlier studies ( Atman, et al., 1999 ; Bursic & Atman, 1997 ; Mosborg, et al., 2005 ; Mosborg, et al., 2006 ). Two measurements of time were made while the designers were at work: time allocated to elements of the design process; and total time engaged in design. The unit of analysis was the team. The data were coded into the nine categories presented by Mosborg et al. (2006, p. 15) : (1) Problem Definition, defining what the problem really is; (2) Gather Information, searching for and collecting information needed to solve the problem; (3) Generating Ideas, thinking up potential solutions (or parts of potential solution) to the problem; (4) Modeling, detailing how to build the solution (or parts of the solution) to the problem; (5) Feasibility Analysis, assessing and passing judgment on a possible or planned solution to the problem; (6) Evaluation, comparing and contrasting two (or more) solutions to the problem on a particular dimension (or set of dimensions) such as strength or cost; (7) Decision, selecting one idea or solution to the problem (or parts of the problem) from among those considered; (8) Communication, the participants’ communicating elements of the design in writing, or with oral reports, to parties such as contractors and the community; and (9) Other, none of the above codes apply. Statements coded as other included administrative questions such as “should we draw this on paper?” and “can I use a calculator?”. Statements coded as “other” also were statements that were vague and ambiguous. Statements were only coded with one of the eight codes if reasonable evidence existed to justify the claim and therefor a statement that was made but not understood by the coders was assigned as “other”.
Data analysis began with segmenting the data sets. A team of three researchers was tasked with the responsibility of segmenting. A segment was defined as a pause bound utterance, as suggested by Atman et al. (1999) . Researchers were instructed to create a new segment in the video timeline for each instance when any student on the team began a new thought, which was typically indicated by beginning to speak after a pause. In previous literature ( Becker, et al., 2012 ; Mentzer & Becker, 2010 ; Mentzer, et al., 2011 ), this segmenting procedure was applied to individuals. For the current study of teams, researchers created these segments each time any member of the team made a transition. The resulting segmented data represented the composite of all team member segments. At some points in the videos, all team members were functioning as one cohesive unit and segmenting was simple and monolithic. In other times, a team of four students might naturally divide into two teams of two and the segments represent start/stop times for each sub-team. By segmenting in this fashion, a divergence in design activities could be coded with two separate codes in two different, but overlapping episodes.
Quantitative measures of inter rater reliability on the segmenting process were not made. The research leadership determined the segmenting would be of reasonable quality if the inter-rater reliability measures for coding were high. If segmenting were done successfully, coding could potentially result in high inter rater reliability. Coding served as a proxy for quality control of the segmenting process. As a preliminary quality control mechanism, the lead researcher reviewed segmented work and provided feedback and guidance as the research assistants progressed.
Two undergraduate students coded the data in three phases. The first phase was to establish calibration of the research assistant’s coding work. The second phase served to document the calibration using Kappa values as a measure of inter-rater reliability. The team of two research assistants coded 25% of each video and compared. The Kappa values averaged 0.71; details are presented in Table 2 (next page) along with the number of references used to generate the values. In the third phase, all videos were fully coded, approximately one-half by one research assistant and one-half by the other.
In the calibration phase, research assistants were provided with a conceptual overview of the coding process, structure, technique and rationale. They were presented with examples from previous work and practiced coding these data. Research assistants then coded a portion of a video and compared with each other. They met with a senior research team member and discussed the individual interpretations and differences to establish clarification on coding. A “Dynamic” Code Book was adopted and maintained. This was a document with very specific examples of the different codes developed by creating a description of the code and compilation of examples in context. This included adding detail and clarifying the meaning of segmenting and coding procedures and providing examples as coders did their work. The document was updated regularly and shared via network real time. As understanding and interpretation was negotiated by the coders and research team leaders, the codebook documents evolved into increasingly specific definitions.
The calibration process was iterative. Each coding session was followed by a debriefing session and the cycle started over. Kappa values began relatively low and rose gradually as the research assistants became more closely aligned in their designations. When average Kappa values for each code approached 0.70, the research team transitioned into the next phase which was documentation. Some effort was focused on calibration, but most effort was allocated toward coding a random 25% of each video and documenting the comparison.
| Design Activity | Cohen's Kappa | References Compared |
|---|---|---|
| Problem Definition | 0.76 | 152 |
| Gathering Information | 0.72 | 630 |
| Generating Ideas | 0.66 | 65 |
| Modeling | 0.68 | 1412 |
| Feasibility | 0.46 | 294 |
| Evaluation | 0.75 | 26 |
| Decision | 0.80 | 15 |
| Communication | 0.88 | 732 |
| Average Inter-Rater Reliability | 0.71 |
In the final phase of coding, the 17 videos were divided among the two research assistants. Earlier work resulted in 25% of each video being coded already; the remaining 75% was coded. The entire video was reviewed and changes were made as needed to the coding structure in context of the newly coded 75%.
Results
The video data were coded by time allocated to: Problem Definition, Gathering Information, Generating Ideas, Modeling, Feasibility, Evaluation, Decision Making and Communication efforts. Activities that the team engaged in were coded. Key differences are discussed in terms of comparisons between individuals and teams, problem types and team gender. At times in the process, team members were all simultaneously engaged in one activity, but, at other times, individual students would engage in different activities. When team members provided reasonable evidence that they were doing two different activities, two or more codes were applied. The total coded data exceeded 100% in all teams because, at times, the team was receiving credit for two or more codes simultaneously. With this study’s small sample size (n=17) statistical analysis was not conducted. However, trends emergent in the time allocation between individuals and teams may provide a foundation for future study.
Individuals vs Teams . Teams averaged 102 minutes in the design process as compared to individuals from previous work who finished, on average, at 92 minutes ( Becker, et al., 2012 ). Table 3 shows the average time invested by the teams in each design activity in this study and the average time invested by individuals in the previous Becker and Mentzer study.
Information Gathering . Teams spent nearly twice the percentage of time engaged in gathering information. Information gathering was coded when students were actively requesting, reading, and reviewing information related to the problem or solution. Information requests could be made of the administrator. Teams working on the hallway traffic design challenge were provided with a laptop and Internet access. Student use of the Internet was generally coded as information gathering and represents a difference from the data collection protocol used with individuals and playground design teams as they did not have access to the computer.
| Design Process Measures | Individual (n=59) (Becker, et al., 2012) | Teams (n=17) | ||
|---|---|---|---|---|
| Minutes (SD) | Percent of time | Minutes (SD) | Percent of time | |
| Total Time | 91.7 (47.4) | 101.7 (18.43) | ||
| Problem Scope | 15.5 | 18.0 | 27.2 | 26.3 |
| Problem Def. | 5.6 (3.1) | 7.7 | 6.8 (5.49) | 6.7 |
| Info. Gath. | 9.9 (13.3) | 10.3 | 20.4 (12.19) | 19.7 |
| Solutions | 63.2 | 70.5 | 55.6 | 55.1 |
| Generating | 2.9 (6.6) | 3.9 | 2.8 (1.62) | 2.9 |
| Modeling | 54.4 (35.4) | 60.2 | 44.2 (13.29) | 43.4 |
| Feasibility | 4.4 (4.1) | 5.4 | 8.0 (4.41) | 8.3 |
| Evaluation | 1.1 (3.5) | 1.0 | 0.5 (0.78) | 0.5 |
| Realization | 8.2 | 7.6 | 24.2 | 23.6 |
| Decision | 0.4 (0.7) | 0.4 | 0.2 (0.32) | 0.2 |
| Comm. | 7.8 (13.0) | 7.2 | 24.0 (12.87) | 23.4 |
| Other | 3.1 | 3.8 | 9.1 | 9.0 |
Modeling and Communication . Time allocated to modeling and communication show differences between teams and individuals. The teams tended to spend less time modeling and more time communicating. Modeling was defined as detailing how to build something, including calculations, estimations, determining locations, and description of how something will be assembled or fabricated. Communicating was defined as the efforts involved in telling someone how to build the playground. Communication efforts focused on sharing the team’s plan with others and could be directed toward a contractor or a board of directors considering the team’s proposal.
Playground vs. Pedestrian Flow problems . The playground problem had been used extensively in previous studies described above, but was potentially less relevant and authentic to high school aged students who are generally too old to use playgrounds and too young to have children of their own playing on playgrounds. The pedestrian flow problem was experienced daily by students and differences in design process times were evident in this study. Table 4 summarizes time allocations as a comparison between problems.
Problem Definition . Time spent on problem definition differed between the pedestrian flow and playground problems. Teams spent more than twice the amount of time reading, reflecting on and considering the problem for the playground compared to the pedestrian flow challenge. The playground problem presentation was longer and more specific. Constraints and criteria were specified as compared to the pedestrian flow in which constraints and criteria were not specified. The research team had anticipated that the lack of definition would permit students to develop their own constraints and criteria relative to their local problem, but, time in the problem definition phase was actually less when constraints and criteria were not provided.
Information Gathering . Teams working on the pedestrian flow problem spent about twice the percentage of time searching for and digesting information relative to the problem than did the teams working on the less familiar playground problem. Examples of information gathered in the playground problem focused on identifying typical components on playgrounds such as swings, slides, monkey bars, and material characteristics such as strength, durability, and cost. Information across both problems included benchmarking, but on different conceptual levels. Searching for playground components was a concrete task resulting in a list of typical play things while the hallway problem yielded much more complex transfer from other schools or public places where traffic congestion was a problem. Students looked at airports as examples of moving people in short periods of time as a potential method of benchmarking and gathered these examples to spawn ideas in their scenario. The transfer from an airport or mall hallway to a school hallway was challenging for students perhaps because the population of users was different (i.e. adults in airports vs. students; adults may be motivated to run to their next flight vs. students who may not be interested in getting to the next class).
Students in the hallway problem appeared to spend more time searching, perhaps motivated by their personal interest. In each administration of the problem, students were obviously bothered by the problem and were quick to engage as compared to the playground problem where students engaged at our request but seemed less intrinsically motivated. The hallway challenge included Internet access, which could have related to the additional search time. Our informal observations seemed to indicate that students not only accessed the computer, they also gathered information from memories of direct observations. They recalled their experiences in airports and malls with pedestrian congestion. They recalled traffic flow rates and locker placements in the school and considered the impacts of this information on their design process.
Feasibility . Teams on the pedestrian flow problem spent more than twice the percentage of time considering feasibility of their solutions as compared to the teams on the playground problem. Feasibility was defined as considering the practicality or viability of a solution or element of the solution. This was differentiated from evaluation in that evaluation included comparing two or more options while feasibility was passing judgment on one potential idea. In the playground problem, feasibility typically centered around cost in addressing the question: “Would an item/component cost too much?” Also, playground design teams considered the extent to which they met the constraints. This differed in the hallway problem because students were not provided with constraints or criteria nor did teams spend time to specify either constraints or criteria for the solution of the problem. Feasibility, however, consumed a much greater percentage of time as students attempted to determine if their solutions would work. Students implicitly must have identified some constraints and criteria as they talked about feasibility but not directly. Typical examples included students discussing the financial cost of a solution or the impact that the proposed solution might have on the problem without explicitly identifying a budget or rationale that costs should be limited or minimal. Some solutions were structural while others were behavioral and students considered their potential solutions and students’ behavioral responses. This led to discussions of teachers’ roles, administrators’ media campaigns for pedestrian traffic patterns and the feasibility consideration: “would it work?”
| Design Process Measures | Individual (n=59) (Becker, et al., 2012) | Teams (n=17) | ||
|---|---|---|---|---|
| Minutes (SD) | Percent of time | Minutes (SD) | Percent of time | |
| Total Time | 108 (10.9) | 95.8 (21.5) | ||
| Problem Scope | 15.5 | 18.0 | 27.2 | 26.3 |
| Problem Def. | 5.6 (3.1) | 7.7 | 6.8 (5.49) | 6.7 |
| Info. Gath. | 9.9 (13.3) | 10.3 | 20.4 (12.19) | 19.7 |
| Solutions | 63.2 | 70.5 | 55.6 | 55.1 |
| Generating | 2.9 (6.6) | 3.9 | 2.8 (1.62) | 2.9 |
| Modeling | 54.4 (35.4) | 60.2 | 44.2 (13.29) | 43.4 |
| Feasibility | 4.4 (4.1) | 5.4 | 8.0 (4.41) | 8.3 |
| Evaluation | 1.1 (3.5) | 1.0 | 0.5 (0.78) | 0.5 |
| Realization | 8.2 | 7.6 | 24.2 | 23.6 |
| Decision | 0.4 (0.7) | 0.4 | 0.2 (0.32) | 0.2 |
| Comm. | 7.8 (13.0) | 7.2 | 24.0 (12.87) | 23.4 |
| Other | 3.1 | 3.8 | 9.1 | 9.0 |
Feasibility seemed more relevant for students to consider in the hallway challenge as compared to the playground problem. Students were familiar with the hallway issues and the solutions had direct impact on their lives. They seemed to have capacity for understanding the complexities of hallway traffic more than complexities in the playground problem. Student design teams seldom considered the issues of safety in the playground, overlooking such facts as the difficulties that 2 year olds, have in negotiating ladders. If the students had been parents of children for whom the playground were being designed, they might have considered safety and functionality with greater understanding, but as 17 and 18 year olds, they seemed to lack a sense of understanding about the functionality and dangers surrounding playground equipment design.
Team Gender Composition . Teams were comprised of single gender and mixed gender. Four all-female teams, seven all-male teams and five mixed gender teams displayed differences in design processes. The gendered nature of the teams was difficult for the research team to control and, as a result, did not split equally across design problems. Three of the four all-female teams were provided with the pedestrian flow problem while only one was provided the playground problem. Five of the seven male only teams received the playground problem while two teams received the pedestrian flow problem. Two of the five mixed gender teams engaged in the playground problem while three attempted the pedestrian flow problem. Results of teamwork disaggregated by gender are presented in Table 5. Total design time varied across groups with female only teams finishing their work nearly 22 minutes before their all male counterparts. Mixed groups of males and females averaged about eight minutes less than all male teams.
Problem Definition . While female only teams spent less time engaged in design, they spent more time on problem definition (8.4 minutes) than did the all-male (7.0 minutes) and mixed gender groups (5.2 minutes). This finding is particularly noteworthy, as most of the female teams engaged in the pedestrian flow problem which, according to data presented in Table 3, drew relatively little attention to problem definition (4.1 minutes).
| Design Process Measures |
Female Only
(n=4) |
Male Only
(n=8) |
Mixed Gender
(n=5) |
|||
|---|---|---|---|---|---|---|
| Minutes (SD) | Percent of time | Minutes (SD) | Percent of time | Minutes (SD) | Percent of time | |
| Total Time | 87.1 (13.3) | 108.8 (7.9) | 101.9 (25.8) | |||
| Problem Scope | 26.6 | 29.9 | 27.1 | 24.6 | 27.9 | 26.3 |
| Problem Def. | 8.4 (6.4) | 9.0 | 7.0 (5.4) | 6.4 | 5.2 (4.2) | 5.3 |
| Info. Gath. | 18.2 (14.9) | 20.9 | 20.0 (9.7) | 18.2 | 22.7 (12.9) | 21.1 |
| Solutions | 44.5 | 50.8 | 58.4 | 54.0 | 59.9 | 60.2 |
| Generating | 2.4 (0.6) | 2.7 | 2.9 (1.7) | 2.6 | 3.1 (2.0) | 3.4 |
| Modeling | 35.6 (13.8) | 40.6 | 47.2 (11.4) | 43.7 | 46.2 (12.8) | 45.3 |
| Feasibility | 6.3 (1.8) | 7.4 | 7.8 (4.4) | 7.2 | 9.8 (5.2) | 10.7 |
| Evaluation | 0.2 (0.3) | 0.2 | 0.6 (0.8) | 0.5 | 0.8 (0.9) | 0.8 |
| Realization | 16.5 | 19.6 | 29.6 | 27.3 | 21.9 | 20.9 |
| Decision | 0.2 (0.3) | 0.2 | 0.2 (0.3) | 0.2 | 0.3 (0.4) | 0.3 |
| Comm. | 16.3 (3.8) | 19.4 | 29.4 (12.7) | 27.1 | 21.5 (14.0) | 20.6 |
| Other | 9.8 | 11.3 | 10.2 | 9.4 | 6.7 | 7.0 |
Modeling and Communication . Most of the differences in overall design time were related to modeling and communication. Females spent about 25% less time modeling than did males and mixed gender groups. The average female modeling time was 36 minutes while males and mixed groups spent 47 and 46 minutes respectively. Male student teams spent nearly twice the amount of time communicating (29 minutes) as did female student teams (16 minutes).
Discussion and Implications
Discussion focuses on four topics: inter-rater reliability, problem definition, evaluation and decision making, and modeling and communication. The development of this study was based on a series of studies situated on the college level ( Atman et al., 2007 ). In these studies, experts were provided with a design problem as a comparison group to college students. These experts provided a verbal protocol individually and were provided with a very comparable experience engaging with the playground problem as were participants in the current study. Overall, experts spent more time engaged in the problem with nearly 132 minutes being their average as compared to teams of high school students whom spent nearly 102 minutes. This difference in time was primarily in modeling and communication. Experts spent more than 55% of their time (about 73 minutes) modeling while teams of high school students spent less time modeling (about 43% or 44 minutes). However, high school students spent more time communicating (about 23% or 24 minutes) than did experts who allocated about 5 minutes or 4% of their time.
Inter-Rater Reliability . Reliability of modeling and feasibility coding was low despite extensive efforts to calibrate by the research team. Some of the lack of agreement could be related to researcher calibration, but the research team suspected that the lack of agreement is also related to lack of clarity by the student teams about the nature of these activities. The students were vague about how modeling was related to other aspects of the design process, particularly communication. What initially appeared to be graphical sketching as a method of developing ideas and laying out a potential solution for discussion evolved into a document for the final proposed solution. Differentiating between modeling and communication in the abstract was simple for the undergraduate research assistants as the difference centered on purpose. If the purpose of the sketching, for example, was to understand and improve appearance, functionality or fabrication techniques and the team used this information to think through challenges and determine specifications, it was coded as modeling. If, on the other hand, team effort was directed at documenting their plans for fabrication for the purpose of telling someone how to build from the plans, it was coded as communicating. Student teams often started modeling and the work evolved into communication. This evolution made precise determination difficult. In cases where the transition was gradual and vague, the coders generally defaulted to modeling until there was evidence that the purpose was an attempt to communicate team intentions/plans. In some cases, teams were very deliberate about this transition. In other cases, the transitions occurred gradually but were clarified later. For example, what might have appeared to be modeling was later determined to be communication and codes were changed appropriately as the coding process progressed through the team’s work.
The boundaries between thinking on paper via sketching and making notes, and communicating with external stakeholders blurred and presented the research team with difficulty identifying student intentions. At the end of the design challenge, students frequently presented rough sketches and messy notes, resulting in poor quality technical communication. If this situation is to be improved, teachers should make clear to their students how to communicate technical information in a persuasive way to external stakeholders. Classroom experiences might be focused on presentation skills where students present their work to other classes or an invited audience of people who are not familiar with the daily student design experiences. This external audience would challenge students to provide details and rationale for decisions made in context which was generally absent from student work in this research.
Problem Definition . Students engaged in dialog about the feasibility of their potential solutions or elements of solutions in the hallway problem more than in the playground problem, but problem definition was considered more in the playground problem. These two activities may be inversely correlated such that a general lack of problem definition would lead to a tendency for students to be quick to question whether a potential solution would work. In the hallway problem, student teams seldom made explicit the constraints and criteria which made determining the feasibility of an idea more difficult and time consuming. The National Center for Engineering and Technology Education suggested, “As designs are considered for viability, optimization is essential. Students should make their value structures and goals for design success explicit early in the decision process. This sense of clarity provides opportunities to select and promote designs that make the most successful balance of trade-offs” (2012, p. 26). In the less structured hallway problem, students would consider an idea and then ask if it would work or be too expensive without having specified the definition of success or budget.
Teachers should encourage students to identify the constraints and criteria as well as how success should be measured early in the design process. Feasibility considerations were slightly higher for team based design problems than individual design problems. Further research might test for a causal relationship, which if present, would indicate that teamwork might facilitate experience and exposure to critical thinking about solutions in the feasibility phase of the design experience. The National Center for Engineering and Technology Education caucus of 2012 suggested that “In collective team efforts, students may hold each other accountable for meeting criteria” (p. 18). The sense of accountability may have manifested in feasibility as students questioned each other’s ideas prompting consideration of flaws and opening the door for improvements. Stakeholder interests were included in student discussions of feasibility in the hallway problem much more frequently than the playground problem. In the hallway problem, they mentioned considerations such as how students would interact with their solution, how teachers would be involved and react to students in a redesigned hallway. They considered impacts of hallway reconstruction on the neighboring rooms and how the changes impact roles of librarians, cafeteria staff, and classroom teachers. Occasionally, parents and shopkeepers were mentioned in the playground problem as stakeholders but with far less emphasis. This may be related to the sense of relevance provided by the hallway problem, because students cared in a very personal way about the success of their design and considered a larger system of stakeholders.
Evaluation and Decision Making . In both design problems, evaluation and decision making activities were rarely observed. Student teams spent very little time comparing alternatives on a criterion, which was our working definition of evaluation. Students also spent very little time choosing among the alternatives. Decision making was defined in this study as a deliberate choice between two or more alternatives. A typical decision and evaluation activity in the playground problem included material selection. Student teams would ponder using wood or metal as a construction material, discuss costs, strength and durability, then make a selection. In the hallway problem, even fewer evaluations or decisions were observed. This lack of evaluation and decision making may be directly related to the fact that students developed few alternative solutions during the brainstorming phase. While they did brainstorm and develop ideas, selection decisions tended to be related to the feasibility of individual components of the solution rather than a comparison of alternative solutions. Students would frequently say, “Let’s put in a slide, it’s cheap” with no externalized comparison of the alternatives.
The general lack of evaluation and decision making may relate to a lack of alternatives for consideration. Students tended to think about new ideas until they had a few viable options and developed those into their final design. The lack of alternatives generally reduced the need to evaluate differences between them and reduced the number of decisions (choices between alternatives) to make. Teachers should encourage students to develop a significant list of alternative ideas before evaluation and decision making. Decisions regarding materials were made on the playground problem, but in both design problems, students did not develop many alternative designs. They considered the advantages of different materials but seldom considered holistically different solutions.
Future Research
Findings from this study suggest potential trends and correlations between individual design activities and group design work as well as suggesting differences between teamwork on two different kinds of problems. This study had a sample size of 17 teams in total. Eight teams were challenged with the playgroup problem and nine teams engaged in the hallway problem. With only eight or nine teams in comparison, statistical analysis was not conducted. This work is potentially foundational to larger studies as it may allude to trends and correlations that could be tested in experimental or quasi-experimental research conditions on a larger scale.
This work may have implications for methods of group based verbal protocol studies. Future research efforts using student observations might increase their inter-rater reliability measures with teams by being able to identify what papers students are using when they are writing or sketching. In work with individuals, reviewing the digitized artifacts and video observation data typically provided ample evidence to determine what students were writing. However, with teams of students and multiple artifacts, researchers were less able to identify which paper was being used during a particular phase of the design process. Cameras positioned from an angle overhead might allow association between papers and content of the writing. However, in this research effort, a wide angle video of four students made identifying what was written and when difficult. In addition, the research team noticed that they were able to code feasibility, for example, consistently, but they had difficulty determining exact start and stop times. One research assistant might include a background statement as a lead in to feasibility while another might code a narrower band of feasibility leading to general agreement between researchers but low Kappa values.
From a methodological perspective, the inverse relationship between time spent in problem definition and time spent in feasibility might provide insight into student thinking about problem definition. Though problem definition in the hallway design task was seldom coded, future researchers could use feasibility as a method of extracting student definitions of the problem. Feasibility considerations have inherent value statements that could provide a proxy for problem definition. Therefore, by analyzing student conversation about the feasibility of an idea, a future research team may be able to identify the implicit constraints and criteria that students do not mention explicitly. For example, if a student judges a potential solution to be too expensive, we can infer that cost is a criterion even though it was not mentioned as one. Students mentioned concerns such as slide or platform height or soccer field location in the playground problem, which relate to the constraint presented that the playground must be “safe”. In this example, the students silently operationalized safety by considering a minimum distance from a nearby road to the soccer field. In addition, students’ brainstorming activities may provide insight to the problem definition in that they tend to think of potential solutions and the commonalities across those solutions may be hints into their problem definitions. As an example, students who list different ways of controlling student hallway traffic such as traffic lights, teachers, or mirrors, may suggest implicitly that widening the hallway is not practical or that they feel constrained by the lack of resources to make major structural changes in the school architecture.
Further research might investigate qualitative differences in the ways and methods in which all female teams engaged in problem definition as compared to males and mixed groups. Female groups spent more time and a much higher percentage of time on problem definition than their peer groups. It might prove beneficial to practitioners to understand the impact of this additional time on the design thinking. Does this additional effort represent a deeper understanding, and does a deeper understand cause improved design performance and more effective solution development?
Additional investigation may clarify differences between gender and modeling and communication activities. All female groups spent dramatically less time modeling and communicating which accounted for most of the differences in overall design problem time. Are females modeling and communicating less or differently than their male only and mixed gender team counterparts? Teachers should consider and make explicit modeling and communication expectations so that group efforts are comparable. If the teacher is expecting significant modeling and communication activity, the teacher should be sensitive to the possibility that team gender composition may impact these behaviors. Teachers need to monitor these activities and clearly articulate the importance and purpose of these elements of design so that teams allocate time where the teacher is expecting.
The National Center for Engineering and Technology Education suggested characteristics appropriate for engineering design challenges for high school students ( National Center for Engineering and Technology Education, 2012 ). This study used the characteristics to adopt an instrument used in previous work. However, relevant design problems might not only improve design challenges as measurement instruments but might also fundamentally change the learning and teaching process. Data were not gathered in this study about the typical classroom design problems that were the foundation of student experience. Future research could investigate impacts on student learning related to authenticity and its impact on motivation and engagement. Students who are actively engaged in learning may interact with the design problem very differently and therefore their learning and experiences may be different from student less engaged. In addition, while most students in this research were seniors and each had taken numerous design related courses, the number of design problems and their duration were not studied but could impact student performance. Further study might investigate classrooms that conduct design problems regularly as a pedagogical approach as compared to classrooms which situate design problems only as a capstone experience.
Summary
This study provided seventeen teams, each comprised of 2-4 high school students, with a team based engineering design challenge. Observational protocol analysis was conducted based on a foundation of previous work, including the adoption of previous coding schemes. Differences between groups and individuals were compared. Teams of students were split in two groups; one set of teams received a playground design problem while the other received a hallway design problem. Teams worked up to two hours after school on the design problems and provided the recommendations resulting from their work at the conclusion of the session.
This material is based upon work supported by the National Science Foundation under Grant No. 0426421. Any opinions, findings, and conclusions of recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Nathan Mentzer ( nmentzer@purdue.edu ) is Assistant Professor in the College of Technology and College of Education, Purdue University.
References
Atman, C., Adams, R. S., Cardella, M., Turns, J., Mosborg, S., & Saleem, J. (2007). Engineering Design Processes: A comparison of Students and Expert Practitioners. Journal of Engineering Education, 96 (4).
Atman, C., Chimka, J. R., Bursic, K. M., & Nachtmann, H. L. (1999). A Comparison of Freshman and Senior Engineering Design Processes. Design Studies, 20 , 131-152.
Atman, C., Kilgore, D., & McKenna, A. (2008). Characterizing Design Learning: A Mixed-Methods Study of Engineering Designers' Use of Language. Journal of Engineering Education, 97 (3).
Becker, K., Mentzer, N., & Park, K. (2012). High School Student Engineering Design Thinking and Performance . Paper presented at the ASEE 2012 Annual Conference and Exposition, San Diego: California.
Bursic, K. M., & Atman, C. (1997). Information gathering: a critical step for quality in the design process. Quality Management Journal, 4 (4), 60–75.
Cardella, M., Atman, C., Turns, J., & Adams, R. S. (2008). Students with Differing Design Processes as Freshmen: Case Studies on Change. International Journal of Engineering Education, 24 (2), 246-259.
Carie Mullins, Atman, C., & Shuman, L. (1999). Freshman Engineers' Performance When Solving Design Problems. IEEE Transactions on Education, 42 (4), 281-286.
Creswell, J. W. (1998). Qualitative Inquiry and Research Design . Thousand Oaks: Sage Publications.
Dally, J. W., & Zhang, G. M. (1993). A Freshman Engineering Design Course. Journal of Engineering Education, 82 (2), 83-91.
Hayes, J. R. (1989). The Complete Problem Solver (2nd ed.). Hillsdale:" "Lawrence Erlbaum Associates."
Householder, D. (2011). Engineering Design Challenges in High School STEM Courses: A Compilation of Invited Position Papers Retrieved November 1, 2011, from http://ncete.org/flash/pdfs/Engr%20Design%20Challenges%20Compilation.pdf
Mentzer, N., & Becker, K. (2010). Exploring Engineering Design Knowing and Thinking as an Innovation in STEM Learning . Paper presented at the P12 Engineering and Design Education Research Summit, Seaside: Oregon.
Mentzer, N., Becker, K., & Park, K. (2011). High School Students as Novice Designers . Paper presented at the American Society of Engineering Education, Vancouver, CA.
Morozov, A., Yasuhara, K., Kilgore, D., & Atman, C. (2008). Developing as Designers: Gender and Institutional Analysis of Survey Responses to Most Important Design Activities and Playground Information Gather Questions, CAEE-TR-07-06. Seattle: University of Washington.
Mosborg, S., Adams, R. S., Kim, R., Atman, C., Turns, J., & Cardella, M. (2005). Conceptions of the Engineering Design Process: An Expert Study of Advanced Practicing Professionals . Paper presented at the 2005 American Society for Engineering Education Annual Conference & Exposition, Portland, Or.
Mosborg, S., Cardella, M., Saleem, J., Atman, C., Adams, R. S., & Turns, J. (2006). Engineering Design Expertise Study, CELT Technical Report CELT-06-01. Seattle: University of Washington.
National Academy of Engineering. (2004). The Engineer of 2020 . Washington, DC: The National Academies Press.
National Center for Engineering and Technology Education. (2012). Incorporating Engineering Design Challenges into STEM Courses. In D. Householder & C. Hailey (Eds.). Logan, UT: Utah State University.
National Center for Engineering and Technology Education. (2013). Research Retrieved December 17, 2013, from http://ncete.org/flash/about.php
Newell, A., & Simon, H. (1972). Human Problem Solving . Englewood Hills: Prentice-Hall.
Pearson, G., & Young, A. T. (Eds.). (2002). Technically Speaking: Why All Americans Need to Know More About Technology : National Academy of Engineering.
Polya, G. (1945). How to Solve It: A New Aspect of Mathematical Method . Princeton: Princeton University Press.
Rubenzer, R. (1979). The role of right hemisphere in learning & creativity implications for enhancing problem solving ability. Gifted Child Quarterly, 23 (March 1979), 78-100.