Abstract
Reported in this article are initial results from of a longitudinal study to characterize the design cognition and cognitive design styles of high school students with and without pre-engineering course experience over a 2-year period, and to compare them with undergraduate engineering students.
The research followed a verbal protocol analysis based on the function– behavior–structure (FBS) ontology, which employs a task-independent approach that is distinct from a task-based or an ad hoc approach. This approach to protocol analysis is applicable across any process-based view of designing and generates results based on a common comparative measure independent of the design task.
In this article, Year 1 results are presented comparing only students in their junior year of high school who had formal pre-engineering course experience (experiment group) with those who did not have formal pre-engineering course experience (control group). Specifically, data collected from design sessions were analyzed for comparison of design issues and processes between experiment and control groups, respectively. Results from analysis of Year 1 data did not reveal any significant differences between the experiment and control groups in engineering design cognition. Based on these results, one would conclude that students with pre-engineering course experience do not demonstrate a stronger focus on the process of producing design solutions than do students without such experience. Although analysis of demographic data from high school participants indicates some degree of common prior preengineering experiences, it did not provide a sufficient explanation for why no significant differences in engineering design thinking were found between these groups. The researchers anticipate that Year 2 data will indicate that as the preengineering students continue engaging in formal engineering design experiences during their final year of high school, some degree of difference in design cognition will be demonstrated.
Keywords : design cognition; verbal protocol analysis; high school preengineering.
Background
Engineering design used as an instructional strategy at the PK–12 level is increasingly being embraced as a core learning method and as a pedagogical tool for integrative STEM education ( Kolodner, 2002 , Wells, 2010 ). As a key stakeholder in this trend toward integration of engineering design in K–12 STEM education curricula, it is critical that the elementary and secondary technology and engineering (T/E) education community understands the impact that such experiences have on student development of design practices. Few studies have examined the cognitive characteristics of K–12 students during T/E design-based learning (DBL) activities. Moreover, the way in which secondary students approach the engineering design process is not well understood ( Katehi, Pearson, & Feder, 2009 ; Silk & Schunn, 2008 ) nor is whether that approach differs between students who have engaged in formal engineering experiences through pre-engineering course work and those who have not. Within the context of increasing opportunities for K–12 students to engage in both formal and informal T/E design activities, investigations regarding the extent to which such high school experiences contribute to a student’s capacity for design thinking (cognition) are needed. The intent of the research reported in this article was to characterize the design cognition of high school students and specifically to compare the design practices between high school students with and without formal pre-engineering design experiences.
Though few would argue that the design literature in engineering education has been somewhat singularly focused on pedagogical issues, there is a growing body of literature from studies that seek to understand the characteristics of design thinking behavior from a cognitive viewpoint ( Cross, 2004 ; Lawson, 2004 ). Among these studies, protocol analysis is the research method of choice ( Atman & Bursic, 1998 ; Dorst & Cross, 2001 ) for investigating design cognition and has been the basis for many of the more recent design cognition studies ( Adams, Turns, & Atman, 2003 ; Atman et al., 2007 ; Christensen & Schunn, 2007 ). The research study presented in this article followed a verbal protocol analysis based on the function–behavior–structure (FBS) ontology developed by Gero (1990) and its extension, the situated FBS (sFBS) ontology ( Gero & Kannengiesser, 2004 ), as a design-based coding scheme. The FBS protocol analysis employs a task-independent approach, which is distinct from a taskbased or an ad hoc approach. This approach to protocol analysis is applicable across any process-based view of designing and generates results based on a common comparative measure independent of the design challenge (task). In this way, the FBS protocol analysis addresses the underlying cognitive processes, as opposed to the standard behavior-based analysis, and therefore provides a uniform basis for comparisons between students with different educational preparation and backgrounds and from different educational environments ( Jiang, Gero, &Yen, 2014 ; Williams, Gero, Lee, & Paretti, 2011 ).
Function–Behavior–Structure Verbal Protocol
FBS Ontology
The FBS ontology presents designing as the process of converting a set of functions into a set of design descriptions whereby those descriptions accurately convey an artifact capable of such functions ( Gero, 1990 ). The design process is characterized in the FBS ontology (Figure 1) using three classes of ontological variables—function, behavior, and structure—as well as the external design requirements given the designer and a final description of the designed structure. Modeled in this way, function (F) is defined as the teleology of a designed object, and the behavior of that object is either what is expected (Be) from the structure or derived (Bs) from the structure. The structure (S) of an object represents individual components and the relationships among them. The external design requirements that the designer is given are designated by R, and the resultant set of design descriptions designated by D. These six ontological variables in the FBS model map onto design issues and serve as the basis for design cognition.
A design description is the result of a designer having progressed through a set of eight distinct processes each of which reflects their movements (Numbers 1–8, Figure 1) among the ontological variables. The first five processes reflect an implied linear sequence of movements that include formulation, synthesis, analysis, evaluation, and documentation. In formulation (1), requirements are transformed into functions and functions into a set of expected behaviors; synthesis (2) results in a proposed structure to satisfy expected behaviors; analysis (3) of the proposed structure produces derived behaviors; in evaluation (4), both expected behavior and behavior derived from structure are concurrently assessed; and documentation (5) generates the design description. The iterative nature of designing is captured in the movement among three types of reformulation processes, which are also denoted numerically in Figure 1: Reformulation I (6) is the reformulation of structure; Reformulation II (7) is a reformulation of expected behavior; and Reformulation III (8) is a reformulation of function.
FBS Coding Scheme: Design Issues and Processes
The coding scheme adhered to in this research is based on this FBS ontology whereby the ontological variables are translated into six design issues. These design issues are coded using the FBS ontology, as exemplified in the sample of participant utterances and associated codes seen in Table 1. The selected utterances were drawn from an engineering design session in which high school participants were asked to design a device that would assist elderly clients in opening a stuck double-hung window. Transformations between the six codes used to label the design issues reflected in participant utterances generate the eight distinct design processes (Table 2).
Table 1
FBS Coding Examples
| Design Issues | Respective Utterance Example |
|---|---|
| Design Requirements (R) | "so they need help in trying to.... for the elderly to raise windows"; "it says a significant amount of force to raise and lower the windows..." |
| Function (F) | "but it'd have to be something that is really easy to twist."; "causes the window to expand on the frame" |
| Behavior Expected (Be) | "that will increase mechanical advantage"; "that may help the elderly lift or..."; |
| Behavior from Structure (Bs) | "so if they like pull the string it actually lifts it"; "so the longer this is the more mechanical advantage you'll have so the easier it will be" |
| Structure (S) | "So one thing I came up with is to cut a notch in the bottom frame of the window right there"; "and have the strings coming back down" |
| Design Description (D) | "let's draw a right side view of this thing to explain it okay I'll let you do that..." |
Unidirectional transformational movements are indicated by the "→" symbol, the "↔ " symbol indicates transformational comparisons, and the numbers associated with each design issue correspond to those depicted in the FBS model (Figure 1).
Table 2
FBS Design Processes
| Progression | Design Process | Transformational Movement |
|---|---|---|
| (1) | Formulation | R → F, F → Be |
| (2) | Synthesis | Be → S |
| (3) | Analysis | S → Bs |
| (4) | Evaluation | Be ↔ Bs |
| (5) | Documentation | S → D |
| (6) | Reformulation I | S → S |
| (7) | Reformulation II | S → Be |
| (8) | Reformulation III | S → F |
Method
The research design followed a two-by-two factorial investigation across two exogenous variables, design experience and maturity, in which experience is formal pre-engineering coursework and maturity was the time between data collected fall of the junior and senior years of high school. The full scope of the research was to characterize the design cognition and cognitive design styles of high school pre-engineering students over a 2-year period and to compare them with undergraduate engineering students as well as high school students without such design experience. Presented in this article are Year 1 results comparing only the high school participants and only addressing the following hypothesis, which was one of six hypotheses posed in this study: High school preengineering students have a stronger focus than high school students with no design experience on the design process of synthesis (i.e., the process of producing solutions).
Using purposeful selection, high school students in their junior year were assigned to experiment (those with formal pre-engineering course experience) and control (those without formal pre-engineering course experience) groups. In teams of two (dyads), students engaged in a predefined engineering design task in which they were to develop a design-only solution. A dyad configuration was used because it has been found to naturally promote authentic verbal interactions during collaborations on developing acceptable engineering design solutions ( Kan & Gero, 2009 ; Purzer, Baker, Roberts, & Krause, 2008 ).
Participants
Participants were drawn from a convenience sample of high school juniors attending one of three rural, mid-Atlantic high schools that offered the same ninth through twelfth grade Project Lead the Way (PLTW) pre-engineering course sequence. Student populations at each of the participating schools were of similar size. Two groups of participants, those with (experiment) and those without (control) formal PLTW pre-engineering course experience, were recruited from each high school, using a small monetary incentive. Prior PLTW course experience for the experiment group ranged from those enrolling in their first PLTW course at the start of their junior year to those with one full year of prior PLTW coursework. Within groups, students self-selected into dyads, 60% of which were mixed-gender. Of the 40 students participating in Year 1, the gender distribution within the experiment group was 64% male and 36% female, and for the control group, it was 65% male and 35% female.
Procedures
Participant recruitment was conducted using typical modes of school communication. Student demographic data (e.g., age, gender, pre-engineering course experience) were collected as students arrived at their session and before dyads engaged in the design task. The design task dyads addressed was that of designing a solution to assist physically impaired elderly nursing home residents with opening difficult-to-open, double-hung windows. Instructions for completing the design task were provided as well as basic information resources regarding the construction and operation of a double-hung window. Dyads were allowed 45 minutes to collaborate on their design task and instructed to include a detailed sketch of the final solution on a whiteboard.
Data Collection and Protocol Analysis
The following sequence of tasks presents the basic set of procedures used for data collection and protocol analysis.
Video Capture . Each design task session was captured using two video cameras that were arranged at two distinct vantage points ensuring sufficient recording of dyad interactions and their development of a final design description (Figure 2). Additionally, the two video cameras safeguarded against potential technological issues or difficulties encountered by either device. The first camera directly captured the white board and dyads engaged in progressive design development and sketching of their solution, while second camera recorded a general view of the entire design session. Both members of a dyad were equipped with a high-sensitivity wireless microphone to ensure that quality audio was captured for successful transcription of student verbalizations into text. The resulting videos provide a time-stamped recording of the entire design session.
Transcription . Video recordings of dyad design sessions were transcribed manually with individual utterances from each dyad member entered verbatim into alternating rows of a spreadsheet. Timestamps were inserted every three minutes to establish reference points throughout the entire video. This approach to transcription resulted in a written version of the verbalizations between participants with time stamps throughout.
Segmentation and coding of text-based verbalizations . The method used to segment the text-based version of dyad verbalizations was conducted on the basis of the FBS coding that was previously described. This method involved concurrent analysis of a given transcript by independent coders. A total of six coders were involved with coding the 40 protocols. All coders participated in training using practice protocols until consistently achieving sufficient intercoder reliability. Coders segmented and coded simultaneously, dividing the utterances until each individual segment contained a single code that reflected only one of the six possible design issues ( Kan & Gero, 2007 ). The use of two independent coders ensured robustness and demonstrated an intercoder reliability ranging from 85% to 95%, which was consistent with prior research ( Williams et al., 2011 )
.Arbitration . After independent coders completed the segmentation and coding of a given transcript, they would meet to arbitrate—compare, discuss, and justify—the FBS codes that they assigned to each segment. When agreement of independently coded segments occurred, a final code was assigned. Segments that differed in assigned codes required coders to engage in arbitration to dispute the assigned coding and reach agreement on the design issue addressed. If coders were unable to agree on an arbitrated code, that segment was left uncoded and was highlighted for subsequent final arbitration between the lead researchers. The final arbitration resulted in a final protocol data set that was readied for use in statistical analyses. The number of segments typically generated from the final protocol for a 45-minute design session was between 200 and 700. Because there are six codes, this implies that, on average, each code would likely appear at least 33 times. This provides a statistically significant data set. Analyses of final arbitrated protocols were conducted using LINKODER ( www.linkoder.com ) to generate descriptive statistics and probability analyses of the FBS ontology. Data were analyzed to determine statistical differences in design issues and processes between the control and experiment groups.
Results
In this article, we report on the analyses of the first year data collected from design sessions of participating high school juniors. These data were analyzed for comparison of design issues and processes between experiment and control groups, pre-engineering (ENG) and nonengineering (NON), respectively.
Design Issues
A comparison of design issue distributions between ENG and control NON groups is illustrated in Figure 3, and descriptive statistics are presented in Table 3. The percent occurrence reflects the average within group frequency of segments associated with each of the six design issues for both groups. The data indicate that both groups expended the majority of their cognitive efforts (~ 40%) in discussions of the design structure (S), which is typical for most designers. Relatively similar total percent effort (~26–28) was expended on behavior from structure (Bs) and expected behavior (Be) combined (~14–18). Comparisons of control and experiment group data using a t -test (Table 3) revealed no significant differences among any of the design issues, although expected behavior approached it. Similarly, comparison of the total effort expended in the problem versus the solution space (P–S Index; Jiang et al., 2014) indicated that there were no significant differences.
Table 3
Statistical Results of Design Issues (Entire Session): ENG vs. NON High School
Juniors
| Design Issue | t - value (%) | p - value |
|---|---|---|
| (R) Requirement | -0.78 | 0.240 |
| (F) Function | 1.05 | 0.153 |
| (Be) Expected Behavior | -1.7 | 0.053 |
| (Bs) Behavior from Structure | 0.43 | 0.334 |
| (S) Structure | -0.23 | 0.410 |
| (D) Description | 1.09 | 0.145 |
| P-S Issue Index | -1.2 | 0.123 |
Design Processes
The distribution of syntactic design processes was computed to discern differences in the cognitive effort expended between control and experiment groups. Similar to computations of design issues, analytical comparisons of the eight syntactic design processes showed no statistically significant differences between ENG and NON groups (Table 4). No statistically significant differences were observed in the P–S Processes Index between these two groups.
Table 4
Statistical Results of Design Processes (Entire Session): ENG vs. NON High
School Juniors
| Design Issue | t - value (%) | p - value |
|---|---|---|
| Formulation | 1.22 | 0.118 |
| Synthesis | -1.01 | 0.163 |
| Analysis | 1.48 | 0.077 |
| Evaluation | -0.16 | 0.436 |
| Documentation | 0.82 | 0.211 |
| Reformulation I | -0.5 | 0.311 |
| Reformulation II | -1.59 | 0.064 |
| Reformulation III | 0.55 | 0.295 |
| P-S Process Index | -0.92 | 0.183 |
Percent occurrences for the eight design processes (Figure 4) indicate that roughly 30% of their cognitive effort was invested in Reformulation I (S > S) and between ~17–21% on Analysis (B e <>B s ).
Discussion and Conclusions
Analysis of Year 1 data did not reveal any significant differences between the experiment (ENG) and control (NON) groups in engineering design cognition. Based on these results, the underlying hypothesis must be rejected: Pre-engineering students do not demonstrate a stronger focus on the process of producing design solutions. To further investigate this apparent lack of difference between ENG and NON groups, the following select demographic data related to prior T/E design experiences were collected: participation in (a) middle school technology education classes, (b) T/E clubs, (c) other T/E-related activities, and because of the rural school settings, (d) farm-related activities. Analysis of these data indicated that of the ENG students, 59% had previously participated in middle school technology education classes, 14% were or had been involved in T/E clubs, 30% engaged in other T/E-related activities, and 30% had T/E-related farm experiences. In each of these demographic categories, students in the NON group had significantly less additional formal or informal T/E-related experiences, 33%, 5%, 17%, and 0% respectively. It is evident from these demographic data that students in the ENG group had far more formal and informal T/E-related experiences.
Although demographic data indicates some degree of common prior preengineering experiences, it does not provide sufficient explanation for finding no significant differences in engineering design thinking between these groups. Other influences such as curricular and pedagogical factors must therefore be considered. Project Lead the Way (PLTW) program documents present entrylevel course outlines that do not specifically target design thinking as a learning goal ( https://www.pltw.org/our-programs/engineering ). This is equally the case for the curriculum used by the middle school technology education programs at participating schools. The initial PLTW course that all pre-engineering participants engaged in was Introduction to Engineering Design (IED). A review of the detailed IED curriculum outline indicates that instructional units give attention to teaching the following set of practices and steps in the design process: technical sketching and drawing skills, modeling skills, geometry of design, documentation, and completion of a prescribed design project using computer-aided design (CAD) software. Authentic open-ended design challenges are not integral to the learning experience provided to students in this entry-level pre-engineering course. In light of this, it suggests that the pedagogical preparation provided to educators delivering the earlier courses in PLTW might not be adequate for intentionally incorporating or promoting design thinking as part of pre-engineering experiences.
Year 2 data of this longitudinal study are currently being collected. The researchers anticipate that as the pre-engineering students continue their engagement in engineering design experiences during their final year of high school, differences in design cognition will be demonstrated to some degree.
Acknowledgment
The authors wish to acknowledge and thank Brooke Bellows, Yoon Lee, Tyler Love, and Mick Smythwood for segmentation and coding of data. Appreciation is also extended to Pinelopi Kyriazi for conducting the statistical testing. This material is based upon work supported by the National Science Foundation under Grant No. 1160345. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This study has been approved by the authors’ respective Institutional Review Boards.
References
Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering designers: The role of reflective practice. Design Studies , 24 (3), 275–294. doi:10.1016/S0142-694X(02)00056-X
Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosberg, S., & Saleem, J. (2007). Engineering design processes: A comparison of students and expert practitioners. Journal of Engineering Education , 96 (4), 359–379. doi:10.1002/j.2168-9830.2007.tb00945.x
Atman, C. J., & Bursic, K. M. (1998). Verbal protocol analysis as a method to document engineering student design processes. Journal of Engineering Education , 87 (2), 121–132. doi:10.1002/j.2168-9830.1998.tb00332.x
Christensen, B. T., & Schunn, C. D. (2007). The relationship of analogical distance to analogical function and pre-inventive structure: The case of engineering design. Memory & Cognition , 35 (1), 29–38. doi:10.3758/BF03195939
Cross, N. (2004). Expertise in design: An overview. Design Studies , 25 (5), 427– 441. doi:10.1016/j.destud.2004.06.002
Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem-solution. Design Studies , 22 (5), 425–437. doi:10.1016/S0142-694X(01)00009-6
Gero, J. S. (1990). Design prototypes: A knowledge representation schema for design. AI Magazine , 11 (4), 26–36. doi:10.1609/aimag.v11i4.854
Gero, J. S., & Kannengiesser, U. (2004). The situated function–behavior– structure framework. Design Studies , 25 (4), 373–391. doi:10.1016/j.destud.2003.10.010
Jiang, H., Gero, J. S., & Yen, C.-C. (2014). Exploring designing styles using a problem–solution division. In J. S. Gero (Ed.), D esign computing and cognition ’12 (pp. 79–94). Dordrecht, the Netherlands: Springer. doi:10.1007/978-94-017-9112-0_5
Kan, J., & Gero, J. S. (2007). Using the FBS ontology to capture semantic design information. In J. McDonnell & P. Lloyd (Eds.), DTRS7 (155–165). London, England: University of the Arts.
Kan, J. W. T., & Gero, J. (2009). Using the FBS ontology to capture semantic design information in design protocol studies. In J. McDonnell & P. Lloyd (Eds.), About: Designing. Analysing design meetings (pp. 213–229). Boca Raton, FL: CRC Press.
Kannengiesser, U., Gero, J., Wells, J., & Lammi, M. (2015). Do high school students benefit from pre-engineering design education? Proceedings of the 20th International Conference on Engineering Design (ICED), Vol 11, pp. 267-276: Human Behaviour in Design, July 27-30, Milan, Italy.
Katehi, L., Pearson, G., & Feder, M. (Eds.). (2009). Engineering in K–12 education: Understanding the status and improving the prospects . Washington, DC: National Academies Press. doi:10.17226/12635
Kolodner, J. L. (2002). Facilitating the learning of design practices: Lessons learned from an inquiry into science education. Journal of Industrial Teacher Education , 39 (3), 9–40. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v39n3/kolodner.html
Lawson, B. (2004). Schemata, gambits and precedent: Some factors in design expertise. Design Studies , 25 (5), 443–457. doi:10.1016/j.destud.2004.05.001
Purzer, S. Y., Baker, D., Roberts, C., & Krause, S. (2008). Development of a team interaction observation protocol and a self-efficacy survey using social cognitive theory as a framework. In Proceedings of the 2008 American Society for Engineering Education Annual Conference & Exposition (pp. 13.415.1–13.415.15). Washington, DC: American Society for Engineering Education. Retrieved from https://peer.asee.org/3203
Silk, E. M., & Schunn, C. D. (2008, January). Core concepts in engineering as a basis for understanding and improving K-12 engineering education in the United States . Paper presented at the National Academy of Engineering/National Research Council Workshop on K-12 Engineering Education,, Washington, DC.
Wells, J. G. (2010). Research on teaching and learning in science education: Potentials in technology education. In P. A. Reed & J. E. LaPorte (Eds.), Research in technology education (pp. 192–217). 59th Yearbook of the Council on Technology Teacher Education. Reston, VA: Council on Technology Teacher Education.
Williams, C. B., Gero, J., Lee, Y., & Paretti, M. (2011). Exploring the effect of design education on the design cognition of mechanical engineering students. In Proceedings of the ASME 2011 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference (pp. 607–614). Washington, DC: American Society of Mechanical Engineers. doi:10.1115/DETC2011-48357
About the Authors
John Wells ( jgwells@vt.edu ), is Associate Professor in the Integrative STEM Education Program at Virginia Polytechnic Institute & State University.
Matthew Lammi ( matthew_lammi@ncsu.edu ), is Assistant Professor in the College of Education at North Carolina State University.
John Gero ( jgero@gmu.edu ), is Research Professor in the Krasnow Institute for Advanced Study at George Mason University & University of North Carolina.
Michael E. Grubbs ( mgrubbs@bcps.org ), is Supervisor of Technology and Engineering Education for Baltimore County Public Schools.
Marie Paretti ( mparetti@vt.edu ) is Associate Professor and Co-Director of the Virginia Tech Engineering Communication Center at Virginia Polytechnic Institute & State University.
Christopher Williams ( cbwilliams@vt.edu ) is Associate Professor and Senior Faculty Fellow in the Department of Mechanical Engineering at Virginia Polytechnic Institute & State University.