Cognitive Processes of Students Participating in Engineering-focused Design Instruction

Todd R. Kelley

Introduction

The engineering education community has identified the important role K- 12 education plays in the success of post-secondary engineering education. Teaching engineering content in technology education programs has become a recent popular trend with curriculum initiatives such as Project Lead the Way, but some states, like New York, have had a course called “Principles of Engineering” since the late 1980s ( Lewis, 2005 ). Teaching engineering design in K-12 might possibly be good for post-secondary engineering education, but does it produce technological problem solvers who have the ability to properly manage an ill-defined problem and develop viable solutions?

Roberts emphasized “the purpose of teaching design is not to bring about change in the made world, but change in the student’s cognitive skills” ( 1994 , p. 172). Furthermore, ill-defined problems are more difficult to solve since they require more cognitive operations than simpler, well-defined problems ( Jonassen, 2000 ). Johnson (1992) suggested a framework for technology education curricula, which emphasizes intelligent processes. “Students should acquire a repertoire of cognitive and metacognitive skills and strategies that can be used when engaged in technological activity such as problem solving, decision making, and inquiry” ( Johnson, 1992 , p. 30). Cognitive and metacognitive skills are important thinking processes required for problem solving, and these skills should be taught to students in technology education courses. Careful examination of the cognitive processes employed by students as they work through an ill-defined technical problem provides a means of evaluating the effectiveness of a curriculum approach designed to develop effective problem solvers.

Clearly, engineering-focused programs using a classic engineering design process model approach the design process differently than technology education programs using the design process featured in the Standards for Technological Literacy ( Hailey, Erekson, Becker, and Thomas, 2005 ). The most notable difference in the design process is that engineering design uses analysis and optimization for the mathematical prediction of design solutions. In contrast, the technology design process emphasizes selecting a design idea, testing the idea through model building, and making final design decisions based upon a trial and error process. These vast differences in the approaches to design causes one to wonder if students from these technology education approaches to design instruction will be able to solve ill-defined problems using an engineering design process. Moreover, although both PLTW and the NCETE seek to develop engineering-focused design, the purposes of these programs are different. Consequently, so are their approaches. While Project Lead the Way ( Project Lead the Way, 2006 ) is described as a pre-engineering program, the National Center for Engineering and Technology Education seeks to develop activities to infuse engineering design into technology education ( Hailey, et al., 2005 ). Both engineering-focused approaches to design instruction seek to provide students with a systematic problem solving method through the application of the engineering design process, but will high school students from these two different groups perform differently when solving the same ill-defined problem? The purpose of this research was to determine if these two different approaches to engineering-focused design instruction will affect how students solve ill-defined problems.

Research Questions

It is critical to closely examine these important questions as the field of technology education considers engineering design as a focus alongside the need for developing technological literacy in K-12 learners, a notion supported by leaders in the field of technology education ( Daugherty, 2005 ; Lewis, 2004 ; Wicklein, 2006 ). This research examined how a high school student who has learned engineering design solves an assigned ill-defined technical problem. This insight can be helpful to develop further curriculum in technology education that will develop individuals who are technologically literate and effective problem solvers. Another benefit of this study is to gain insight into how a high school student, who has learned engineering design methods, manages cognitive processes as he or she engages in problem solving when confronted with a time constraint. Finally, it is beneficial to identify where students fail to properly manage cognitive strategies and to identify what cognitive strategies are not utilized in the problem solving process.

Participants

This research study examined students participating in two different engineering-focused design instruction: Project Lead the Way and a technology education program seeking to impart engineering design (NCETE partner). For the latter group, four participants were drawn from programs of a participating teacher in NCETE in-service workshops conducted at North Carolina A&T University. Three subjects were selected from Project Lead the Way schools by recommendation from North Carolina A&T NCETE partners. The Project Lead the Way participants completed the course Principles of Engineering and were currently enrolled in the capstone course titled Engineering Design and Development, which is typically taught to seniors in high school. The participants selected from a technology education high school program not using Project Lead the Way curriculum were students who were taught by an during the summer of 2006. The participants from both groups were selected by their instructors for their problem solving abilities and willingness to participate in the study. It is important to note that the NCETE partnered school was currently generating new curriculum with a focus on engineering design which is why many course titles may not appear to reflect an engineering design focus; see Appendix B (available online at scholar.lib.vt.edu/ejournals/JTE/ ). The researcher selected participants for both groups who were homogeneous in educational background by requiring the same criteria for the prerequisites of mathematics and science as defined by the Project Lead the Way program ( Project Lead the Way, 2006 ). The researcher conducted the study near the end of the semester so the participants gained as much training on engineering design as possible. Demographic information for the participants can be found in Appendix B & C. General demographic information about the instructors, curriculum, class size, and course titles can be found in Appendix D & E. (all appendices available online at scholar.lib.vt.edu/ejournals/JTE/ )

Methodology

This study compared the cognitive processes used by the participants from the two curricular approaches to technology education as they used a design process to work through an ill-defined technical problem. The same ill-defined technical problem was presented to all the participants. Each participant was asked to carefully read the technical problem, identify all constraints he or she imposed on the problem, and then asked to begin to develop a solution. Each participant worked in isolation from other participants or classmates. The study used a “think-aloud” protocol method used in similar studies ( Ericsson & Simon, 1993 ; Kruger & Cross, 2001 ; van Someren, van de Velde, & Sandberg, 1994 ). Atman & Bursic (1998) suggested that using a verbal protocol analysis for assessing cognitive processes of engineering students is a powerful method to understand the process students take when developing a design solution. Atman and Bursic stated: “analysis of a verbal protocol enables us to look at a subject’s process in detail rather than simply ‘grading’ a final solution. That is, we can now grade the ‘process’ as well as the final design” ( Atman & Bursic, 1998 , p. 130). Moreover, verbal protocol analysis has been endorsed as a sound method for capturing and assessing engineering student’s design processes ( Atman & Bursic, 1998 ). Consequently, the participants were asked to verbalize their thoughts as they worked through the ill-defined problem. The researcher prompted participants to keep talking through the problem when he or she stopped verbalizing his or her thoughts; beyond this, the researcher did not interact with the participants. The participants were given a total of 30 minutes to work through the early stages of the engineering design process; however, several participants’ sessions did not use the entire time. Although this time constraint limited engagement in the engineering design process, it was adequate to study how the student framed the problem and began to develop an initial design plan. The data collection included frequency and duration of time of the various mental processes allowing the researcher to break coding data into units of time including time on code, total time on each code, percentage of time, and total time of the testing session. This method of organizing data by time has been used in similar problem solving studies ( Welch, 1999 ). Frequency was also recorded, tallying each iteration of the cognitive strategy used by the participant. Group mean scores were computed and reported for all cognitive processes used for both groups (see Tables 3 & 4 ).

The open-ended problem that was given to the participants described typical conditions in underdeveloped areas of the world where the domestic water is often transported by women and girls. This activity often causes physical stress on these women and children, resulting in acute medical conditions. The problem statement provided some general information about current constraints on this problem as well as solutions that are currently being employed. The statement asked the participants to provide details about how they would proceed to develop strategies to improve the current conditions in these underdeveloped areas. The participants were asked to list all constraints that they imposed on the problem. The problem that the participants were asked to solve is presented in Figure 1.

Framing the Problem

This study only examined the early stages of the design process. Certainly in the time constraint of thirty minutes, a student was unlikely to reach the final stages of the design process; therefore he or she was also unlikely to employ all of Halfin’s (1973) mental processes. However, one of the most important stages of the engineering design process occurs at the onset of being presented with a technical problem: ‘framing the problem’ is this important stage. Experts in the field of design identify that framing the problem is a critical step to the design process and occurs as soon as the designer is presented with a technical problem ( Dym, Agogino, Eris, Frey, & Leifer, 2005 ; SchÖn, 1983 ). This early stage of the engineering design process often finds engineers seeking to locate the problem space where the search for the solution begins, starting conditions are identified, and goals are stated. This problem space creates a partial structure from which a solution space can be formed. The solution space structure begins to be developed as ideas are generated; this structure is transferred back to problem space to again consider solution implications. This method seeks to generate cohesion of problem and solution ( Cross, 2004 ).

Data Gathering and Analysis

The participants were videotaped for further analysis by the researcher. The tape was used to record each participant’s voice as he or she verbalizes their thoughts, as well as to record any actions such as sketching, measuring, or any other non-verbal cues. Cross (2004) indicated that one weakness of the ‘think aloud’ verbal protocol method was that it was extremely weak at capturing nonverbal thought processes, using observation in combination with the ‘think aloud’ method was employed to help capture non-verbal cues. This technique of combining a verbal protocol with a video of the testing session is known as observational protocol and is a data collection method used to assess student design and problem solving strategies ( Laeser, Moskal, Knecht, & Lasich, 2003 ). The data collection included frequency and duration of time of the various mental processes.

www.mexmission.com/images/typical/people

Problem

In certain underdeveloped areas of the world the majority, if not all, of domestic water is transported by women and young girls, causing considerable physical stress and resulting in medical conditions that are particularly acute during child-bearing and birth. Small villages are scattered throughout rural areas of the world where this has become a major issue, in part due to the steep mountainous terrain.

Currently, water is typically held in plastic or metal vessels and carried in the arms, balanced on the head, or attached to the ends of a rod and carried across the shoulders. Families who can afford beasts of burden (mules, camels, cattle, etc) employ them in this activity, although this is the exception.

Cultural and political constraints often hinder installation of modern water management systems; therefore temporary measures are needed to improve current conditions.

Your Task:

Describe how you would proceed from this problem statement in order to improve the current condition in these underdeveloped areas. Please list all constraints that you impose on this problem. As you work through this problem, ‘think aloud’ your strategies for deriving a solution.

This research study focused on cognitive processes from a list of 17 mental processes that were identified by Halfin (1973) . Halfin used writings from ten high-level designers including Buckminster Fuller, Thomas Edison, and Frank Lloyd Wright. Halfin used a Delphi technique to identify mental processes that were universal for these expert engineers and designers. Hill (1997) developed a computer analysis tool called the O bservation P rocedure for T echnology E ducation M ental P rocesses (OPTEMP) to assess problem-solving activities in technology education by employing Halfin’s code of mental processes. The study herein used a revised and updated OPTEMP computer program to assist in coding and recording the frequency and duration of time of the cognitive processes employed by students as they worked through the selected ill-defined technical problem. The researcher coded the actions and cognitive processes used by each participant as he or she worked through the technical problem. The number of frequencies and the time spent on each strategy were compiled and a total was recorded in the OPTEMP output.

Microsoft Excel software was used to process the data files generated by the OPTEMP program. Careful analysis of the percentage of time and frequency spent on the various cognitive strategies provided insight into mental processes employed by the students as they worked to frame the ill-defined problem as well as a comparison of group means scores.

Findings

Although a thirty-minute or shorter examination is inadequate in understanding the entire process taken by problem solvers, it can provide great insight into an individual’s ability to organize the problem, constraints, and criteria in order to begin developing a solution. Importantly, the reader is reminded that the findings of this study are very limited in their generalizability.

Are students in these different programs using similar cognitive processes as they solve ill-defined problems?

The research revealed that both groups used similar cognitive strategies as they worked though the ill-defined problem. Both groups employed at least six of the ten mental processes that were identified in the test sessions. The cognitive strategy analysis (AN) was the most common mental processes employed. This code was recorded when the researcher witnessed the participant breaking down the problem and identifying constraints and criteria. The participants spent from 19 to 54 percent of their time doing this. The group mean was 10.70 minutes for the PLTW group and 7.42 minutes for the NCETE group. The duration of time that the two groups spent on the various strategies varied considerably (See Tables 2 -4).

Will students in these programs perform similarly when presented with the same ill-defined problem to solve?

The results of this research revealed that the two groups did perform differently with respect to time spent developing solutions (coded DE). Often this mental process is considered the most critical in determining how an individual designs a solution. Kruger and Cross (2001) proposed that designers are either solution driven or problem driven. Welch and Lim (2000) have noted that novice designers often become stuck in the problem space and fail to generate solutions. The results of this study reveal that group NCETE partner group spent more time generating solutions than the PLTW group. The NCETE group spent from 18 to 32 percent of their time designing and talking about solution ideas. In contrast, the PLTW group only spent from 3 to 8 percent dialoging design solutions. Comparing the group means, the NCETE group spent an average of 5.40 minutes generating design solutions in contrast to an average of 1.77 minutes spent by the PLTW group. Although creative designers are known for generating multiple solutions, there is a danger in generating solutions too quickly due to an incomplete understanding of the problems ( Welch, 1999 ). It is important to consider that while the NCETE group spent more time generating solutions, the PLTW group spent a considerable amount of time defining and analyzing the problem. Comparatively, architects are problem solvers who generate multiple solutions to design problems, whereas engineers are often trained to locate a single solution that works in a timely and cost effective manner ( Akin, 2001 ). Although participant number six developed only one design idea, eight frequency counts are reported ( Table 2 ) and represent discussions of a single design idea. Participant number six was convinced that the idea was the best solution, possibly based on his knowledge of similar cultures who have struggled with this problem. Ball, Ormerod, & Morley (2004) refer to this approach to solving problems as “case-driven” and refer to it as a novice designer approach. The case-driven approach is used to quickly move to a solution by recognizing the similarity of the current problem to a problem encountered in the past and to apply a solution from the earlier problem. Conversely, Cross (2004) suggested that expert problem solvers with experience in designing move quickly from the problem frame to proposing a solution. Considering that this participant spent a great deal of time identifying the constraints and criteria (analysis) and very little time simply defining the problem, he may be demonstrating his ability to design quickly and efficiently as opposed to lacking creative idea generation (See Table 3 ).

Of Halfin’s 17 mental processes, seven processes were never employed by either group. A close examination of the seven missing processes resulted in a a logical explanation for most of them. For example, models/prototypes (code MP) were never employed, quite possibly due to the limited time constraints and lack of available modeling materials. Actually, use of models and prototypes was not expected by the researcher at this stage of the design process. Interpreting data (ID) was not often employed by participants (only one participant used it to a very limited extent) in this study. This is likely due to the fact that there were little data to interpret from the problem statement.

Measuring (ME) was a mental process that could be applied to this ill-defined problem if a heuristic (as suggested by Koen, 2003 ) was applied to the constraints presented in the problem. However, none of the participants employed this strategy. Measuring, as defined by Halfin is “the process of describing characteristics (by the use of numbers) of a phenomenon problem, opportunity, element, object, event, system, or point of view in terms, which are transferable” ( 1973) . Considering that a major distinction between the technology and engineering design processes is that engineering design applies mathematical prediction and optimization, this missing cognitive process is significant. The absence of this cognitive strategy causes one to speculate whether or not students in an engineering-focused design program have any increased ability or need to use mathematics to predict design solution compared to students from technology education programs without an engineering design focus, at least with respect to solving an ill-defined problem. Thus, this study does not support the notion that students in an engineering-focused program apply mathematical prediction and optimization in their problem solving. The other missing cognitive processes from both groups included creating (CR), experimenting (EX), observing (OB), testing (TE) and visualizing (VI).

Are there important cognitive processes missing from students’ performances in both groups?

As mentioned above, measuring (ME) was never utilized by any participant in the study. Computing (CO) was only used by two participants, one from each group applied a quantity to estimate potential distances traveled or the altitude of the mountain terrain. However, no participants used estimations to predict the results of design solutions. This has been identified as a missing piece in the technological design process ( Hailey, et al., 2005 ; Wicklein, 2006 ). The minimal use of this cognitive strategy should be a concern for those who believe students in engineering related programs have the ability to apply their math skills to predict design solutions.

Reliability

The measure of inter-coder reliability revealed a high degree of consistency. Two researchers independently coded 10 % of four of the seven protocols as outlined by Evans (1995) . Segments were selected at the beginning, middle, and at the end of the assessed protocols to ensure that the reliability check was conducted at various stages of the testing session. The total times that each coder ascribed to Halfin’s mental processes are presented in Table 5. Standard deviations ranged from .523 for Analysis to .092 for Managing and Predicting.

Discussion

As the field of technology education has been moving to include engineering, a variety of new curriculum projects have emerged. Some examples of curriculum projects include Project Lead the Way, and ITEA’s Engineering by Design, Engineering the Future, and Engineering is Elementary. As these engineering oriented programs are implemented into schools and new curriculum is implemented, it is important to evaluate their effectiveness in increasing students’ cognitive abilities with respect to problem solving. One way to do this is to examine students as they work to solve ill-defined problems. The method used in this study can provide a heightened awareness of what is really happening in the minds of the students as they work to solve a problem. Technology education programs have often emphasized design and problem solving ( Flowers, 1998 ; Foster, 1994 ; Plaza, 2004 ), but little research has been done to determine how effective these activities are in developing skills, skilled problem solvers, and excellent designers ( Lewis, 1999 ). More research needs to be conducted in technology education to examine the cognitive capabilities of students and observational protocols are a sound methodology that is cost effective. According to the results of this study, students do perform differently with respect to solving ill-defined problems when grouped by engineering-focused programs. Additional research should be done to extend the results of this study by increasing the sample size and expand the sample to include other technology education programs with and without an engineering focus. It is critical for the field of technology education to consider the characteristics and outcomes it would like to develop in its students. Among these outcomes are students who are creative problem solvers who can generate multiple solutions on the one hand or problem solvers who can quickly locate the most efficient and cost effective solution on the other hand. Certainly, a case can be made for both types of problem solvers, quite possibly a blend of experiences in problem solving would be appropriate for the field to consider as the integration of engineering design continues.

Acknowledgement

The author acknowledges Drs. Roger Hill and David Gattie of the University of Georgia for their guidance in the design of this study.

References

Atman, C. J. & Bursic, K.M. (1998). Verbal protocol analysis as a method to document engineering student design processes. Journal of Engineering Education, 87 (2),121-132.

Akin, O. (2001). Variant in design cognition. In C. Eastman, M. McCracken, & W. Newstetter (Eds.), Knowing and learning to design: Cognition in design education (pp.) Elsevier Science Press.

Ball, L. J., Ormerod, T. C., & Morley, N. J. (2004). Spontaneous analogizing in engineering design: A comparative analysis of expert and novices. Designing Studies, 25 (5), 495-508.

Cross, N. (2004). Expertise in design: an overview. Design Studies, 25 (5), 427- 441.

Daugherty, M. K., (2005). A changing role for technology teacher education. Journal of Industrial Technology Education, 42 (1), 41-58.

Douglas, J., Iversen, E., & Kalyandurg, C. (2004). Engineering in the K-12 classroom: An analysis of current practices and guidelines for the future. A production of the ASEE Engineering K12 Center.

Dym, C. L., Agogino, A. M., Eris, Ozgur,E., Frey, D. D., & Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education, 95 (1),103-120.

Ericsson, K. A. & Simon, H. A. (1993) Protocol analysis: Verbal reports as data , MIT Press: Cambridge, MA.

Evans, J. St. B. T. (1995). Relevance and Reasoning. In S. E. Newstead, & J. St. B. T. Evans (Eds.) Perspectives on thinking and reasoning: Essays in honor of Peter Waton (pp. 147-172). Howe, UK: Lawrence Erlbaum Associates Ltd.

Flowers, J. (1998). Problem solving in technology education: A taoist perspective. Journal of Technology Education, 10 (1), 20-26.

Foster, P. N. (1994). Technology education: AKA industrial arts. Journal of Technology Education, 5 (2), 15-30.

Hailey, C. E., Erickson, T., Becker, K., & Thomas, T. (2005). National center for engineering and technology education. The Technology Teacher, 64 (5), 23-26.

Halfin, H. H. (1973). Technology: A process approach. (Doctoral dissertation, West Virgina University, 1973) Dissertation Abstracts International, (1) 1111A.

Hill, R. B. (1997). The design of an instrument to assess problem solving activities in technology education. Journal of Technology Education, 9 (1), 31-46.

International Technology Education Association. (2000). Standards for technological literacy: Content for the study of technology. Reston, VA: Author.

Johnson, S. D. (1992). A framework for technology education curricula which emphasizes intellectual processes. Journal of Technology Education, 3 (2), 26-36.

Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48 (4), 63-85.

Koen, B. V. (2003). Discussion of the method: Conducting the engineer’s approach to problem solving. New York: Oxford University Press.

Kruger, C., Cross, N. (2001) Modeling cognitive strategies in creative design. In J. Gero & M. Maher (Eds.), Computational and cognitive models of creative design V. (pp.). University of Sidney, Australia.

Laeser, M., Moskal, B. M., Knecht, R. & Lasich, D. (2003). Engineering design: Examining the impact of gender and the team gender composition. Journal of Engineering Education, 92 (1), 49-56.

Lewis, T. (1999). Research in technology education-some areas of need. Journal of Technology Education, 10 (2), 41-56.

Lewis, T. (2004). A turn to engineering: The continuing struggle of technology education for legitimatization as a school subject. Journal of Technology Education, 16 (1), 21-39.

Lewis,T. (2005). Coming to terms with engineering design as content. Journal of Technology Education, 16 (2), 37-54 Lubart, T.I (2000-2001).

McVearry, R. D. (2003, April). High-tech high schools build bridges to college. Engineering Times. Alexandria, VA: National Society of Professional Engineers. Retrieved December 2, 2006 from http://www.nspe.org .

Plaza, O. (2004). Technology education versus liberal arts education? Journal of Technology Studies, 30 (1), 16-18.

Project Lead The Way—Sample curriculum (2006). Retrieved May 24, 2007, from http://www.pltw.org/curriculum/sample-curriculum.html

Roberts, P. (1994). The place of design in technology education. In D. Layton (Ed.) Innovations in science and technology education: Vol. 5, 171-179. Paris: UNESCO.

SchÖn, D. (1983). The reflective practitioner, Basic Books: New York.

van Someren, B., van de Velde, W., & Sandberg, J. (1994). The think aloud method: A practical guide to modeling cognitive processes. Academic Press: London, UK.

Welch, M. (1999). Analyzing the tacit strategies of novice designers. Research in Science & Technical Education, 17 (1) 19-34.

Welch, M. & Lim, H. (2000). The strategic thinking of novice designers discontinuity between theory and practice. Journal of Technology Studies, 26 (2) 34-44.

Wicklein, R. C. (2006). Five good reason for engineering design as the focus for technology education. The Technology Teacher, 65 (7), 25-29.

Todd R. Kelley ( kelley30@uga.edu ) is an Assistant Professor in the Department of Industrial Technology at Purdue University , West Lafayette, Indiana.