The Use of Executive Control Processes in Engineering Design by Engineering Students and Professional Engineers

Raymond A. Dixon and Scott D. Johnson

Brophy, Klein, Portmore, and Rogers (2008) admitted that, as industries are driven by the rapid development of enabling technologies, they must become more flexible and adaptive to remain competitive. This flexibility is achieved through a workforce that can utilize newly available technologies and generate innovations of their own. They further suggested that such technological capability in the workforce can only be possible if students entering higher education are prepared differently at the K-12 level, through programs that target the development of technological literacy.

Driven by the goal to improve technological literacy, the Standards for Technological Literacy: Content for the Study of Technology (ITEA, 2002) provide a framework for increasing students’ technological literacy at all levels of the K-12 curriculum through the integration of engineering design. In reference to the design component of the Standards for Technology Literacy , Lewis (2005) argued that it is “the single most important content area set forth in the standards, because it is a concept that situates the subject more completely within the domain of engineering” (p. 37). Consistent with its usage in society, engineering design provides an ideal platform for engineering and technology educators to integrate mathematics, science, and technology concepts for students to solve real-world (ill-structured) problems innovatively and creatively.

Executive Control Processes

A cognitive construct that is important when solving engineering design problems is executive control process, or metacognition. Flavell (1978) and Brown (1978) define metacognition as knowledge and cognition about cognitive phenomena, or the monitoring of one’s own memory, comprehension, and other cognitive processes. Kellogg (1995) refers to metacognition as cognition about cognition, or thinking about thinking. It is a central feature of human consciousness that enables one “to be aware of, monitor, and control mental processes” (p. 211).

In a synthesis of the literature on metacognition, Meijer, Veenman, and van Hout-Walters (2006) found that several studies identify some commonalities of higher order (executive control) cognition. For example, Schraw and Moshman (1995) subdivide metacognitive control processes into planning, monitoring, and evaluation; while Pintrich and DeGroot (1990) divide metacognition into planning, monitoring, cognitive strategies, and awareness. O’Neil and Abedi (1996) also agree with the aforementioned researchers’ perception of metacognition; indicating that it includes planning, monitoring, and evaluation.

Davidson, Deuser, and Sternberg (1995) discuss four metacognitive processes that are important contributors to problem-solving performance across a wide range of domains and problem types, including well-structured and ill-structured problems (see Figure 1). When a problem is given (including a design problem), the solver must decide what is known about the problem, what design criteria are expected, and what the constraints might be. They then use representations such as metaphors, analogies, and propositions to make sense of the problem and develop a solution.

Metaphors and analogies are important representations used by designers in design problem solving ( Casakin & Goldsmith, 1999 ; Daugherty & Mentzer, 2008 ; Hey, Linsey, Agogino, &Wood, 2008 ). Metaphorical reasoning allows one to make conceptual leaps across domains from a source to a target, such that a new situation can be characterized and understood by reference to a similar one. In respect to designing, metaphors are often used in the early stages of the design process to assist the designer in framing the problem. Besides being used descriptively to define the problem and understand the situation, metaphors can also be used prescriptively as a solution generation tool. For example, “the metaphor, Shower Is A Reset, can be used to generate solutions that could support people’s feeling of starting anew even to the point of activating the shower with a button” (Hey et al., 2008, p. 288).

An analogy can be defined as the “illustration of an idea by means of another idea that is similar or parallel to it in some significant features” ( Hey et al., 2008, p. 283 ). Analogies make possible the solution of a problem in the target domain by superimposing upon it a solution from the base domain ( Lewis, 2008 ). In contrast to metaphors, analogies are generally used to solve functional issues and are used mainly during the generation of solutions, rather than in the framing of the design problem.

According to Paivio (1990) , propositions are the most versatile of representational concepts because they can be used to describe any type of information. They are strings of symbols that correspond to natural language and which “represent information in the same way regardless of whether the information is experienced verbally, as a spoken or written sentence in whatever language, or nonverbally, as a perceptual scene” ( Paivio, 1990, p. 31 ). The relevance of propositions for engineering design lies in the fact that they can be expressed as general principles, rules-of-thumb, or heuristics; specific physical laws, such as those used in physics; or mathematical formulas ( Greca & Moreira, 1997 ).

According to Davidson et al. (1995) , planning entails dividing the problem into sub-problems and devising the sequence for how the sub-problems should be completed. Individuals are more likely to engage in planning when solving ill-structured problems because the situation is often novel and complex, so planning or structuring brings clarity to one’s intended actions. The plan is often revised or modified as the problem solver confronts obstacles during the solution process. This is consistent with Jonassen’s (1997) view that ill-structured problems possess multiple solutions because they can have multiple representations and multiple problem spaces. Research shows that individuals with less expertise in solving a particular type of problem spend less time in global “up front” planning, and relatively more time in attempting a solution, than do experts across age levels and areas of expertise ( Davidson et al., 1995 ). Studies show that designers select features of the problem space to which they choose to attend and identify areas of the solution space they choose to explore ( Cross, 2006 ). Junior engineering students tend to gather a lot of information when solving a design task, while more experienced designers ask for less information, process data instantly, and quickly build an image of the problem.

While Davidson and associates (1995) assert that monitoring as an executive control process is concomitant with evaluation, some researchers, however, treat both as separate processes (see Flavell, 1979 ; Kincannon et al., 1999; Schraw & Moshman, 1995 ; Veenman, van Hout-Wolters, & Afflerbach, 2006 ). For the purpose of this study, both were treated as separate processes. Schraw and Moshman (1995) define monitoring as one’s awareness of comprehension and task performance, as well as the ability to engage in periodic self-testing while learning or solving a problem. The monitoring process relies on a variety of memories (such as idiosyncratic memories, emotional memories, and problem-related memories) and also on abstract rules. Although engineering design problems are ill-structured and contextually driven, the problem solver must apply abstract rules or propositions, like those used when solving well-structured problems in knowledge domains such as mathematics and physics, in order to achieve an optimal solution.

Evaluation is the appraisal of the products and regulatory processes of problem solving. According to Schraw and Moshman (1995), this typically includes re-evaluating one’s goals and conclusions. The representations used by problem solvers are referenced as they appraise their performance. Davidson et al. (1995) purport that evaluation includes control over the internal representations formed, and those that still need to be formed, for understanding and solving the problem. Jonassen (1997) further adds that evaluating one’s performance after the implementation of a solution includes the designer appraising: (a) whether the solution produced is acceptable to all the parties involved, (b) whether the solution is within the problem constraints articulated, (c) whether the solution is elegant or parsimonious, and (d) whether the effects of the solution could be optimized.

Conceptual Framework

The framework for this study was conceptualized by integrating the model for creative design, which illustrates the co-evolution of the problem and solution spaces during engineering design problem solving (see Dorst & Cross, 2001 ; Maher, Poon, & Boulanger, 1996 ), with executive control processes such as planning, monitoring, and evaluation. According to Maher, Poon, and Boulanger (1996), whenever engineers are solving design problems, their problem and solution spaces co-evolve with an interchange of information between the two mental spaces. Dorst and Cross (2001) confirmed the accuracy of the Maher et al. model in a protocol study of nine experienced industrial designers whose designs were evaluated on overall quality, creativity, and a variety of other aspects. For simplicity, the co-evolution of the problem and solution spaces is illustrated in Figure 2 by the overlap of the two ellipses. Superimposing elements of the Davidson et al. (1995) metacognitive model on the problem and solution spaces of Maher et al. raises questions about how designers use executive control processes throughout their problem and solution spaces.

Purpose of the Study

The purpose of this study was to investigate if there are differences in the cognitive process of engineering students and professional engineers as they use executive control processes (i.e., planning, monitoring, and evaluation) in the problem and solution spaces while solving an engineering design problem conceptually. The following research questions guided the study:

1. In what ways do the executive control processes (planning, monitoring, and evaluation) of engineering students and professional engineers differ in their problem and solution spaces?
2. How are propositions, analogies, and metaphors distributed throughout the use of executive control processes by engineering students and professional engineers?
3. What is the overall design strategy of the professional engineers and engineering students?

Method

A qualitative comparison of novice and expert engineers was conducted. A purposeful sampling procedure was used to select the participants. According to Gall, Gall, & Borg (2007) , in purposeful sampling the goal is to select cases that are likely to be “information rich” in respect to the purpose of the study. The executive control processes of a small group of mechanical engineering students were compared with a small group of professional mechanical engineers.

Participants

An email was sent inviting juniors and seniors in a four year mechanical engineering program at a Midwestern university to particate in the study. Six mechanical engineering students agreed to participate, three junior and three senior undergraduates. The four professional engineers were recommended by a former associate dean of a college of engineering, who is also a member of the American Society of Mechanical Engineers. Each professional engineer is recognised as an expert in mechanical engineering design. Except for one professional engineer, their individual number of years in the profession exceeded the minimum 10 years of experience it generally takes to achieve expertise in a particular domain ( Phye, 1986 ). The small sample size is typical of verbal protocal studies ( Jiang & Yen, 2009 ; Trickett & Trafton, 2006 ).

The Design Task

Each participant was given the same design problem for which to find a conceptual solution. Before the design task was administered, it was vetted by two professionals in the field, an Engineering Technology professor with over 20 years teaching experience and a Mechanical Engineering professor with over 10 years experience as a manufacturing consultant and over three years experience teaching manufacturing principles. This review helped ensure that the design task was sufficiently ill-structured and of an appropriate difficulty level to engage the students and professional engineers. The final design task was then reviewed by a professor who teaches the senior design project course, and the task was pilot tested with a mechanical engineer with over 20 years experience (see Figure 3).

Procedure

The design task was administered at a time and place convenient for each participant. Pencils, erasers, and sketchpads were provided, along with the instructions for the design task. Each participant was allowed approximately one hour to complete the design solution. A $25 gift card was given to each participant. Participants were required to produce only one conceptual design.

Data were collected primarily through Verbal Protocol Analysis. The first stage of data collection, referred to as concurrent protocol, was carried out while the design problem was being solved. The second stage of data collection, referred to as retrospective protocol, was performed immediately after the problem was solved.

Each participant had the choice of doing a verbalization practice session of about five minutes, thinking aloud as they solved a simple mathematical problem, to prepare them for the study. After they were comfortable with the thinking aloud process, the task was administered. The participants were encouraged to speak aloud whatever they were thinking as they solved the problem. Their think-aloud verbalizations were audio recorded. If the participants stopped talking, they were prompted or reminded to continue to speak aloud what they were thinking.

After each participant completed the engineering design problem, an interview was conducted to clarify sections of the protocol and to allow the participant to explain the executive control processes that were applied. Like the concurrent protocol, the interviews were audio recorded. Their response to the reflective interview questions served as a supplementary data source to the concurrent protocols. A general interview guide format was used. According to Gall et al. (2007) , with the general interview format, no set of standardized questions is written in advance because the order in which the topics are explored and the wordings of the questions are not predetermined.

Data Analysis

The audio recordings of the concurrent and retrospective protocols were transcribed. The transcribed protocols were then segmented into think-aloud utterances, divided into sentences, and coded. The quality of the sketches was not evaluated since the objective of the study was to examine the mental processes of the engineering students and the professional engineers while they solved the design task. The sketches and notes, however, acted as a reference to clarify some sections in the protocols.

The purpose of segmenting is to break the transcribed verbal protocol text into units (or segments) that represent discrete thoughts and can be coded with a pre-defined coding scheme. Codes were provided for nine predefined constructs identified from the literature reviewed on metacognition, analogies, problem solving, and design (e.g., Casakin & Goldschmidt, 1999 ; Cross, 2006 ; Schraw & Moshman 1995 ; Hey et al., 2008 ). The codes were consistent with the constructs described in the model for metacognitive processes in problem solving ( Davidson et al., 1995 ) and the model for creative problem solving ( Dorst & Cross, 2001 ; Maher et al., 1996 ).

The constructs representing the participants’ mental representation were proposition, analogy, and metaphor. Those representing the participants’ executive control processes were planning, monitoring, and evaluation. The mental spaces describing the problem-solving episode were problem space, solution space, and overlapping space (see Table 1, next page). There were a total of 270 utterance segments (150 for the professional engineers and 120 for the engineering students).

Reliability coding was conducted by having two persons code seven pages of one transcript ( Miles & Huberman, 1994 ). A reliability kappa coefficient of 0.76 was calculated for the first coding. All disagreements between coders were resolved through discussion. A second coding was done by both coders on the same number of pages of another transcript and a reliability kappa coefficient of 0.9 was calculated. One coder then completed the coding of the remaining transcripts.

Table 1
Constructs, Codes, and their Meaning

Construct	Code	Meaning
Propositions	Prp	Mathematical and engineering science formula and rule-of-thumb or heuristics used for example in analysis—e.g., F= mv 2 r; "lowering the frame will lower the center of mass".
Analogies	Anl	Comparing an idea with another idea that is similar in structural and relational features—e.g., comparing the surface texture of a leaf with the surface texture of a plate in a battery; Comparing two types of motorbikes
Metaphors	Mta	Allows one to make conceptual leaps across domains from a source to a target so that a new situation can be characterized and understood by reference to a familiar one. They help to provide meaning to a design situation—e.g., viewing a gas station design problem as an oasis. Understanding a design situation by comparing an electronic book delivery design to a restaurant metaphor ( Hey et al., 2008 ) .
Planning	Pla	Dividing the problem into sub-problems and strategizing how to reach a solution—e.g., Gathering data, prioritizing the requirements in design brief, identifying constraints.
Monitoring	Mon	Engaging in periodic self-testing and assessment of the quality of design as one progress to a solution—e.g., Performing analysis; testing the accuracy of a formula, calculation, or sketch for the accuracy of a clamping force.
Evaluation	Eva	Appraising or judging whether the solution of a design meets constraints, costs, and all the demands of the stakeholder; judging quality of two or more design—e.g., Appraising whether one component is designed with the cheapest material that can guarantee the required strength and quality required by the customers.
Problem space	Prb-sp	Includes design activities such as gathering information, defining the problem, identifying constraints, specifying evaluation criteria, and initially searching alternative solutions.
Solution space	Sol-sp	Includes activities such as developing a solution, sketching, drawing, deciding between two alternatives, optimizing a selected solution, and determining specifications.
Overlapping spaces	Prb-Sol	The mental space where information is interchange between problemand solution spaces. Involves consulting the design brief to make verification then returning to the solution or start a new solution. Activities include analysis and the selection of alternative solutions.

Results

Executive Control Process Frequency and Characteristics

As illustrated in Figure 4, the frequency of planning activities for both groups decreased, while the frequency of the monitoring and evaluation activities for both groups increased, as they progressed from the problem space to the solution space. The frequency of the professional engineers’ executive control processes was higher in the solution space (83) than the engineering students (59). Overall, the engineering students had a higher frequency of planning activities than the professional engineers.

The engineering students showed major increases in the frequency of their monitoring activities in the problem space (6) and solution space (42). The professional engineers also showed major increases in their monitoring activity from the problem space (2) to the solution space (45); however, the engineering students displayed more monitoring activities in the overlapping space (24) than the professional engineers (10). The professional engineers did not show any signs of evaluation in the problem space and showed very little in the overlapping space (2), but the frequency of evaluation in the solution space increased significantly (35). The professional engineers used more executive control activities on average than the engineering students.

Table 2
Characteristics of Executive Control Processes

Metacognitive	Characteristics
Regulation	Engineering Students	Professional Engineers
Planning	Spent more time planning.	Spent less time planning
Monitoring	Used analogies to help in the framing and understanding the problem. “So what I am doing right now is trying to think of other road vehicles, their seating like for example four wheelers, their seating and the racks are much wider, so we could possibly make the rear a little wider by extending the frame…” The majority of metal representations were exhibited during monitoring, and analogies were used more frequently than propositions. Safety seems to be the main factor that drives the assessment and optimization of the quality of a solution. “The exhaust I think might cause a problem with the rider. I think the more shielding would have to be implemented to prevent the rider or any cargo from burning.”	Planning strategies more driven by engineering science principles rather than analogical features. “So I lowered the center of gravity of the load and extended the wheel-base for stability. Okay I have an initial concept for moving forward.” The majority of metal representations were exhibited during monitoring, and analogies were used more frequently than propositions. Most of the monitoring activities focus at improving the customer safety and comfort. “But this I mean to make the passenger more comfortable we’ve got to do a better job of seating”
Evaluation	Spent less time on evaluation	Spent more time on evaluation

Table 2 illustrates the main characteristics of the engineering students’ and professional engineers’ executive control processes. These characteristics were identified by themes that were common in the protocols for all four of the engineering students and three of the professional engineers.

Mental Representations and Design Strategy

The engineering students and the professional engineers used different amounts of propositions, metaphors, and analogies in their planning, monitoring, and evaluation. Only three of the engineering students (Don, Gus, and Len) and one professional engineer (Mac) used metaphors while they were planning. One engineering student (Len) and one professional engineer (Mac) used a metaphor while carrying out monitoring activities.

The engineering students used only analogies in their evaluation, except for one (Hank) who used both analogies and propositions. In contrast, two of the four professional engineers used both analogies and propositions in their evaluation; one used only analogies, and one did not use any mental representation. Overall, most of the mental representations that were used by the engineering students and professional engineers occurred while they were monitoring their design solution. One engineering student (Hank) deviated from this pattern, using most of his mental representation during evaluation. The second highest number of mental representations was used during the planning of the engineering students and during the evaluation of the professional engineers.

There were several differences and similarities in the engineering design strategy used by the engineering students and professional engineers. The professional engineers, on average, took a longer time to solve the design task than the engineering students (professional engineers 47.17 minutes; engineering students 30.17 minutes). The protocols revealed that some students and professional engineers showed a determination not to deviate from an early concept. This behavior is similar to findings indicating that experienced mechanical engineering designers and senior design students tend to attach to early solution ideas and concept ( Ball, Evans, & Dennis, 1994 ; Ullman, Dietterich, & Stauffer, 1988 ). For example, the student Len, stuck with an ATV design idea from the beginning to the end of his design.

LEN: (After about 2 minutes into his solution) …and if you like a back seat like an ATV type it would be considerable more comfortable than having two people on one motorcycle. (About 25 minutes later) …okay for safety my original design is definitely safer because it’s two people sitting in an enclosed area and the bars here would be metal so they at more of a roll bar like on ATVs.

The professional engineer Kirk showed a similar attachment to a concept that he had from the beginning of his solution.

KIRK: (After about 3 minutes into his solution) …my initial thought was some sort of an articulated vehicle that would be attached to the rear of the motorcycle that would carry the passenger and/or luggage and provide stability. (About 32 minutes later) …my original concept for two rear wheels revolves around a rickshaw type concept where you would still essentially have four tires for the total vehicle. The rickshaw would provide a stable ride for the passenger to get out carry lots of load; it would be a really nice solution.

The general design recommendation from both groups was a motorbike with a carriage compartment at the back; flatter, lower seats with a backrest; and broad wheels and locks to secure the helmets. There was remarkable similarity, and not much variance, in the alternative solutions of both the engineering students and the professional engineers. For example, both groups considered using a saddlebag in the center of the bike, a four wheel ATV type vehicle, a three wheel ATV type vehicle, a bike with a passenger carriage to the side, and a bike with a luggage carriage that is pulled from the back.

Discussion and Conclusions

The small purposeful sample used does not allow for generalized statements to be made about the mechanical engineering design process of professional engineers and engineering students. The findings from this qualitative study, however, confirm previous findings of other studies and provide useful insights about the executive control processes of student and professional mechanical engineer designers.

Three conclusions were drawn from the findings. The first is that expert planning and monitoring is driven by propositions, while the novice planning and monitoring is influenced by analogical comparisons. One possible reason the students used more analogical comparisons in their planning and monitoring is because they were not familiar with the type of design problem, and so they drew upon similar types of design to aid them in defining the problem and finding solutions. In contrast, the professional engineers, because of their years of experience, could easily understand the nature of the problem and, therefore, relied more on engineering science formulas and heuristics in their planning and monitoring.

The second conclusion is that mental representations are used mostly when the engineering student and professional engineers are monitoring their design solutions, and the professional engineers are more balanced than the students in their use of analogies and propositions. This conclusion is reflective of one of the themes identified by Jonassen, Strobel, and Beng Lee (2006) in a qualitative study of engineers in their natural working environment. They found that instead of relying on one form of representation, engineers use multiple forms of problem representation in their day-to-day practice.

The third conclusion is that evaluation plays a larger role in the solution space of professional engineers, while engineering students do more planning in the problem space. The decrease in planning activities and increase in monitoring and evaluation activities, as the designers move from the problem space to the solution space, were consistent with what Davidson et al. (1995) implied about metacognition in problem solving. The findings, however, indicate that the engineering students did more planning than the professional engineers. This conflicts with literature on metacognition in problem solving. For example, Davidson et al. (1995) stated that “individuals with less expertise in solving a particular problem seem to spend relatively less time in global ‘up front’ planning for solution, and relatively more time in attempting to implement a solution than do experts” (p. 218). Atman et al. (2007) also found that expert mechanical engineers spent twice as much time in problem scoping activities, such as problem definition and gathering information, which are elements of planning. The professional engineers in this study may not have needed to spend much time for planning due to their past experience, as planning may be so familiar to them that they simply move into articulating their thoughts about solutions.

It is not surprising that the professional engineers used more monitoring and evaluation in the solution space. In fact, the literature on metacognition indicates that experts excel in these self-regulatory and appraisal skills. Experienced engineers were observed to make preliminary evaluations of their tentative decision, perform final evaluation, balance systems of benefits and tradeoffs, and use guidelines and rules-of-thumb when making decisions ( Ahmed, Wallace, & Blessing, 2003 ; Crismond, 2007 ). The time spent in decision making is likely to be related to the time spent generating and evaluating solutions ( Radcliffe & Lee, 1989 ).

Implications

The fact that the professional engineers used multiple forms of representations strengthens the suggestion of Jonassen and associates (2006) that design curriculum and pedagogy should not rely exclusively on algebra, calculus, and trigonometric formulas to represent problems, but students should be taught how to supplement these propositional representations with other alternative qualitative problem representations.

The Standards for Technological Literacy: Content for the Study of Technology state, “as a part of learning how to apply design process students in grade 9-12 should… evaluate the design solution using conceptual, physical, and mathematical models at various interval of the design process in order to check for proper design and note areas where improvement is needed” (ITEA, 2002, p. 123). Evaluation is recognized as a higher order cognitive skill at which experts excel. Therefore, design curriculum and teaching strategies should target the development of these skills. Engineering and technology students should be taught how to use both quantitative and qualitative analytical methods to frame their strategy and monitor their design conceptualizations. In later stages of the design process, students can be taught how to determine the best alternative solutions by the conducting of scientific tests. This will also improve their evaluative skills. According to Crismond (2007) , “Students can develop their own guidelines based on tests they conduct by formulating design rules-of-thumb. Design rules-of-thumb can strengthen the link between science and engineering design and amount to intermediate abstractions that link the concrete realities of a particular mechanism and product with relevant concepts and laws from engineering and the natural sciences” (p. 27).

The increased evaluation activities by the professional engineers were evident primarily when they reflected on or reviewed their processes and solutions. Self-monitoring and evaluation are associated with higher levels of design quality (Crismond, 2007) . Therefore, strategies used in grades 9-12 ETE classrooms should allow students to reflect on and critique their own and other’s design process and product. Crismond recommends that giving students practice at identifying others’ design strategies can make their design-oriented metacognition more accurate and automatic.

---

Raymond A. Dixon ( rdixon@uidaho.edu ) is an Assistant Professor in Curriculum and Instruction at the University of Idaho. Scott D. Johnson ( sjohnson@illinois.edu ) is Professor Emeritus of Human Resource Education at the University of Illinois at Urbana-Champaign.

References