Experts vs. Novices: Differences in How Mental Representations are Used in Engineering Design

Raymond A. Dixon and Scott D. Johnson

Mental representation is an important cognitive construct when solving engineering design problems. When students are given a design problem, they must decide what is known, the constraints they have to work with, and what is required by the customer. They then use mental representations, such as metaphors, analogies, and propositions, to make sense of the problem and develop a solution.

Several studies have investigated the use of mental representations in problem solving. For example, Greca and Moreira (1997) investigated the use of mental models, propositions, and images by college students in solving physics problems involving electrical and magnetic fields. Their findings suggested that students work mostly with propositions unrelated to, or interpreted according to, mental models. Gick and Holyoak (1980) investigated the provision of source analogs prior to the tackling of a problem that is superficially different, but conceptually similar. Casakin and Goldschmidt (1999) examined the use of visual analogs by expert and novice designers in their work. The results of both studies indicated that people are good at utilizing prior problem and solution information when they are directed to do so, but then may not be efficient in detecting analogous information under unprompted conditions. Other studies ( Holyoak & Koh, 1987 ; Keane, 1987 ) show that past analogies are more readily activated when there are surface similarities in the target problem and the analogy.

Conceptual Framework Guiding the Study

There are several types of mental representation, but for the purpose of this study propositions, metaphors, and analogies were investigated. A proposition refers to the smallest unit of knowledge that one can sensibly judge as true or false. According to Paivio (1990) , propositions are the most versatile of representational concepts because they can be used to describe any type of information. They are strings of symbols that correspond to natural language. Unlike language, however, propositional representations are assumed to be “completely amodal, abstract, conceptual structures that represent information in the same way regardless of whether the information is experienced verbally, as a spoken or written sentence in whatever language, or nonverbally, as a perceptual scene” ( Paivio, 1990 , p. 31). The relevance of propositions for engineering design lies in the fact that they can be expressed as general principles, rules of thumb, or heuristics; as specific physical laws, such as those used in physics; or as mathematical formulas ( Greca & Moreira, 1997 ). Mathematical formulas, scientific principles, and heuristics are important tools that engineers use when performing design activities ( Eide, Jenison, Mashaw, & Northrup, 2002 ).

Metaphors and analogies are important representations used by designers in design problem solving ( Casakin & Goldsmith, 1999 ; Daugherty & Mentzer, 2008 ; Hey, Linsey, Agogino, & Wood, 2008 ). Metaphorical reasoning allows one to make conceptual leaps across domains from a source to a target, such that a new situation can be characterized and understood by reference to a similar one. In respect to designing, metaphors are often used in the early stages of the design process to help the designer frame the problem. Besides being used descriptively to define the problem and understand the situation, they can also be used prescriptively as a solution generation tool.

An analogy can be defined as the “illustration of an idea by means of another idea that is similar or parallel to it in some significant features” ( Hey et al., 2008 , p. 283). Analogies make the solution of a problem in the target domain possible by superimposing upon it a solution from the base domain ( Lewis, 2008 ). In contrast to metaphors, analogies tend to be used more during the generation of solutions and ideation phase of design, rather than during the framing phase to assist in understanding the problem. Analogies are generally used to solve functional issues. Analogies can be categorized as between-domain (large distance) and within-domain (local). Large distance or between-domain analogies exist when there are little surface similarities between the source and target, while local or within-domain analogies exist when there are greater superficial similarities between source and target ( Christensen & Schunn, 2007 ). An example of a between-domain analogy is trying to develop a door handle for the auto industry and comparing the door handle with a telephone or an oyster. A within-domain or local analogy is comparing the door handle to various car door handle designs. Designers use analogies to support concept selection. Analogies also assist the designer in predicting the performance of design concepts ( Hey et al., 2008 ).

Table 1
Example of Proposition, Analogy, and Metaphor

Mental Representations	Examples
Propositions	Mathematical and engineering science formula and rule of thumb. Example of Formula: F = mv 2 / r. Example of Heuristic: lowering the fame will lower the center of mass.
Analogy	Comparing an idea with another idea that is similar in structural and relational features, e.g., comparing the surface texture of a leaf with the surface texture of a plate in a battery. Within-Domain Analogies —analogies that are from the same domain, e.g., comparing two types of scissors; comparing two types of bicycles. Between-Domain Analogies —analogies drawn between two ideas from different domains but are used to resolve functional issues in a design, e.g., comparing the shape of car to the shape of a fish for aerodynamic reasons; comparing a device to remove blood clots to a plumbing or piping system ( Hey et al., 2008 ).
Metaphor	Allows one to make conceptual leap across domains from a source to a target so that a new situation can be characterized and understood by reference to a familiar one. They help to provide meaning to a design situation, e.g., viewing a gas station design problem as an oasis; understanding a design situation by comparing an electronic book delivery design to a restaurant metaphor ( Hey et al., 2008 ).

The framework for this study was conceptualized by integrating the model for creative design, which illustrates the co-evolution of the problem and solution spaces during engineering design problem solving (see Dorst & Cross, 2001 ; Maher, Poon, & Boulanger, 1996 ), with mental representations, such as proposition, metaphor, and analogy. Whenever engineers are solving design problems, their problem and solution spaces co-evolve with an interchange of information between the two mental spaces. As solutions are conceptualized, the designer will check and verify information relating to constraints, specific data about the problem context, specification, cost, and so forth in the problem space. This verification often results in the emergence of a new or parallel solution space. The interchange of information between the problem and solution spaces is illustrated by the overlap of the two ellipses in Figure 1 (next page).

The problem space includes design activities, such as defining the problem, searching for information, identifying constraints, and specifying evaluation criteria. Metaphors are more likely to be generated within the problem space because they are often used descriptively in the early stages of the design process to frame the problem and better understand the design situation ( Hey et al., 2008 ). Because the designer is trying to understand the problem, it is expected that fewer propositions (mathematics and engineering science principles) and analogies are used by the designer in the problem space.

After a number of possible solutions are generated, then the best of these solutions must be selected for further analysis. During the analysis phase, potential solutions that are not suitable may be discarded or, under certain conditions, retained with a redefinition of the problem and a change in the constraints and criteria ( Eide, Jenison, Mashaw, & Northup, 2002 ). Analysis primarily involves the use of heuristics, mathematical formulas, and principles of engineering science—all of which are propositional in nature—to achieve proper functionality of the component or system. During this process, references are continually made to the criteria and constraints that are stipulated in the problem. This is illustrated by the overlap of the two ellipses in Figure 1. It is also expected that analogies and propositions have more presence in this overlapping space.

As the designer approaches a solution, more judgmental decisions are made about the merit of the solution. It is expected that analogy and proposition are the predominant representations within the solution space, since they are used primarily to resolve and refine functional issues of the design ( Hey et al., 2008 ).

In a review of various types of design expertise, Cross (2004) provided a comprehensive body of empirical information describing the characteristics of expert mechanical engineers, industrial engineers, and architects when solving design problems. Some of these are:

• Expert designers select features of the problem space to which they chose to attend (naming) and identify areas of the solution space that they chose to explore (framing). In addition, expert architects' approach to problem solving was characterized by strong paradigms or guiding themes, while novices had weaker guiding themes.
• Expert designers and advance student designers exhibited fixation to their principal solution concept for as long as possible, making "patches" or slight modifications rather than discarding for alternatives.
• Whenever the cognitive cost for following a particular strategy becomes too high, expert designers will abandon or deviate from a principled, structured approach.
• Expert designers use non-linear strategies in problem solving. Often an interleaving of problem specification with solution development, drifting through partial solution development, and jumping into exploring suddenly recognized partial solution. They also use a mixture of breadth-first and depth-first approaches. Novices tend to follow a more linear depth-first approach.
• Unlike novices, experts have the ability to alternate rapidly between activity modes (examine-drawing-thinking) in rapid succession to make novel decisions.
• Outstanding designers seem to have the ability to work along parallel lines of thought. This means they maintain openness, even ambiguity about features and aspects of the design at different levels of detail, and consider these levels simultaneously as the design proceeds.
• Outstanding designers rely implicitly, or explicitly, on first principles in origination and development of concepts.
• Experts’ creative solutions arise when there is a conflict to be resolved between the expert’s own high level problem goal (their personal commitment) and the established criteria for acceptable solution by a client or other requirements.
• The superior performance of experts is domain specific and does not transfer across domains ( Cross, 2004 ).

Purpose of the Study

This study investigated the mental representations of student and professional engineers while they solved an engineering design problem. The intent was to gain a deeper insight into the differences that exist in the cognitive processes of engineering students and professional engineers as they use mental representations (i.e., propositions, metaphors, and analogies) to solve the engineering design problem. The following research questions guided the study:

1. How does the frequency of propositions, metaphors, and analogies used by engineering students and professional engineers differ in the problem and solution space?
2. How do the attributes of the propositions, metaphors, and analogies used by engineering students and professional engineers differ when they are solving a design problem?

Method

A comparative case study of engineering students and practicing engineers was conducted. A purposeful, maximum variation sampling process was used ( Gall, Gall, & Borg 2007 ). Maximum variation sampling, a special type of purposeful sampling, entails the “selecting of cases that illustrate the range of variation in the phenomena to be studied” ( Gall et al., 2007 , p. 182).

Sample Selection

Purposeful samples of mechanical engineering students and professional engineers from the Midwestern United States were selected. The student participants were three juniors and three seniors who had completed one or more courses with engineering design elements in their content. Each professional engineer possessed at least an undergraduate degree in mechanical engineering and had worked as an engineer for 7 to 40 years. Except for one professional engineer, their individual number of years in the profession exceeded the minimum 10 years of experience it generally takes to achieve expertise in a particular domain ( Phye, 1986 ). A total of four professional engineers participated.

The Design Task

Each participant was given the same engineering design problem for which to find a conceptual solution. Before administration, the design task was vetted by an Engineering Technology professor with over 20 years teaching experience and a Mechanical Engineering professor with over 10 years experience as a manufacturing consultant and over 3 years experience teaching manufacturing principles. This was to ensure that the design task was sufficiently ill-structured, and of the appropriate difficulty level, to engage the students and professional engineers. The design task was then checked by a professor who teaches the senior design project course, and the task was pilot tested with a mechanical engineer with over 20 years experience (see Figure 2, next page).

Procedure

The design task was administered at a time and place convenient for each participant. Pencils, erasers, and sketchpads were provided along with the instruction for the design task. Each participant was allowed approximately one hour to complete the design solution. A $25 gift card was given to each participant.

Data was collected primarily through Verbal Protocol Analysis. The first stage of data collection, referred to as concurrent protocol, was carried out while the design problem was being solved. The second stage of data collection, referred to as retrospective protocol, was performed after the problem was solved. To prepare them for the study, each participant had the option of doing a five minute session to practice thinking aloud as they solved a simple mathematical problem. The task was administered after they were comfortable with the thinking aloud process. The participants were encouraged to speak aloud whatever they were thinking as they solved the problem. Their verbalizations were audio recorded. If the participants stopped talking, they were reminded to continue to speak aloud and say what they were thinking.

After each participant completed the engineering design problem, an interview was conducted to clarify sections of the protocol and to allow the participant to explain representations used and metacognitive strategies applied. Like the concurrent protocol, the interviews were audio recorded. Their response to the interview questions served as a supplementary data source to the concurrent protocols. A general interview guide format was used ( Gall et al., 2007 ).

Data Analysis

After each participant completed the design task, the audio recording of the verbal protocol was transcribed. The transcribed protocols were then segmented into think-aloud utterances and coded. The problem space was primarily identified by activities, such as gathering information, defining the problem, identifying constraints, specifying evaluation criteria, and initially searching alternative solutions. The solution space was identified by activities, such as deciding between two alternatives, developing a specific solution, optimizing a selected solution, and determining specifications. The overlapping space was specifically identified in the protocol by verbatim transcription that indicated the designer was mentally transiting from the solution space to the problem space in order to gather additional information or to verify data, constraint, specification, etc., then returning to the solution, or starting a new solution. Design activities include analysis, additional information gathering, and the selection of alternative solutions.

The segmenting took place in two stages. In the first stage, larger units of analysis, called think-aloud utterances , were identified and segmented from each other. Think-aloud utterances comprise those words spoken aloud by a participant that were followed by some period of silence ( Hartman, 1995 ). A total 270 utterances were segmented (150 for the professional engineers and 120 for the engineering students). Codes were assigned to each segment using nine predefined constructs—heuristic, formula, analogy, within-domain analogy, between-domain analogy, metaphor, problem space, solution space, and overlapping space.

Reliability coding was conducted by having one additional person code seven pages of one transcript ( Miles & Huberman, 1994 ). A reliability kappa coefficient of 0.76 was calculated for the first coding. All disagreements between coders were resolved through discussion. A second coding was done by both coders on the same number of pages of another transcript, and a reliability kappa coefficient of 0.9 was calculated.

Results

Frequency and Types of Mental Representations

Figure 3 (next page) illustrates that the engineering students used almost equal percentages of mental representation in their problem and overlapping spaces, 21% and 20% respectively. However, 59% of their mental representations were generated in the solution space. The professional engineers surprisingly used a very small percentage (2%) of mental representations in the problem space, 22% in the overlapping space and 76% in the solution space. The conservative use of mental representations in the problem space by the experts might be indicative of the ease with which the experts were able to understand the problem and transit into a solution mode. They then invest most of their mental representation in finding solutions.

The number of propositions used by the engineering students increased from the problem space to the solution space. Five percent was used in the problem space, 7% in the overlapping space, and 32% in the solution space. The professional engineers did not use any propositions in the problem space, 6% in the overlapping space, and 34% in the solution space. The use of propositions was less in the problem space and more in the overlapping and solution spaces for both the professional engineers and the engineering students.

The total number of metaphors used was small in comparison to other mental representations. The engineering students used a total of 4 metaphors (5%), while the professional engineers used a total of 3 metaphors (6%). Two of the metaphors used by the students were in the problem space, 1 in the overlapping space, and 1 in the solution space. In contrast, 2 of the metaphors used by the professional engineers were in the overlapping space, 1 in the problem space, and none in the solution space.

The types of metaphor used were not from very distant domains and seemed to be influenced by key terms in the design question, such as “taxi,” and mental images that the designers generated of the conditions in which the taxi is expected to operate. The following are examples of metaphors used. The names assigned to participants from here onward are pseudonyms:

MAC: …I’m struck by the difficulty of balancing large loads and a passenger on a motorcycle in this rough terrain. My initial thought was some sort of an articulated vehicle that would be attached to the rear of the motorcycle that would carry the passenger and/or luggage and provide the stability . [Professional engineer]

LEN: Let’s see, so I’m thinking, try to keep the design small like almost like a compact type car . [Engineering student]

The percentage frequency of analogies used by the engineering students was 13% in the problem space, 12% in the overlapping space, and 38% in the solution space. As was the case with the use of propositions, the professional engineers did not use any analogy in their problem space. They used 12% analogy in their overlapping space and 42% in their solution space. It was also expected that analogies would be used less in their problem space and more in their overlapping and solution spaces. This proved to be true for the professional engineers and the engineering students. Overall, the engineering students surpassed the professional engineers in the percentage of analogies used (63% and 54% respectively). The percentage use of analogies by the professional engineers in their solution space exceeded those of the engineering students (42% professional engineers and 38% engineering students).

Proposition and Analogy Attributes

Figures 5 and 6 (next page) depict the type of propositions (formulas and heuristics) used by the engineering students and professional engineers respectively. Engineering students primarily used heuristics in their engineering design, while the professional engineers used heuristics and formulas more equally. Formulas and heuristics were primarily used to resolve functional issues that the designers encountered in their solution. The following are verbatim reports of occasions when the engineering students and professional engineers used propositions, such as formulas and heuristics, in their protocol.

VEL: “So if that’s F and G this would be cosine 30 and then sine 30 or wait the other way around…Then this force would or we could use like F equals MA. Then that force minus the force in the other direction would be equal to MA. Then we could determine which acceleration we would want to calculate the force .” [Engineering student using formula]

LEN: “The only problem with that is it might throw off the balance of the bike but you probably just have to put more of a counter weight in the front .” [Engineering student using heuristic]

RAY: “If you’re carrying two people and cargo, that’s extra weight. You know force, mass times acceleration, and work is force times distance and then horsepower is what …W work over time. So I would look at probably, I don’t think you need to go twice as big .” [Professional engineer using formula]

MAC: “And so my thinking there maybe I would go to two tires in the rear to provide additional heat dissipation capability, because of the smaller diameter .” [Professional engineer using heuristic]

Figures 7 and 8 (next page) illustrate, respectively, that the engineering students used more within-domain analogies, while the professional engineers used both within-domain and between-domain analogies almost equally. A small percentage of analogies from both groups were identified as unclear because their attributes could not be identified as within-domain or between-domain.

GUS: “That doesn’t look like it’s too comfortable for the passenger so like thinking back to types of four wheelers I’ve ridden they always had…here is the seat so I would modify it for the motor cycle.” [Engineering student using between-domain analogy]

LINA: “Let’s see, a device to prevent the… theft of helmets. I know a lot of motorcycles have something where in order to lift up the seat you actually have to put in your key and underneath the seat you have these little metallic…like little brackets basically.” [Engineering student using within-domain analogy]

RAY: “I wonder if this lock isn’t automatic for the release of the helmet. Well you know cars have, you don’t actually put your key in the car anymore to open up the door.” [Professional engineer using between-domain analogy]

VEN: “I’m trying to picture in my mind since we are talking about motorcycles and since I don’t know a lot about them, I am trying to picture essentially other kinds of motorcycles and why they may be inherently stable.” [Professional engineer using within-domain analogy]

Discussion and Conclusions

The results of this study paint a picture of how four professional engineers differ from six engineering students in their use of mental representations on a conceptual engineering design task. Three major conclusions are drawn from the findings: (1) The use of mental representations, such as propositions, analogies, and metaphors, in the different mental spaces is important in engineering design; (2) Different from novices, experts rarely employed propositions or analogies in their problem space; and (3) Expert engineering designers differ from novice engineering designers on their use of within-domain analogies, between-domain analogies, heuristics, and formulas.

The type of mental representations in design varies in the problem and solution spaces of designers. In fact, within the solution space, solutions are generated by recalling forms or graphical representations and functions. In addition, ideas are evaluated by comparison with the laws of nature, capability of technology, and the requirements of the design problem itself ( Ullman, 2003 ). The findings from the protocols indicated that the frequency of use of the various types of mental representations vary in each of these mental spaces, and the use of analogy and proposition is more prevalent, particularly within the solution space.

The greater use of analogies by the engineering students was one of the surprising findings of this study. The literature on analogical reasoning shows that analogies are important cognitive tools in design problem solving ( Daugherty & Mentzer, 2008 ; Hey, Lensey, Agogino, & Wood, 2008 ; Lewis, 2008 ). A study by Ball, Omerald, and Morley (2004) showed that experts displayed greater evidence of analogical reasoning than did novices, irrespective of whether such analogizing is schema-driven (between-domain) or case-driven (within-domain). One explanation for this obvious disparity is the type of question and the amount of time the students spent within the problem space and the overlapping space. The retrospective protocols of both groups indicated that the participants did not have any experience in solving that type of design problem before. Except for one student who recently purchased a motorcycle and one expert who owned a motorcycle for a short time when he was younger, none were fully conversant about motorcycles. Because of the difficulty of the problem, the students spent more time planning in the problem space. They also used more analogies in both the problem space and the overlapping space. Not being acquainted with this type of engineering design problem would naturally cause the students to use more analogical representations to understand and frame the problem and to create mental models from which they generate solutions. The professional engineers’ general experience and confidence, however, would cause them to immediately start exploring the solution space, accounting for the use of less propositions and analogies in the problem space and more usage in the solution space. This is consistent with earlier findings by Getzels and Csikszentmihalyi (1976) on problem finding and the creative process in art, which reported that experts differ from novices in the length of their search through the problem space, getting to the solution space faster than novices.

Christensen and Schunn’s (2007) explanation of the use of the various types of analogies may offer some insight into the findings that relate to the third conclusion. They claimed that problem-identifying analogies were mainly within-domain, explanatory analogies were mainly between-domain, and problem-solving analogies were a mixture of within- and between-domain. The engineering students tended to spend more time in a problem identification mode than a problem-solving mode, possibly because of the challenging nature of the design problem, while the professional engineers were more in a problem-solving mode, as was seen by their almost equal use of both types of analogies.

There was a level of over reliance by the engineering students on the use of heuristics while the professional engineers tended to use engineering science formulas and heuristics equally. Again, the fact that this type of design problem represents uncharted territory for most of the engineering students might explain why they used heuristics or rules of thumb in search for possible solutions. According to Davidson, Deuser, and Sternberg (1995) , heuristics can be used to construct mental representations when a problem solver finds that a current representation is not working. Another reason might be the cognitive cost that is involved in using heuristics. Some students found it difficult to remember certain engineering science formulas. Using heuristics, rules of thumb, or shortcuts is cognitively economical and reduces the cognitive load that students have to endure when trying to remember all the details of a formula.

Recommendations for Curriculum and Instruction

During conceptual design activities, the tasks in the curriculum that target the solution space—such as generating alternatives, analysis, optimization, and decision making—should be structured so that students are allowed to be exposed to the use of multiple forms of representations. The findings indicate that this is one way in which the experts’ design cognition differed from the engineering students—in their balanced use of different mental representations. The content of curriculum and the teaching strategy used should not emphasize exclusive use of engineering science or mathematical formulas, but should also encompass heuristics and other strategies that develop students’ mental models and build, not only their analytical, but also their qualitative representations. In fact, Jonassen, Strobel, and Lee’s (2006) research on the everyday problem-solving strategy of engineers showed that only a small minority of workplace engineers regularly uses mathematical formulas to represent problems. They recommended that teaching in classrooms should supplement mathematical formulas with alternative qualitative representations. The objective is to build the student’s repertoire of a variety of representations that would increase their ability to produce functional descriptions of design solutions, which correlate with high quality designs.

The ability to look beyond the disparate surface feature of source analogies and the design problems that they target, and identify common conceptual structures that link them together, is not easy and usually takes years of substantial experience solving different types of design problems. Gentner, Loewenstein, and Thompson (2003) opined that specific instructional intervention, such as accelerated example-based learning, may improve students’ ability to solve problems in an expert-like manner. The same principle can be applied in design instruction. Instructions that expose students to a wide variety of design examples, and which allow students to make active comparisons, critiques, and evaluations to understand the underlying concepts that make certain designs similar or different, will likely result in the formation of highly structured schemas, thus improving students’ ability to make analogical comparisons that go beyond surface similarities.

Recommendations for Future Research

Two recommendations are offered for future research. First, experimental studies can be conducted to show what difference exists in the quality of students’ design process and products when they use any one, or a combination of the three representations—formulas, heuristics, and analogies—in engineering design. Second, verbal protocol analysis can be used to examine the use of mental representation in the problem space and solution space by working design groups of engineering students and professional engineers, as they solve a design problem over an extended period, to determine if similar results are obtained as with single participants.

---

Raymond A. Dixon ( rdixon@uidaho.edu ) is an Assistant Professor in the College of Education at the University of Idaho. Scott D. Johnson ( sjohnson@illinois.edu ) is Professor Emeritus in Education Policy, Organization and Leadership at the University of Illinois at Urbana-Champaign.

References