Matthew Lammi and Timothy Thornton

Abstract

The purpose of this study was to understand the cognitive processes and procedures employed by an expert and a novice engineer in a realistic radio frequency (RF) systems engineering design challenge by using verbal protocol analysis (VPA). The engineering design challenge encompassed engineering, political, and social constraints. The audio data were then transcribed, segmented, and coded for analysis. The processes and strategies of the expert and novice were juxtaposed for analysis. The expert and novice shared some similarities in their cognitive processes and strategies. However, the expert’s domain knowledge and experience was vastly distinct from that of the novice.

Keywords: Engineering design, Systems Design, Design Cognition, Expert- Novice engineer, Engineering and technology education

Introduction

Technological and engineering literacy are critical components of a prosperous society. One dimension of both technological and engineering literacy can be defined as “. . . a way of thinking and acting” ( Pearson & Young, 2002, p. 3 ). Cognitive science addresses ways of thinking as a window to the human mind, shedding light on thought processes and how the mind is structured ( Adams, Turns, & Atman, 2003 ). Because technology and engineering are better understood within the domain of cognition, the further the promulgation of a technologically and engineering literate society. One way to study cognitive processes and strategies is through verbal protocol analysis ( Kruger & Cross, 2001 ).

Engineering is a topic of interest not only limited to the postsecondary training of engineers, but it is also found in the K-12 settings as an educational discipline rich in innovation, problem solving, and higher order thinking skills ( Brophy, Klein, Portsmore, & Rogers, 2008 ). Although engineering literacy is becoming a part of the American educational landscape, there is much to be understood about what engineering literacy is and how to teach it to nonengineering K-12 students. One aspect of engineering literacy is putting engineering habits into thought and action. To better understand these habits, one can employ an expert/novice perspective, where the expert is an engineer and the novice is the presumed student. This perspective describes the point where a student currently is (novice) and where that student could be (expert). The aim of this study was to further the knowledge base of engineering cognition by describing the cognitive processes and strategies of both an expert and a novice in the design of a wireless communications system.

The research questions for this study were:

• What cognitive processes and strategies are used by an expert and a novice in engineering design?
• How do the expert’s and the novice’s cognitive processes and strategies compare?

Research Literature

This study is based on the foundation of cognitive science as it pertains to engineering and technology education ( Brown, 2001 ). Engineers and technologists are given the task to solve problems, both in the classroom and in practice. Design is a category within problem solving that is cognitively intensive ( Jonassen, 2000 ). Cognition is more than simply to know something; it stems from the Latin word cognoscere, meaning to become acquainted with (Cognition, 2013). To become intimately acquainted with a particular field of practice, one has to acquire thorough knowledge and develop intricate associations. This knowledge and these associations are represented in cognitive science by complex arrays of networks known as schema ( Brown, 2001 ). One of the goals of engineering and technology education is to expose the student to, and hopefully move the student closer to, the skills and thinking of an expert in the field. By observing and analyzing cognition, research may reveal further insights into how experts and novices approach and strategize engineering and technology design. These insights might then aid in engineering curriculum and practices.

Engineering Design

The pervading concept of design is interwoven throughout engineering processes and culture ( Burghardt & Hacker, 2004 ). Design is a nebulous process that may be perceived from either a scientific or an artistic viewpoint ( Cross, 2001 ). Design is dynamic and iterative; therefore, it is not easily represented by simple linear models ( Mawson, 2003 ). Design typically commences with defining and formulating the problem ( Cross, 2004 ). Formulating the problem includes the gathering of pertinent data, delineating the overall goal, and creating an initial plan or “next steps.”

Engineering design is more than the manipulation of numbers and the solving of scientific equations. The processes employed in engineering design encompass a broad variety of topics and fields of study. Bucciarelli (1988) , an ethnographer, described engineering as a social process. The National Academy of Engineering (2004, 2005) clearly stated that engineering education was lacking if it did not include the global perspective in engineering design such as social, political, and environmental issues. The global perspective in engineering is part of systems engineering. Systems engineering involves viewing design from the whole-systems level rather than from an isolated modular perspective.

Jonassen (2000) placed design in its own problem type in his taxonomy of problem solving. Design is not only listed as complex and ill-structured, but it also requires higher order problem-solving skills. Engineering design typically entails resolving the designer’s goal and the criteria set forth by clients or other external parties ( Cross, 2002 ). Very often the external criteria are associated with resources, such as capital or time. Jonassen and Tessmer (1996) further asserted that as a problem type, design skills are influenced by domain knowledge, cognitive skills, and affective traits. This has been supported by Ericsson (2001) , who stated that focus and commitment are also factors in expertise.

Because design is an important aspect of both engineering and technology, it has been the focus of numerous studies involving engineering cognition ( Atman & Bursic, 1998 ; Atman, Kilgore, & McKenna, 2008 ; Cross, 2002 ; Ericsson & Simon, 1993 ; Jonassen, 2000 ; Lammi & Branoff, 2012 ). These studies used verbal protocol analysis (VPA), or a variation of this analysis, as a major component in gathering data. If VPA is used, the participants verbalize their thoughts out loud while engaged in a task or while solving problems. The participants typically engage in a hypothetical engineering problem or challenge in order to stimulate increased cognitive activity. The VPA is performed in a room where there are few distractions to help the participant maintain mental focus. The participant is also accompanied by a researcher or assistant who records the verbalization with either an audio or video recorder. Although Hayes (1989) conceded that verbal protocols are typically incomplete, he also claimed that under controlled conditions there is no evidence that verbal protocols distort or interfere with a participant’s thinking while that participant is engaged in a task.

Expert versus Novice

Students and experts vary according to their ability in engineering design. These differences of engineering design cognition are often analyzed against the expert and novice continuum ( Atman et al., 2008 ; Cross, 2002 ; Lawson & Dorst, 2005 ). The novice is limited by experience and knowledge, resulting in a partial and simple schema. The expert has a vast depth of experience and focused practice within a domain, resulting in deep and rich schemata ( Cross, 2004 ). However, experience and knowledge alone do not ensure expertise. The manner in which the experience and knowledge is ordered and interrelated has a great impact on expertise.

An expert is able to recognize large amounts of information, or chunks ( Egan & Schwartz, 1979 ). From these chunks, an expert can recognize what information is relevant to the issue at hand. This enables the expert to quickly and efficiently wade through data and facts with fast retrieval from her/his memory or schemata. This process may be compared to the routing of data packets in a computer network. The switches are constantly “learning” new and efficient pathways to connect data from one end user to another. The more complete and expanse the connections, the quicker the routing of the data packets. As more equipment and nodes are added to the network, the possibility of a more efficient pathway is introduced. It is evident that adding more nodes to the network alone does not increase efficiency. Rather, it is the deliberate and continual attempts to reroute by the switching equipment that increases efficiency. Hence, when new information or experience is added to the human mind, it is only useful if a purposeful association is made. This deliberate and focused effort was explained by Ericsson (2001) as the primary difference between experts and those who are only proficient in their domains. However, as the solution space evolves and elucidates further constraints, the expert returns to and references, or redefines, the problem space iteratively until the design is implemented, tested, and concluded. These attributes can be combined together to highlight the “know how” that is often demonstrated by an expert. The literature in engineering design cognition has primarily employed verbal protocols analyzed against the expert-novice continuum. Although systems have garnered attention in recent literature ( Davis & Sumara, 2006 ), research regarding cognition in systems engineering design is limited.

Methods

The purpose of this investigation was to understand and compare both an expert’s and a novice’s cognitive processes and strategies while they are engaged in the design of a systems engineering challenge. There were two participants in this study, an expert and a novice in wireless systems design. A small number of participants was chosen to allow an in-depth analysis of the data. Each of the VPA generated hundreds of data points that were coded and analyzed. The design challenge given to the participants was a hypothetical radio frequency (RF) systems design. The hypothetical setting was chosen to help capture the participants’ thinking within the bounds of a VPA.

An RF network is a system of cellular phone towers and accompanying equipment distributed throughout an area to provide cellular phone service. RF systems designs encompass engineering, political, and social variables and constraints. The design challenge was a simulated open-ended RF engineering problem.

VPA was used in this study to gather participants’ cognitive strategies and processes as they performed their tasks. Both were invited to share everything they were thinking during the design challenge. To increase trustworthiness and minimize leading questions during the VPA, both participants were only prodded to verbalize if there was at least a five-second pause in sharing their thoughts. As both were encouraged to share all of their thoughts, the resulting transcription was not always coherent or grammatically correct. The VPA was followed immediately by an interview to clarify ambiguities that emerged during the challenge. Additionally, the researcher annotated observations during the challenge, and a design artifact was collected and analyzed. Following the design activity, the audio data were transcribed, categorized, and coded for analysis ( Glesne, 2006 ).

Participants

The sample for this study included two participants drawn from the opposite ends of the expertise continuum in the domain of RF engineering system design. As such, they were selected regarding their skill set within RF engineering system design. Although RF systems engineering is not typically taught at U.S. universities, the coursework in electrical or electronic engineering generally serves as a basic foundation. Additionally, an RF engineer must also have a solid understanding of wave propagation theory in addition to digital communications. To gain proficiency in RF systems design, the engineer must grasp the societal and political impacts while working collaboratively across a wide array of teams (ranging from construction crews to executive management). Expertise in RF engineering is generally obtained through extensive practice in industry because of the frequent complex human interactions that must be balanced with sound engineering design.

An expert RF engineer is not only the most senior engineer among peers, but this person often consults other engineers nationally and internationally. Even within the domain of RF engineering, there are subdomains where one may achieve further expertise: design, optimization, and spectrum allocation. The expert for this study, Robert (pseudonym), had over thirteen years of RF systems engineering design experience working for a major cellular provider in various positions (ranging from manager to internal consultant). This expert received a bachelor’s degree in electrical engineering and continued his education through self-learning and corporate training.

The other participant, Gary, was a novice and at the other end of the spectrum of RF systems engineering. He was a professor in electronic engineering technology and had taught electronics at the postsecondary level for more than 35 years. Although this participant was a novice in RF systems engineering design, he had a breadth of skills in pedagogy and undergraduate electronics. He was chosen as the novice because of his background in electrical theory and practical experience with electromagnetic wave propagation; however, he did not have any specific training in RF systems engineering design.

Design Challenge

The participants were asked to design a new RF network in an isolated college town as if they were engineering design consultants. This challenge took place in a small office; only the participant and a researcher were present. Prior to the VPA, both participants were invited to perform a warm-up activity to prepare them to think out loud. In this warm-up activity, both participants gave a virtual tour of their homes. The participants described in detail the interior of their homes, including the windows, wall colors, and type of wood of the cabinets.

Immediately after the warm-up activity, the participants were given a three- dimensional aerial map overlaid with major and minor transportation thoroughfares to aid in the design, as seen in Figure 1. Each participant was invited to place potential cellular towers on this map. Constraints were placed in the design challenge to create a realistic ill-defined scenario. The constraints were to limit capital expenditures and abide by the zoning to not exceed 60-foot towers, and design cell sites to be hidden or stealth. Additionally, both were made aware of high cellular traffic venues, such as a university with 18,000 students and a fictitious annual wakeboarding event that would draw 10,000 individuals.

A follow-up interview was also conducted immediately following the design challenge. The interview questions included the following: why each participant chose varying cellular sites, why certain methods and strategies were employed, and what they were thinking during prolonged pauses. Additionally, the 3-D map served as an artifact for triangulation with the participant’s verbalization and interview responses.

Data Collection and Analysis

The audio from the design challenge was transcribed into a word processor. The transcription was broken into units or segments. The segments consisted of a sentence, unless a separate thought or idea surfaced necessitating further segmenting. The segments were coded into distinct mental processes used in engineering. Various methods have been used in coding verbal protocols ( Atman & Bursic, 1998 ; Kruger & Cross, 2001 ), in contrast, the coding for this study was done from the perspective of the researcher as themes emerged. Although there are various engineering coding schemes, for purposes of this research, a thematic approach was employed to discover any salient themes that emerged. Existing, well-defined coding schemes could potentially limit the outcomes and findings. Furthermore, RF systems engineering is a phenomena that has not been widely researched, especially through VPA.

The Verbal Protocol Analysis

To help the participants relax and have their minds free from distraction the VPAs took place on Friday afternoons when work was slow. To further minimize distractions, the VPAs took place in a quiet and secluded fluorescent-lit room with little decorations. Each participant and the researcher sat at a huge wood laminate table at the middle of the room while they shared their thoughts on solving the design challenge.

Results

Because both participants had multiple years of experience at the systems level in electronics, they both initially utilized a top-down approach in their design. Such an approach begins with the big picture and then breaks the design into its components. Robert, the expert, initially stated, “Is this for the whole area, or is it . . . ?” Not only did Robert commence with this method, he also designed the system to interact with potential existing systems. Both participants also used an iterative process evaluating and visualizing their design against the various constraints. However, Robert was able to more thoroughly analyze and balance the constraints, such as zoning and leasing. Robert quickly noted, “The zoning limitations listed here as stealth design – hmm. Okay, now these are competing requirements: limiting capital expenditures and stealth.”

Both the expert and the novice frequently returned to foundational principles for predictions and site locations. Gary, the novice, was fully aware of his limitations and stated repeatedly that he did not have the experience and knowledge to make an accurate design. At one point Gary stated, “I have a lot of questions, but I am not sure.” Conversely, the expert was able to make mental predictions or visualizations of the design and relied heavily on experiential and episodic memory. Robert discussed his experience with universities saying, “The university populations historically have a really high penetration rate for mobiles.” Although both participants recognized high cellular traffic areas, Robert knew how to quantify and optimize the design. Robert stated, “We are around 80-85% penetration rate now. So, obviously we are going to want to [get] very good coverage along the interstates and highways to support where people frequently use their mobile phones in travel.” One possible explanation for this was that the novice did not recognize the particularly high cellular phone traffic implied by a university or a wakeboarding competition.

Robert’s, as an expert, design strategy revealed differences from that of Gary’s, as a novice. Robert approached the design from a personal viewpoint, drawing heavily from previous experiences and precedents. The expert made frequent references to his experiences, particularly with respect to capital expenditures. Robert commented on the zoning requirements impacting the capital funds, “Because you have lower antenna heights required by the stealth design, you know there is an elevated cost to build sites.” Although the participants were given the same tasks, Robert set about the design from the context and point of view of a consultant. He felt that he had to produce a design that was feasible, both financially and with respect to RF engineering. Robert not only produced design, but he also made statements about how it would be zoned, leased, and constructed. Context is an important factor in problem solving, and it was evident in Robert’s responses. From the expert’s perspective, Robert spent a considerable amount time managing and justifying his design.

One of the most striking contrasts between the participants was the attention Robert gave to the optimization of capital expenditures. It is noteworthy how quickly he recognized the two rival requirements of reduced costs and stringent zoning restrictions commonly known as stealth. This same theme pervaded his entire design process. Even though Robert made 16 references about costs, Gary mentioned costs only 3 times. Additionally, Robert’s design proposed only 7 sites (versus 15 for Gary), substantially reducing the cost of the proposed design. Although Gary recognized financial costs in his design, Robert framed nearly every design aspect within the context of costs. This is not surprising since Gary’s career is in academe, and Robert’s was exclusively in industry, daily working within budgets.

Another striking difference between the expert and the novice was the amount of knowledge in the domain. Figure 2 is a pair of concept maps that reveal the disparity in knowledge differences. The researchers created the concept maps to visually highlight the differences reported between the responses of the novice and expert. Gary did not have the breadth and depth of knowledge that Robert did. Gary also did not allude to or even mention spectrum considerations.

However, the novice did have a working knowledge of radio frequency electromagnetic wave propagation. Gary did mention zoning, leasing, and capacity, but this could partially be accounted for by the design brief. Although not shown on the concept maps, Robert not only mentioned the different aspects within RF design, he also made many connections and associations between concepts.

Robert demonstrated the idea of satisficing, or the yielding of an ideal design for one that is only satisfactory. This was expressed as he managed limited capital and accounted for stealth zoning. Robert also made use of techniques unique to his trade, or gambits, to help overcome the stealth requirements. The expert employed water towers, rooftops, and stadium lights as economical alternatives to other costly stealth solutions. Gary was prompted for further analysis and design but he replied, “Experience would probably tell a person more information whether [the system design] is enough or . . . not.” Gary was aware that he lacked the relevant experience and domain-specific knowledge to elaborate on his design.

Discussion

From the study we can see how an expert and a novice are alike and how they differ regarding RF engineering system design. The expert exhibited expansive practical knowledge within his domain. The expert also maintained a systems perspective throughout his design by accounting for costs, zoning, and other teams’ needs. Furthermore, the expert approached the design challenge from a distinct context. Engineering and technology educators might do well to broadly educate their students to become systems thinkers ( National Academy of Engineering, 2005 ). This systems approach to teaching could include costs, organizational behavior, and political and societal impacts. The design method may be taught, but emphasis should be placed on the idea that there is no universal problem-solving model. Lastly, systems-level engineering could be infused into the curriculum as a top-down approach. This approach emphasizes breadth as well as depth, with the depth being situated in context and not isolated. Presenting the overall concept and then delving into components is an alternative method for reaching varying types of students’ learning. This article has presented a few ideas that could be infused to engineering and technology education practice and research that could further increase technological and engineering literacy.

This study included only two participants, one on each end of the expert-novice continuum. Any findings or conclusions were made in light of this limitation. Further research that includes a greater number of participants would be more conclusive. Nonetheless, the results of this study could help be a springboard for other studies and serve as another datum point among other similar studies.

---

Dr. Matthew Lammi is an Assistant Professor of STEM Education at North Carolina State University, Raleigh. He is a member of the Alpha Pi chapter of Epsilon Pi Tau.

Timothy Thornton is a doctoral candidate within the department of Technology, Design, and Engineering Education at North Carolina State University, Raleigh. He is a member of the Alpha Pi chapter of Epsilon Pi Tau.

References

Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering designers: The role of reflective practice. Design Studies, 24 (3), 275-294.

Atman, C. J., & Bursic, K. M. (1998). Verbal protocol analysis as a method to document engineering student design processes. Journal of Engineering Education, 87 (2), 121-132.

Atman, C. J., Kilgore, D., & McKenna, A. (2008). Characterizing design learning: A mixed-methods s tudy of engineering designers’ use of language. Journal of Engineering Education, 97 (2), 309-326.

Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P-12 classrooms. Journal of Engineering Education, 97 (3), 369-387.

Brown, D. (2001). Cognitive science concepts and technology teacher education. Journal of Technology Studies, 27 (1), 33-42.

Bucciarelli, L. L. (1988). An ethnographic perspective on engineering design. Design Studies, 9 (3), 159-168.

Burghardt, M. D., & Hacker, M. (2004). Informed design: A contemporary approach to design pedagogy as the core process of technology education. The Technology Teacher, 64 (1), 6-8.

Cognition. In Merriam-Webster’s online dictionary. (2013). Retrieved from http://www.merriam-webster.com/dictionary/cognition

Cross, N. (2001). Designerly ways of knowing: Design discipline versus design science. Design Issues, 17 (3), 49-55. doi:10.1162/074793601750357196

Cross, N. (2002). Creative cognition in design: Processes of exceptional designers. In T. Hewett & T. Kavanagh (Eds.), Creativity and cognition (pp. 6-12). New York: ACM Press.

Cross, N. (2004). Expertise in design: An overview. Design Studies, 25 (5), 427-441.

Davis, B., & Sumara, D. (2006). Complexity and education: Inquiries into learning, teaching, and research. Mahwah, NJ: Erlbaum.

Egan, D., E., & Schwartz, B., J. (1979). Chunking in recall of symbolic drawings. Memory & Cognition, 7 (2), 149-158.

Ericsson, K. A. (2001). Attaining excellence through deliberate practice: Insights from the study of expert performance. In M. Ferrari (Ed.), The pursuit of excellence through education (pp. 4-37). Hillsdale, NJ: Erlbaum.

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

Glesne, C. (2006). Becoming qualitative researchers: An introduction (3rd ed.). Boston, MA: Pearson, Allyn and Bacon.

Hayes, J. R. (1989). The complete problem solver (2nd ed.). Hillsdale, NJ: Erlbaum.

Jonassen, D. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48 (4), 63-85.

Jonassen, D., & Tessmer, M. (1996). An outcomes-based taxonomy for instructional systems design, evaluation, and research. Training Research Journal, 2, 11-46.

Kruger, C., & Cross, N. (2001). Modeling cognitive strategies in creative design. In J. S. Gero & M. L. Maher (Eds.), Computational and cognitive models of creative design V (pp. 205-226). Sydney: Key Centre of Design, University of Sydney.

Lammi, M., & Branoff, T. (2012). High school students’ habits of mind and action in engineering design. Paper presented at the ASEE Annual Conference, San Antonio, TX.

Lawson, B., & Dorst, K. (2005). Acquiring design expertise: A first attempt at a model. In J. S. Gero & M. L. Maher (Eds.), Computational and cognitive models of creative design VI (pp. 213-229). Sydney, Australia: University of Sydney, Key Centre of Design Computing and Cognition.

Mawson, B. (2003). Beyond ‘The design process’: An alternative pedagogy for technology education. International Journal of Technology and Design Education, 13 (2), 117-128.

National Academy of Engineering. (2004). The engineer of 2020: Visions of engineering in the new century. Washington, DC: The National Academies Press.

National Academy of Engineering. (2005). Educating the engineer of 2020: Adapting engineering education to the new century. Washington, DC: The National Academies Press.

Pearson, G., & Young, A. T. (Eds.). (2002). Technically speaking: Why all Americans need to know more about technology. Washington, DC: The National Academies Press.