Examination of Assessment Practices for Engineering Design Projects in Secondary Technology Education (Second article in 3-part series)

Todd R. Kelley, Ph.D.
Purdue University

Robert C. Wicklein, Ed.D.
University of Georgia

Introduction

There is a growing interest in the topic of engineering design for technology education. At the 2007 and 2008 International Technology Education Association (ITEA) conference held in San Antonio, over 80 presentations were related to engineering topics. Further evidence of the influence and impact of engineering design content comes from the large number of well documented curriculum projects designed to infuse engineering content into technology education such as Engineering by Design; Project ProBase; Project Lead the Way, and Introduction to Engineering ( Dearing & Daugherty, 2004 ). Likewise, state curriculum standards exist for the teaching of engineering design in technology education ( Massachusetts Department of Education, 2001 , Advisory Committee on Engineering and Technology Education in Georgia. (2008) . Moreover, authors in the field of technology education have provided a strong rationale for engineering design to be the focus for technology education ( Hill, 2006 ; Lewis, 2004 ; Wicklein, 2006 ). In a very short time, the field has moved from “coming to terms” with engineering design ( Lewis, 2005 ) to research studies that suggest the technology education teachers value this focus and are already on the move towards infusing engineering design into technology education ( Dearing & Daugherty, 2004 ; Gattie &Wicklein, 2007 ; Kelley, 2008 ). Based on these efforts to infuse engineering practices within the technology education curriculum it is appropriate to now investigate how technology education teachers are assessing engineering design activities within their classrooms. This research study was guided by the following questions:

1. To what degree do current assessment practices of secondary technology educators reflect engineering design concepts?
2. What are the similarities and differences of assessment practices of secondary technology educators when grouped by traditional and block schedules?
3. What are the greatest and least emphasized engineering design assessment practices by secondary technology education teachers?

Related Literature

Welch (2001) indicated that research on assessment practices in technology education was sparse. Furthermore, Lewis (2005) indicated that assessment of the teaching and learning of design was still an undeveloped aspect of technology education. Arguably, design has been at the center of technology education teaching and learning for some time and therefore should also be at the center of assessment criteria. Lewis (2005) provides a strong rational that design is the single most important category in the Standards for Technological Literacy ( ITEA, 2000/2002 ). Design, as a subject and as a process as outlined in the Standards, is the catalyst to explain and understand how all man-made things work which fall within the domain of engineering. Lewis identified that of the twenty standards in the document, four directly address design. However, Lewis also indicated that assessment of the teaching and learning of design was still an undeveloped aspect of technology education. Several studies in technology education have focused on the assessment of design, engineering design, and problem solving. Halfin (1973) was a pioneer in the development of a coding process to assess an individual’s design and problem solving thought process. Halfin used biographical and autobiographical data to evaluate the intellectual processes used by ten high-level designers (e.g., Buckminster Fuller, Thomas Edison, Frank Lloyd Wright) to solve technological problems. Halfin employed the Delphi research technique to identify 17 mental processes that were universal for these expert engineers and designers. Halfin’s coding process has been used in several research studies using an observation protocol methodology to assess students’ design and problem solving capabilities ( Hill, 1997 ; Kelley, 2008 ). Similar studies have also used observation assessments to evaluate students engaged in the design process and these methods have been found to be an effective assessment technique ( Lewis, Adams, Punnakanta, Littleton, & Atman, 2001 ). Custer, Valesey, & Burke (2001) developed and validated an instrument for assessing student learning in design and problem solving. This research was founded on the concept that problem solving can be condensed into a set of discrete, observable behaviors able to be captured using appropriate rubrics. The examples of research in technology education that focuses on assessing students’ abilities in design and problem solving listed above have provided a foundation of knowledge to build upon, but there is clearly a need for more research on assessment of engineering design thinking.

One recent study sought to identify appropriate assessment strategies for engineering design at the secondary level. This Asunda and Hill (2007) study determine the critical features of engineering design that can be incorporated within technology education learning activities. The researchers also developed a rubric for assessing these identified features. The study used a phenomenological approach through a semi-structured interview process working with three professors of engineering education. The interview process revealed four core themes for emphasis in technology education with an engineering design focus. The four core themes were (a) the process of engineering design; (b) societal benefits of engineering design; (c) attributes of engineering design; and (d) assessment. Qualitative data from the interviews of the participants revealed that participants used a variety of assessment practices to evaluate students design projects including; a) student portfolios, b) assessment by a panel of engineering faculty for industry based-projects, and c) individual and group presentations. This data was used to construct an assessment rubric for evaluating the design (process and product), the communication (oral and written), and the teamwork demonstrated throughout the activity.

Methodology

This descriptive study drew a full sample of high school technology teachers from the current ITEA membership list (September 2007). The sample consisted of all high school technology teachers regardless of whether they indicated they were teaching engineering design in their classroom. The identified population of this study consisted of a total of N=1043) high school technology education. The original research design for this study called for an increase of the initial mailing of the survey by 48.1 percent, the average success rate of an initial mailing ( Gall, Gall, & Borg, 2007 ). However, close communication with ITEA personnel revealed that ITEA survey mailings typically yield a 20-25% rate of return (Price, personal communication). The researcher determined that a full sample mailing to all ITEA high school members was necessary. A cover letter was sent electronically through e-mail for all ITEA members in the sample who listed an active e-mail address in the fall of 2007. The electronically delivered cover letter contained a URL for the on-line questionnaire. The on-line questionnaire was managed by HostedSurvey.com. The on-line questionnaire was developed using the guidelines and recommendations outlined by Dillman, Tortora, and Bowker (1999) . There was a request to return the survey on a specified date.

The researchers sent out the surveys to the population of 1043 high school ITEA teachers. After waiting three days past the specified date of return, which was three weeks after the initial mailing, the researcher contacted non-respondents by sending a follow-up e-mail delivered letter containing the URL for the on-line survey link. This has been a proven method used by other researchers to achieve compliance from non-respondents ( Gall et al., 2007 ).

Instrument

Results of Asunda and Hill’s (2007) study created a framework in the survey instrument to identify appropriate assessment strategies for secondary technology educators when assessing engineering design activities. The researchers used the elements from Asunda and Hill’s rubric to create eight instrument items related to assessment practices for engineering design projects. See Table 1 for a complete list of the eight individual instrument items for assessment practices.

Participants were required to respond to each curriculum content item in two ways, (1) the frequency of using the assessment practices and (2) the amount of time per typical use of the assessment practice. A six-point Likert type scale was used to collect this data, see Table 2.

Limitation

In order to determine statistical significance for this population size N =1043, Krejcie and Morgan’s (1970) method was to locate sample size for a given population size; the required sample size for the size of this population was set at 285 ( Gay & Airasin, 2000 ). Again, the survey was sent out to all secondary education ITEA members in order to increase the chances of achieving an appropriate response rate. The final results of the study yielded a total of 226 respondents; therefore, the results of this study cannot be generalized to the entire population. However, the researchers compared the demographic data results of this study with demographic results of a similar national status study of technology education ( Gattie & Wicklein, 2007 ) that did receive a response rate level to generalize to the population. The demographic results of both studies were very similar, thus suggesting that these results were representative to the population. However, the researchers acknowledged that statistical significance was not achieved in this study.

Results

The top mean scores for individual items were as follows: provide evidence of idea generation strategies (e.g. brainstorming, teamwork, etc.) (mean of 2.92), develop a prototype model of the final design solution (mean of 2.69), and work on a design team as a functional inter-disciplinary unit (mean of 2.53). Overall, the assessment practice category yielded relatively low mean scores for a 5 point Likert scale, none of which yielded a mean of 3 or higher. The lowest mean scores were items using mathematical models to optimize, describe, and/or predict results (mean of 1.72), while proper record design information in an engineer’s notebook also yielded a low mean of 2.01. See Table 3 for total results of the assessment practice category.

A composite score was generated for assessment strategies for traditional and block scheduling (see Figure 1). Computing a composite score for the assessment practices of high school technology teachers by using mean scores for time per typical use and frequency of use provided an indicator to reveal areas of emphasis and deficiencies regarding assessment practices. The researchers split the files; separating traditional and block scheduling results in order to accurately calculate a composite score. Splitting the file was necessary because the units of day and units of duration were different between the groups. A comparison of the difference between the total hour composite score for each of the assessment strategies between the two groups is reported in Table 4.

Figure 1. Composite Score for Assessment Strategies Based on Time Per Use

Comparisons of the difference between the total hours and % of total hours for each of the assessment strategies between the two groups are reported in Table 4. The differences in total hours between traditional and block scheduling was analyzed to determine if there were major differences between the two groups for each of the assessment strategies. The assessment strategy that assessed the developing a prototype model of the final design solution received the greatest total hour difference of 5.03 hours. The assessment strategy that required students to use design criteria such as budget, constraints, criteria, safety, and functionality resulted in the greatest consensus among responders with only a 0.15 of an hour difference with traditional scheduling dedicating 9.76% and block scheduling dedicating 9.61% of their time on this assessment strategy. The assessment strategy that focused on the use mathematical models to optimize, describe, and/or predict results resulted in the lowest emphasized item for assessment practices with traditional scheduling teachers dedicating 2.42 % and block scheduling teachers dedicating 3.70% of their time utilizing this assessment practices. Over one third of the time technology education teachers spent on assessing students engineering design projects was devoted to two items: evidence of idea generation strategies (e.g. brainstorming, teamwork, etc.) with 22.47% for traditional and 23.96% for block scheduling, and the item develop a prototype model of the final design solution with 22.84% for traditional and 17.22% for block scheduling.

Conclusions

According to the results of this study, secondary technology education teachers place the lowest emphasis on assessing the use of mathematics to optimize and predict design results (Traditional 2.42 %, Block 3.70% of assessment practice time). These results are strong indicators that the engineering analysis phase of the engineering design process is not emphasized very much in assessment practices. This is a major concern considering a number of leaders in technology education have indicated that a major difference between the technological design process and the engineering design process is analysis and optimization ( Hailey, et al., 2005 ; Hill, 2006 ; Gattie & Wicklein, 2007 ). Without a strong and consistent emphasis on the analytical process to solve technological problems students and teachers are limited in their ability to utilize a comprehensive engineering design process therefore defaulting to the standard trial and error methodology to solve problems. It can be argued that the mathematical modeling and analysis is the heart of engineering design and that without this focus on the design process little or no actual engineering is taking place. This is an important issue to consider especially when it has ramification of damages to the reputation of the technology education field. Individuals inside as well as outside the field of technology education might have rationale to accuse technology education of once again changing the name on the door and not changing the practice ( Clark, 1989 ). Sanders (2008) has observed that many technology education teachers are fond of the appeal of integrating math and science into technology education; when in reality it is rare for technology teachers to identify specific science and mathematical concepts as student learning outcomes for their lessons or activities. Sanders goes on to state “…it is even rarer for technology teachers to assess a science or mathematics learning outcome” (2008, pp. 20-26). Technology education teachers are still emphasizing the importance of building prototyping in their assessment practices. The assessment item developing a prototype model of the final design solution just edged out the idea generation item as the top assessment strategy for traditional schedule teachers with 22.84% of their time dedicated to assessing prototypes; this assessment strategy was the second highest emphasized for block scheduling teachers with 17.22% of their assessment time dedicated to this category. Allowing students to build prototypes is an appropriate and important part of the engineering design process. However, constructing prototypes without first using mathematics and science to optimize and predict design results is not authentically engaging in the engineering design process. A strong rationale for implementing the engineering design process over other design processes (e.g., trial and error) is that engineering design requires mathematical and scientific analysis to fully inform the designers to allow them to make educated decisions regarding optimal design before prototype building begins. Technology education teachers who indicate that they are implementing an engineering design process and not requiring or assessing students engaged in some mathematical predictions before prototyping are still using the “trial and error” method and are not truly engaging in the power of the engineering design process.

Another area of lesser emphasis was assessing student’s record keeping of design information in an engineer’s notebook (2.01 mean for frequency of use, 1.78 mean for time per typical use). It is unclear if technology educators are implementing the use of engineer’s notebooks in the classroom and just not using them as an assessment tool. Engineer’s notebooks are not only used in engineering schools at the collegiate level, they are also used in engineering practice; therefore technology educators who use engineering notebooks to assess students’ design thinking and record keeping skills would be implementing an authentic assessment technique. Moreover, Hill (2006) suggests implementing the use of an engineering design notebook can help students use a systematic approach to design and problem solving.

Another low mean score item was providing evidence of formulating design criteria and constraints prior to design solutions (Mean of 2.33 (time); Mean of 2.19 (frequency)). Identifying constraints and criteria early in the design process is an important feature of the engineering design process but is a practice not widely adopted within the field of technology education ( Hill, 2006 ). The low mean score of this individual item confirms this statement.

Summary

As a field, we should review the results of this study (see Figure 1) and ponder on the statement by Young and Wilson : “assessment is a public declaration of what is valued” (2000, p ii). This is an appropriate time to reflect upon the purpose of technology education. Can technology education provide a real-life context for the application of mathematics and science through an engineering design focus? Or is this approach to curriculum revision just another way to legitimize the subject of technology education by using the term engineering? ( Lewis, 2004 ).

The researchers recognize that it is unlikely each of the assessment practices identified in the instrument would or should have equal emphasis by the classroom teacher. However, when research results indicate that items such as using mathematical models to optimize, describe, and/or predict results receive less than 4% of the total year of assessment time, it strongly indicates that this is a category of engineering design assessment not widely used as assessment criterion. For years, technology educators have been encouraging students to design and build the fastest model car ( LaPorte, 2005 ), the strongest model bridge ( Volk, 1996 ), or the highest reaching rocket ( Hill, 2006 ) it is the researchers belief that the time has come for technology educators to aspire to help students to use mathematics and science to make the most educated decisions regarding their design solutions. One strong indicator that the field of technology education has truly begun to infuse engineering design into the classroom will be when students begin approaching technology teachers and say “According to my calculations, we are not ready to build the prototype because the current design will not work”. This statement will likely never happen and the field will not authentically infuse the engineering design process unless technology educators implement and assess the use of mathematical models to predict design results and optimize student’s final design solutions.

Todd R. Kelley, Ph.D., is an Assistant Professor in the Department of Industrial Technology at Purdue University. He can be reached at trkelley@purdue.edu.
Robert C. Wicklein, Ed.D., is a Professor at the University of Georgia. He can be reached at wickone@uga.edu.

References