Student Professional Development: Competency-Based Learning and Assessment

Jacqulyn A. Baughman, Thomas J. Brumm, and Steven K. Mickelson

Abstract

This case study examines the implementation of competency-based learning (CBL) and assessment as a measure of student professional development. Students enrolled in an industrial technology undergraduate course at a Midwestern university participated in this study. Based on the degree program outcomes, the “top five” course competencies were identified, and their key action items were assessed using an industry-based, 360-degree assessment process. Significant differences in the average initial and final assessed values were used to determine professional development gains. Findings showed that self-assessed professional gains were achieved, self-assessed results were higher than peer results, and overall peer assessments indicated aggregate gains in professional development. This case study provides a foundational framework for further research studies in competency- based learning and assessment.

Keywords: competencies, professional development, competency-based assessment

Background

Because most college-aged students are entering adulthood, the attitudes, interests, values, and character development that underlie their behaviors may not be at a professional level ( Hayward, Noonan, & Shain, 1999 ). Student development has been described as “the ways that a student grows, progresses, or increases his or her developmental capabilities as a result of enrollment in an institution of higher education” ( Rodgers, 1990 , p. 27), and is about becoming a more complex individual ( McEwen, 2005 ). The complementary theory used to explain and understand student development allows educators to “proactively identify and address student needs, design programs, develop policies, and create healthy environments that encourage positive growth in students” ( Evans, Forney, & Guido-DiBrito, 1998 , p. 5). Existing student development theories are very much interrelated ( Gardner, 2009 ). Psychosocial development theories are concerned with the content of development including growth or change related to how students view themselves and their abilities, the relationships they have with others in their lives, and the future direction of their lives ( Chickering & Reisser, 1993 ). This encompasses adult development and career development ( McEwen, 2005 ).

Competencies are the result of integrative learning experiences in which skills, abilities, and knowledge interact to form learning bundles that have a currency related to the task for which they are assembled; interest in competencies is accelerating throughout the world ( R. Voorhees, 2001 ). Until recently, competencies have been discussed from the demand side of employment, and consideration has been given primarily to the needs of employers. Competency models can be used by the supply side of the labor market as well, such as a learner or student, incumbent worker, or hopeful and expectant new employees applying for a position to achieve job stability ( Ennis, 2008) . Competency-based models enjoy an obvious connection to aspirational student learning statements, because they shift the focus from instructional delivery to student performance ( A. Voorhees, 2001 ). Competency-based learning (CBL) involves redefining program, classroom, and experiential education objectives as competencies or skills and focusing coursework on competency development ( Brumm, Mickelson, Steward, & Kaleita, 2006 ).

Postsecondary education has become progressively responsive to the needs of business and industry, where learning is closely tied to competencies and performance-based assessment of those competencies ( Gardner, 2009 ). Building a bridge between the educational paradigm that depends on traditional credit hour measures of student achievement and the learning revolution can be found in competencybased approaches ( R. Voorhees, 2001 ). These competencies are crucial for students before, during, and after attending postsecondary institutions ( National Center for Education Statistics [NCES], 2002 ). In a 2002 report, the U.S. National Postsecondary Education Cooperative Working Group on Competency-Based Initiatives determined three reasons why it is important to implement competency-based initiatives in colleges and universities:

The definition of workplace competencies is the application of knowledge, skills, attitudes and values, and behaviors ( Ewell, 1984 ). These competencies are directly measurable through actions or demonstrations of the existence of those competencies in the individual. Thus the opportunity to gain practice in the application of competencies and focused reflection in a workplace connects with experiential learning, which is defined as “the process whereby knowledge is created through the transformation of experience and knowledge results from the combination of grasping and transforming experience” ( Kolb, 1984 , p. 41).

Since the 1990s, competencies have become code words for the human resources and strategic management practices of recruiting, selecting, placing, leading, and training employees and evaluating employee performance. Competency-based assessment and feedback has become a predominant workplace reality, which is commonly used as an organizational development tool for the learner ( McCarthy & Garavan, 2001 ). A competency-based assessment tool popularized in the 1980s, mostly as an executive development tool that gained currency in the 1990s, is the multi-rater or 360-degree feedback process (McCarthy amp; Garavan, 2001). The fundamental premise is that data gathered from multiple perspectives are more comprehensive and objective than data gathered from only one source ( Dyer, 2001 ).

Many organizations use some form of the 360-feedback assessment process ( Nowack, 1993 ), and it is implemented in a variety of ways. Ratings from self and others, however, constitute the core of the 360-degree feedback process ( Tornow & London, 1998 ). Self-ratings are the first step to development for the feedback recipient. The value lies in the diversity of information it provides to the feedback recipient and how it is interpreted. It can be perceived as a positive self-development platform, in stark contrast to traditional top-downward evaluation process. Under ideal circumstances, it is used as an assessment for personal development rather than evaluation ( Tornow & London, 1998 ). Widespread in many organizations around the world ( Brutus et al., 2006 ), this process is reportedly used by 90% of Fortune 500 companies in the United States ( Carruthers, 2003 ). The popularity of this practice has stimulated much research enthusiasm in the academic field ( Dai, De Meuse, & Peterson, 2010 ).

Incentivizing Competency-Based Learning

Institutional accountability, articulation and student transfer issues, and workplace market alignment have become critical drivers that can provide the impetus for institutions to shift to competency-based models ( A. Voorhees, 2001 ). Increasingly, accreditation requirements challenge faculty to look ahead to anticipate emerging skills or a change in the emphasis on certain skills that could impact the preparedness of engineers and technology graduates for employability in the knowledge-intensive workplace. Competencies provide students with a clear map and the navigational tools needed to move expeditiously toward their goals ( R. Voorhees, 2001 ). The advantage of competency-based learning (CBL) is that competencies are transparent; that is, all participants in the learning process understand the learning goals and outcomes. Competency expectations have increased significantly across all sectors of the economy, and the abilities employers expect new college graduates to demonstrate the first day on the job have been ratcheted up to an “über level” ( Hanneman & Gardner, 2010 ).

Purpose of the Study

Specifically, the primary purpose of this study was to measure student professional development utilizing an industry-based, 360-degree competency assessment process. An additional goal was the development of a framework for CBL and assessment that can be used in other higher education settings.

The Foundation

Competency-Based Approach to Accreditation

The chosen Midwestern university’s unique approach to accreditation requirements was to address them through development of workplace competencies ( Brumm, Mickelson, et al., 2006 ). Identification of key industry employer needs drove this rationale: “employers of the graduates of our program are increasingly focusing on workplace competencies in their hiring practices, and student development of competencies is, therefore, critical to career success after graduation” ( Brumm, Mickelson, et al., 2006 , p. 1163). Through collaboration with Development Dimensions International, Inc. (DDI), a global provider of competency-based performance management tools and services, 14 unique workplace competencies were developed ( http://learn.ae.iastate.edu/ Competencydefinitions.pdf). These competencies were mapped directly to outcomes of degree programs. Seven, which were regularly mentioned by employers, were identified as “core” competencies. Each competency was defined clearly, concisely, and independently. Specific to each definition, a set of observable and measurable key actions was developed. By closely tying competencies with performance-based assessment of those competencies, a bridge is built between traditional measures of student achievement and competency-based approaches ( R. Voorhees, 2001 ).

Course Connectivity

Competency-based models rely on both the judgment of those external to the learning process and on measurable assessment ( R. Voorhees, 2001 ). A conceptual model of learning based on competencies does not work solely at the level of skill, abilities, and knowledge but seeks to formulate curriculum and assessment at the competency level; this embodies integration of skills, abilities, and knowledge needed to become part of the disciplinary community of practice ( Jones, 2001 ). Competencies have a stronger impact on student learning when they are linked to and embedded within specific courses and across the curriculum ( DDI, 2004 ).

A lean/cellular manufacturing course for senior-level undergraduate students provided the opportunity to design a CBL experience. Based on the instructor’s industry background, professional development based on competency assessment was considered critical to prepare students for success in the workplace environment. The intent of the course design was to provide students the opportunity to “step through the looking glass” and understand the roles competencies and competency assessment play in professional/career development. In this pursuit, all coursework and activities developed were focused on competency development. This chosen Midwestern university’s Industrial Technology assessment plan already contained competency-based learning tools that easily integrated into the course: 14 workplace competencies and a competency assessment format. Based on previous stakeholder assessment feedback, all 14 workplace competencies would not be utilized for the 360-degree process. Thus, a review of the course “core” competency frequency, coupled with the instructor’s 360-degree assessment industry experience, was used to identify the top five course competencies: (a) analysis and judgment, (b) communication, (c) initiative, (d) continuous learning, and (e) teamwork. These top five competencies were the basis for the implementation of the 360-degree assessment process, and they are shown in Table 1.

Method

Twenty-six students enrolled in a lean/cellular manufacturing course in the Industrial Technology program at the Midwestern university that participated in this study. The top five competencies were used for initial and final assessments, of both self and peers, during the semester. Key actions associated with each competency were assessed utilizing the department’s Likert-scale format. These assessment ratings were based on how often a key action was performed, ranging from 1 to 5 with 1 = never or almost never, 2 = seldom, 3 = sometimes, 4 = often, and 5 = always or almost always.

The top five competencies, along with the assessment process, were introduced to students the first day of the course. The students completed an online initial competency self-assessment the first week of class focused on these five competencies. During the second week of class, industry teams were formed, and industry mentors were assigned for the semester’s lean manufacturing project. During the first five weeks, students experienced in-class simulations and other instructional activities involving lean tool applications, including: 5S, value stream mapping, A3, standard work, JIT, and jidoka ( Pascal, 2007 ). At mid-term, student teams presented their lean project progress/status overview, and completed an “initial” online peer/team member assessment. The instructor provided confidential peer feedback to each student the following week. The student lean project teams spent the next five weeks predominantly out of the classroom working on-site with their industry mentors. During the 14th week, final self- and peer-competency assessments were completed. The instructor provided confidential results for peer assessments the following week.

Results

All initial and final competency assessments were analyzed with SPSS 19 software using paired sample t-testing. The t-test is the optimal data analysis method used to compare the means of paired samples and is recommended for small sample sizes (N < 30). The selfand peer-competency assessments were assigned to all students. One student didn’t complete the initial, and another didn’t complete the final self-assessment. These were not included in the data analysis (N = 24). Definitions of the top five competencies are shown in Table 1. The competencies’ key action items, shown in Table 2, were assessed, and an average value was reported.

Self-Assessment

The average results for key action items within each of the top five competencies, based on the initial and final self-assessments are shown in Figure 1. Additionally, Table 3 provides specific paired t-Test self-assessment results for the key actions (KA). Significant differences (p < .05) are indicated with an asterisk (*). Overall, an increase in final over the initial assessed average value was found in at least one key action (KA) item (*) for each of the five top competencies. These measured average increases serve as an indicator of selfassessed professional development.

Figure 1. Self-assessed average for five workplace competencies' key actions. (N = 24). *p < .05, two-tailed. Self-assessment average results. The Likert scale used was based on how often a key action was performed, ranging from 1 to 5 with 1 = never or almost never, 2 = seldom, 3 = sometimes, 4 = often, and 5 = always or almost always.

Self- vs. Peer Assessments

A comparison of the results for the key actions between all self- and peer assessments is shown in Figures 2 and 3, respectively. In the initial assessment, significant differences (*) were detected in specific key action items in two of the five competencies (analysis and judgment, and teamwork), between self and peer results. In all cases, the self-assessed average results were higher than peer-assessed average results. In the final assessment results, significant differences in specific key action item averages were also found for two of the five competencies (initial and teamwork). Once again, self-assessed average values were higher than peer-assessed average values. Results indicate that for both the initial and final assessments, KA2 in the teamwork competency was the significant difference commonality. Additionally, Table 4 provides specific paired t-Test self-assessment results for the key actions (KA). Significant differences (p < .05) are indicated with an asterisk (*).

Figure 2. Initial self- vs. peer-assessed average ranking for key actions. (N = 24). *p < .05. The Likert scale used was based on how often a key action was performed, ranging from 1 to 5 with 1 = never or almost never, 2 = seldom, 4 = often, and 5 = always or almost always.

Peer Assessments

The average results for key action items contained within each of the top five competencies for the initial and final peer assessments are shown in Figure 4, with significant differences (p < .05) indicated with an asterisk (*). Overall, in four of the five competencies, significant differences (*) in the average assessed value were found in at least one key action item. These key action items experienced an increased average value in the final average assessed value over the initial. As a peer assessment/student aggregate, this measured increase serves as an indication of professional growth over the semester.

Figure 3. Final self- vs. peer-assessed average rankings for key actions (N = 24) *p < .05. The Likert scale used was based on how often a key action was performed, ranging from 1 to 5 with 1 = never or almost never, 2 = seldom, 4 = often, and 5 = always or almost always.

Findings

A 360-degree assessment process was implemented into an undergraduate course utilizing the department’s competency assessment format. Key action items associated with the top five course competencies were assessed. The self-assessment results showed higher final average assessed values in at least one key action item for each of the five competencies. No commonalities in the key action items between the initial and final self-assessment results were observed. The measured increases in key action final average results indicated self-perceived professional gains achieved (Figure 1). The comparison of self- v. peer-assessment results showed two commonalities:

1. Higher average values were all detected in the self-assessments, and
2. Teamwork competency: KA2 showed higher self-assessed values in both the initial and final assessments (Figures 2 and 3).

The overall peer-assessment results showed higher average final results in at least one key action item for each of the five course competencies (Figure 4).

Discussion and Conclusions

The results are indicative of the complex task of comparing self-perception to others, which involves social information processing and interpersonal insight ( London, 1994 ). Psychological mechanisms related to how we operate in social environments may become impediments to accurate self-assessment. In this study, significant differences detected in comparing self-assessments v. peer assessments showed self-assessments with higher average values. As Tornow (1993) found, self-assessments are, on average, higher than others, including peers. Although peer ratings often tend to be far lower than self-ratings, they are fast becoming one of the most valued sources of appraisal as opposed to the usual supervisor ratings ( McCarthy & Garavan, 2001 ). According to Jones and Bearley (1996) , this is a direct consequence of an organization’s increased focus on self-managed work teams and flatter structures. Peer feedback provides insight into how one behaves in team situations; it also explains the influencing behaviors that serve to gain commitments when no direct authority can be exercised ( Lepsinger & Lucia, 1997 ). Classroom research has demonstrated reasonable agreement between self- and peer ratings ( McGourty, Dominick, Besterfield-Sacre, Shuman, & Wolfe, 2000 ), and correlations ranging from 0.12 to 0.39 ( Reilly, 1996 ) have been reported. Correlations results found in this study ranged from –0.429 to 0.534 for the initial self- vs. peer assessments and from –0.394 to 0.354 for the final assessments. Researchers have suggested that low agreement may be due to real behavioral or skill differences in the target student as perceived by sources with different perspectives such as fellow students ( Tornow, 1993 ).

Figure 4. Peer-assessed average ranking for key actions (N = 24) *p < .05. Assessed average results for each key action (KA) related to the top 5 course competencies. The Likert scale used for assessment was based on how often a key action was performed, ranging from 1 to 5 with 1 = never or almost never, 2 = seldom, 3 = sometimes, 4 = often, and 5 = always or almost always.

This case study was limited to the assessment of the top five workplace competencies determined for one course, one semester (16 weeks), and small sample size (N = 24). A great deal of research has been directed at the relationship between individual characteristics and rating tendencies; research has focused on characteristics of the raters, the ratee, or both. These characteristics were not the central focus of this study. Additionally, self- and peer evaluations are not entirely free of bias, also not addressed in this study. Rather, the focus was to determine if competency assessment could be implemented into the classroom to measure and detect evidence of perceived student professional development.

The value of competency assessment as a measure both in this study and in industry is that it provides input that can be utilized into professional self-development efforts. This study provided a framework for competency-based learning and assessment that can be utilized in a higher education environment. Despite its limitations, the implications for future research are evident. More studies are needed to collect and analyze data regarding competency-based learning and the use of multisource/360-degree assessments to measure student professional development in an educational setting. It gives us an inkling of the possibilities and impact that future studies can provide, not only to improve our approach to student assessment, but also in curricular improvement efforts that better prepare students for their professional endeavors.

Dr. Jacqulyn A. Baughman is a Lecturer in the Department of Ag & Biosystems Engineering, and Director of Graduate Education for the Biorenewable Resources and Technology Graduate Program at Iowa State University, Ames. She is a member of the Alpha Xi chapter of Epsilon Pi Tau.

Dr. Thomas J. Brumm is an Associate Professor in the Department of Ag & Biosystems Engineering, Director of Assessment for the College of Engineering, and Professor-in-Charge of Engineering-LAS Online Learning at Iowa State University, Ames. He is a member of the Alpha Xi chapter of Epsilon Pi Tau.

Dr. Steven K. Mickelson is the Charles R. & Jane F. Olsen Professor of Engineering and Chair of the Ag & Biosystems Engineering Department at Iowa State University, Ames.

References