JVTE v15n1 - Vocational Education Teachers' Perceptions of Their Use of Assessment Methods
Vocational Education Teachers' Perceptions of Their Use of Assessment Methods
Howard R. D. Gordon
Marshall University
Abstract
The purpose of this study was to describe West Virginia secondary vocational education teachers' use of student assessment information in making instructional decisions. A cluster sample of 240 teachers was needed for the study. A four-part questionnaire was designed to measure selected variables. Descriptive statistics were used to summarize the data. Teachers had an average of 15 years of teaching experience and nine years of related work experience. Attitudes toward assessment were viewed as "positive" by respondents. Secondary vocational education teachers neither agreed nor disagreed that they were constrained in their assessment activities. However, vocational education teachers must have opportunities during the school day to collaborate on the analysis of student work and to plan appropriate instructional improvements.
The Carl D. Perkins Vocational and Applied Technology Education Act of 1990 required accountability of all states that accept federal funds to support vocational programs. This was to be achieved through a system of specified performance measures and standards, which track both academic and occupational competency gains. Measures of performance were requested to address occupational competency attainment while measures of learning and competency gain were to reflect the achievement of basic and advanced academic skills. Goals 2000 and the School-to-Work Opportunities Act are two of the federal acts that emphasize high standards for all students while providing a framework and some financial incentives for public education to prepare all students for the world beyond school.
Elegant arguments can be mounted regarding the probable effectiveness of the goals, testing, and standards initiatives. One may raise questions about institutional capacity, available resources, and the actual feasibility of the mandate type of policy approach. The more immediate concerns of many, however, are more technical than philosophical. Many measurement authorities, as well as school practitioners, are concerned about the ability of standardized measures commonly available to measure the desired outcomes with an acceptable degree of validity and precision (Asche, 1991) .
Statewide assessment cannot attempt to measure and thereby reflect all that local schools are able to achieve in terms of student outcomes. It has become necessary for schools to measure the attainment of their unique educational objectives (Perlman, 1991) . The logical place for assessment activities measuring student achievement and performance at the local level is in the classroom under the auspices of the classroom teacher. Wolf (1992) indicated that American education would become galvanized when new and more probing assessments are utilized to hold districts, teachers and students more accountable.
The American Federation of Teachers, the National Council on Measurement in Education, and the National Education Association (1990) supported the concept of an instructional-assessment linkage by subscribing to the view that "student assessment is an essential part of teaching and that good teaching cannot exist without good student assessment" (p.1) . Warmbrod (1993) drew attention to the linkage between instruction in agricultural education and assessment. He stated that "a dimension in the content of agricultural education that has roots in the critical analysis and systematic study of teaching is the assessment of learning and the evaluation of the effectiveness of educational programs" (p.7) .
Prior to the 1980s, the primary focus of research in assessment was not focused on the classroom but rather upon the area of standardized forms of assessment (Stiggins & Bridgeford, 1985) . Classroom assessments, made by teachers have not been examined in depth. Stiggins and Conklin (1992)stated that
"Although we were in an outstanding position to construct and administer high-quality, large-scale-testing programs in the early 1980s, we were far less able to teach teachers how to address the task demands of the day-to-day measurement of student achievement." (p. 11)
A review of literature identified six general categories of classroom assessment methods as representing the various individual assessment instruments or procedures which teachers use to generate information for decision-making. The categories of assessment methods revealed were (1) objective paper and pencil items, (2) standardized test scores, (3) performance assessments, (4) informal observations, (5) essay-type items, and (6) portfolios.
The educational measurement community is engaged in an equally serious rethinking of the structure of assessment (Wolf, 1992 ; Mehrens, 1992 ; Wiggins, 1989) . Traditional, selected-response methods (multiple choice, matching, true-false) are being criticized for a variety of reasons: they can lead to narrowing of curriculum, test preparation practices may inflate scores in high stakes situations, and there are consistent differences in average performance between racial/ethnic and gender groups. (Koretz, Linn, Dunbar, & Shepard 1991 ; Shepard & Dougherty, 1991 ; Smith & Rothenberg, 1991) .
Many educators believe that assessment can play an important role in systematic educational changes such as those being envisioned for vocational education (Wolf, 1992) . However, there is a paucity of information in the literature about the assessment practices of vocational education teachers. How vocational education teachers use assessment information in the classroom and whether its use is effective can play a major role in enhancing and documenting both instruction and learning.
Purpose and Objectives
The purpose of this study was to describe West Virginia secondary vocational teachers' use of student assessment information in making instructional decisions. The specific objectives were as follows:
- To describe selected characteristics of secondary vocational education teachers in West Virginia: age, gender, related work experience, teaching experience, program area taught, education level, and certification route.
- To describe secondary vocational education teachers' perceptions of their use of student assessment data for making instructional decisions.
- To describe secondary vocational education teachers' attitudes toward the assessment process.
- To depict secondary vocational education teachers' perceptions of constraints in the assessment process.
Method
Research Design
This descriptive study was designed to examine the use of assessment information, obtained from six types of student assessment methods, in addressing 10 instructional decisions. The dependent variable in this study was teacher use of student assessment information in instructional decision making. The variables, attitudes toward assessment and constraints to the assessment process were the independent variables of interest.
Population and Sampling
The target population ( N = 647) was all teachers in West Virginia who taught full time at secondary vocational technical centers during the 1997- 98 school year. The 1997 - 98 West Virginia Education Directory was used to identify the population, which served as the sampling frame for the study. Using the formula recommended by Krejcie and Morgan (1970), a sample of 240 teachers was needed for the study. A cluster sampling technique was used to randomly select secondary vocational education teachers from the 32 secondary vocational technical centers in the state of West Virginia. Twelve secondary vocational technical centers were randomly selected to participate in the study in order to achieve the desired sample size of 240 (12 schools with an average of 20 teachers per school). Secondary vocational technical centers were numbered from 1 to 32, and the 12 schools were selected using the random number generator in Microsoft Excel. According to Gay (1996) , cluster sampling is more convenient when the population is very large or spread out over a wide geographic area. Sometimes it is the only feasible method of selecting a sample. It is not always possible, for example, to obtain or compile a list of all members of the population; thus in such cases, it is not possible to use simple random sampling.
Instrumentation
A four-part questionnaire was designed by the investigator for use in measuring selected variables. The dependent variable, teacher use of assessment information, was measured in part one of the instrument. The six types of assessment methods used in the study included: objective paper and pencil items, standardized test scores, performance assessments, informal observations, portfolios, and essay type items.
Participants were asked to indicate the extent to which they use results, from each of the six assessment methods, in making 10 different types of instructional decisions. The 10 instructional decisions that were addressed were: planning instruction, diagnosing student weakness, monitoring student progress toward course objectives, communicating student achievement with parents, motivating students to learn, evaluating the effectiveness of instruction, evaluating the instructional materials used, grouping students for instructional activities, encouraging students to assess their own work, and assigning grades. A 5-point Likert scale ranging from "no use" to "considerable use" was used.
Attitudes toward assessment were measured in part two of the questionnaire using a semantic differential scale. The scale was comprised of nine bi-polar adjectives, which described the concept "assessment. " A seven-point scale was used for each pair of adjectives. Participants were asked to use the scale for each adjective pair to describe their attitudes toward the overall assessment.
In part three of questionnaire, nine statements were used to measure teachers' perceptions of constraints to their assessment activities. Major constraints identified in the literature included: time, money, technology and assistance, training, autonomy in making assessment-related decision, and availability of assessment materials. The 5-point Likert scale used in this section ranged from strongly disagree to strongly agree. Participants were asked to indicate their level of agreement with each of the constraint statements. Teacher characteristics were listed in part four of the instrument.
To ensure validity of the data, a panel of experts was used to establish content and face validity. The panel consisted of three vocational education teachers, a regional teacher educator, and two professors of vocational education. The instrument was field-tested for reliability with a sample of vocational education teachers ( n = 11) not selected for participation in the study. Changes indicated by the validation panel and field test were made. Internal consistencies for the scales in the instrument were as follows (Cronbach's alpha): Use of Assessment Information .94, Attitudes toward Assessment .95, and Constraints in Assessment .64, acceptable according to Nunnally and Beinstein (1994) .
Data Collection
The Total Design Method (TDM) of conducting surveys (Dillman, 1978) was followed in all stages of the questionnaire construction and implementation process. A packet containing a cover letter, instructions for administering the questionnaire and copies of the questionnaire were mailed to the principal of each school selected for the study. A total of 240 questionnaires were sent to 12 principals during October of 1997. A total of 144 usable questionnaires were returned for a response rate of 60%. Because a number of questionnaires were returned uncompleted and there was no way to conduct appropriate follow-up procedures to control for non-response, the results of the study can only be generalized to the 144 teachers who provided usable data.
Data Analysis
Data were analyzed using the Statistical Package for the Social Sciences (SPSS Version 6.1 for Windows). Descriptive parameters, including frequencies, percentages, means, and standard deviations were used to organize and summarize the data.
Results
Demographic Characteristics
The largest number of teacher respondents fell within the 42-51 year age bracket (52.1%). Of the 144 cases, 56% of the respondents were male and 44% were female. Teachers had an average of 15.02 years of teaching experience and 9.66 years of related work experience. Trade and industrial education teachers comprised the largest group in the sample and accounted for 43% of the cases.
A graduate degree had been earned by 39% of the respondents. Respondents, who had completed a teacher preparation program on the job, and before receiving a bachelor degree, comprised 43% of the cases.
Teacher's Use of Assessment Information
Data in Table 1 depict teachers' uses of assessment information and educational decision making. Information generated from performance assessment was considered to be of more use to teachers in addressing educational decisions than any of the other assessment methods ( M = 4.19). Teachers rated performance assessment information to be of much use ( M = 4.45) when specifically addressing the task of assigning grades.
Teachers reported that information derived from informal observations ( M =3.96) was of more use than all other assessment methods except performance assessment. Informal observations provided information that was considered by teachers to be of much use ( M = 3.80 - 4.16) when addressing all 10 decision areas.
Information obtained from objective paper and pencil items ( M = 3.91) was found to be of less use than both performance assessments and informal observations, but of more use than essay items, portfolios, and standardized test scores. Information from this assessment method was found to be of much use ( M = 3.68 - 4 .27) for nine of the decision areas. Teachers revealed that objective and paper pencil items were of some use when grouping students for instructional activities.
Portfolios were found to be of only some use ( M = 2.81) in providing information in the overall decision making process. However, teachers found portfolios to be of some use ( M = 3.02) when specifically communicating student achievement results to parents.
Standardized test scores ( M = 2.75) and essay type information ( M = 2.72) were the two assessment methods found to be of less importance to teachers when compared to the other methods. Teachers reported that standardized test scores were of some use when evaluating the effectiveness of their teaching methods ( M = 3.03) and of limited use ( M = 2.37) when assigning grades. For addressing educational decisions, information obtained through the implementation of essay type methods was found to be of some use for all ten decision areas ( M = 2.50 - 2.84).
Table 1
Means a Standard Deviations for the Use of Assessment Information Generated From Six Assessment Methods (n=144)
Decision Area
Objective Item
Standardized Test Score
Performance Assessment
Informal Observation
Portfolio
Essay Item
M SD
M SD
M SD
M SD
M SD
M SD
4.00 1.01
2.84 1.13
4.24 .82
4.09 .87
2.68 1.17
2.81 1.17
3.96 .99
3.02 1.17
4.20 .93
4.16 .86
2.73 1.24
2.84 1.18
4.13 .92
2.72 1.19
4.27 .89
4.00 .95
2.78 1.23
2.81 1.10
3.89 1.15
2.83 1.27
4.03 1.02
3.80 1.09
3.02 1.35
2.72 1.20
3.70 1.12
2.62 1.10
4.27 .83
4.00 .97
2.85 1.25
2.61 1.09
4.14 1.05
3.03 1.29
4.36 .78
4.06 .90
2.89 1.28
2.82 1.30
3.90 1.03
2.85 1.20
4.06 .91
3.84 1.02
2.79 1.21
2.72 1.15
3.49 1.18
2.67 1.22
3.93 1.10
3.95 1.12
2.62 1.24
2.50 1.18
3.68 1.23
2.60 1.23
4.16 .97
3.80 1.13
2.97 1.30
2.60 1.27
4.27 .97
2.37 1.36
4.45 .86
3.97 1.03
2.78 1.36
2.77 1.27
Overall Means
3.91
2.75
4.19
3.96
2.81
2.72
Note : Educational Decision Areas:
1. Plan for instruction
6. Evaluate instruction
2. Diagnose student weakness
7. Evaluate instructional materials
3. Monitor student progress
8. Group students
4. Communicate student achievement
9. Encourage self-assessment
5. Motivate students
10. Assign grades.
a Based on scale: 1= of no use; 2= of limited use; 3= of some use; 4= of much use; 5= of considerable use.
Attitudes toward Assessment
A semantic differential scale was used to measure teachers' attitudes toward the overall assessment process. Teachers were asked to respond to nine parts of bi-polar adjectives, which were on either end of a 7-point scale. Figure 1 provides an overview of the distribution of mean teacher responses along a 7-point scale. The scale was as follows: 1 = extremely negative; 2 = very negative; 3 = negative; 4 = neutral; 5 = positive; 6 = very positive; 7 = extremely positive. Teachers reported that they perceived the assessment processes to be valuable, efficient, and important ( M = 5 .77 - 5 .38). Overall, there was no indication that a negative attitude ( M < 4 .0) existed towards the assessment process.
Worthless
x
Valuable
Unsuccessful
x
Successful
Inefficient
x
Efficient
Unimportant
x
Important
Bad
x
Good
Unfair
x
Fair
Disreputable
x
Reputable
Rigid
x
Flexible
Tense
x
Relaxed
Scale 1-7
1
2
3
4
5
6
7
Figure 1. Distribution of Attitude Mean Scores
Constraints to the Assessment Process
Data in Table 2 describe responses to each of the constraint questions. Teachers tended to agree that they decided on what assessment methods to use in their courses ( M = 4.28). Teachers also tended to agree that additional planning time would allow for assessment methods to be used more effectively ( M = 3.94). Over two-thirds of the items were rated as "neutral" by respondents in this study ( M = 2.69 - 3.00).
Table 2
Means a and Standard Deviations for Teacher Perceptions of Constraints to the Assessment Process (n = 144)
Item
M
SD
I decide what assessment methods to use in courses I teach
4.28
.97
Additional planning time would allow me to use assessment methods more effectively
3.94
1.10
Equipment is available in my school for use in scoring tests
3.06
1.41
Quality published assessment materials are hard to find
3.00
1.18
Funds are available for buying published assessment materials
2.91
1.32
College courses were of little help in preparing me to assess student learning
2.86
1.25
In-service activities have helped developed my assessment skills
2.81
1.15
I have assistance in preparing student assessment activities
2.75
1.37
I do not have information on published assessment materials
2.69
1.20
Note. a Based on scale: 1 = strongly disagree; 2 = disagree; 3 = neither agree or disagree; 4 = agree; 5 = strongly agree.
Discussion and Conclusions
Based on the results of this study, the typical secondary vocational education teacher:
- is likely to be in the age bracket of 42-51 years old;
- completed an average of 15 years of teaching and 10 years of related work experience;
- completed a graduate degree;
- completed a teacher preparation program on the job; and
- is likely to be a teacher in trade and industrial education.
Some of these findings are partially explained in a study reported by Lynch (1993). According to Lynch's study (1993) , secondary vocational education teachers tend to have less formal education than others, but they have more related occupational experience and credentials. This emphasis on occupational experience in lieu of formal education is concentrated in trade and industrial education, where it has been guided by state policies in a tradition going back to back to the Smith-Hughes Act of 1917 (U.S.Congress, 1917) .
Performance assessments are particularly useful to secondary vocational education teachers. Given the fact that vocational education teachers use a competency-based curriculum, this finding revealed that performance assessment was of more use than the other five methods used in this study. This finding supports a study completed by Kershaw (1993) . According to Kershaw's study, secondary vocational education teachers in Ohio were more likely to use performance assessments than any other assessment methods.
Secondary vocational education teachers were more likely to use informal observations when grouping students for instructional activities. This finding suggests that when prompted, vocational education teachers can provide rich and detailed descriptions of their pupils. Secondary vocational education teachers placed much use on objective paper and pencil methods in assigning grades. This finding indicates that this method is well suited for assessing students' recall of factual knowledge. Portfolios were not found to be of much use for generating information in decision making. This was probably due to the fact that teachers have used different criteria for rating portfolio work or come up with different scores even when they use the same criteria (Borthwick, 1995) .
Secondary vocational education teachers were less likely to use standardized tests and essay type methods for assessing students than objective items, performance assessments, informal observations, and portfolios. This finding is supported by other studies (Green, 1990 ; Kershaw, 1993) regarding the lack of standardized test scores in addressing educational decisions. This finding may also suggests that secondary vocational education teachers are not proficient in interpreting the use of information generated from standardized assessments, as well as how to construct and score essay questions for assessments.
Attitudes toward assessment were viewed as being positive by secondary vocational education teachers. This suggests that vocational education teachers rely on the information generated by tests to provide them with the basis for improving instruction. In their review of literature, Scharfer and Lissitz (1987) concluded that although teachers may be ill trained to use accepted measurement practices, they see assessment as an important part of their professional role and have a positive attitude toward it.
Secondary vocational education teachers neither agreed nor disagreed that they were constrained in their assessment activities. The results of this study mirror the findings of previous studies on constraints and assessment use (e.g., Gullickson, 1984 ; Kershaw, 1993) . In addition to the predominantly neutral findings, both studies cite "limited time for planning" as a high level of agreement among respondents. Secondary vocational education teachers may therefore have less motivation to use the data if there is lack of time to address problems related to assessment quality.
Implications and Recommendations
There are a variety of ways for students to discover their workplace-related skills and knowledge with the help of educators. It is important to understand, however, that no single assessment method can completely measure a student's range of skills and knowledge in a content area. Thus, it is necessary to use several types of assessment methods to help students learn about their skills in even a single content area. Both written, short-answer forms of testing and diverse methods of performance assessment are likely to be used by vocational educators. However, vocational educators rely much more on diverse methods of performance assessment.
So far there are no clear, unambiguous rubrics for evaluating each aspect of a portfolio. States that have begun to use portfolios on a large scale have had difficulty achieving acceptable quality in their scoring (Stecher & Herman, 1997) . However, one approach is to establish guidelines for the contents of the portfolios so that they all contain similar components. Specific learner outcomes can be identified for each component and then techniques can be developed for assessing student performance in terms of these outcomes.
Based on the findings of this study, it is recommended that pre-service and in-service providers place more emphasis upon essay construction and scoring. They should also place more emphasis on interpreting the use of information generated from standardized test scores.
Teachers must have opportunities during the school day to collaborate on the analysis of student work and to plan appropriate instructional improvements. School administrators can convey the importance of this work by providing time for assessment as an integral part of teachers' responsibilities.
One of the main criticisms leveled against traditional assessments is that they are used to sort students and, on that basis, to deny educational opportunities (Darling - Hammond 1991) . The consequence of instructionally sound assessment is quite different -- it enhances the opportunity to learn. The assessment data are not used to label students. They simply provide information on areas in which students already do well, and focus on what they need to learn. Vocational teachers can use the results of such assessments to determine appropriate learning experiences and to guide the redesign of school programs and structures so that teacher and student performance improves. Obviously, trends in assessment will impact workforce education practice. For this reason, vocational educators need to take an active role in debates regarding assessment techniques and criteria.
Educators and employers believe that the work world is changing and vocational education must adapt if it is to serve students well. The changes in the workplace are complex and not completely understood, but most observers believe that future employees will need integrated academic and vocational knowledge, a broad understanding of occupational areas, the ability to interact creatively with their peers, and higher-order cognitive skills that allow them to be flexible, learn rapidly, and adapt to ever-changing circumstances. To the extent this belief is true, vocational training needs to place greater emphasis on integrated learning, critical thinking-skills, and connections between vocational and academic skills, rather than on the mastery of the narrow, occupation-specific skills that characterized vocational education in the past. This new vision may also require broader changes in vocational education, including rethinking the organization, goals, content, and delivery of services, as well as the manner in which students and programs are assessed.
References
American Federation of Teachers, National Council on Measurement in Education, & National Education Association. (1990). Standards for teacher competence in educational assessment students . Washington, DC: Authors.
Asche, F.M. (1991). Educational reform and vocational education: Review with implications for research and development . The Journal of Vocational Education Research, 16 (3), 1-34.
Borthwick, A. (1995). Body of evidence. Vocational Education Journal, 70 (3), 24-26,48.
Darling - Hammond, L. (1991). The implications of testing policy for quality and equality. Phi Delta Kappan, 73 , 220-225.
Dillman, D. (1978). Mail and telephone surveys: The total design method . New York: John Wiley & Sons, Inc.
Gay, L. R. (1996). Education research: Competencies for analysis and application (5 th ed.). New Jersey: Prentice Hall, Inc.
Green, K. E. (1990, April). Differences between teachers and students in opinions about testing and test use. A paper presented at the annual meeting of the National Council on Measurement in Education, Boston, MA. (ERIC Document Reproduction Service No. ED 318 775)
Gullickson, A.R. (1984). Teacher perspectives of their instructional use of tests. The Journal of Educational Research, 77 (4), 244-248.
Kershaw, I., IV. (1993). Ohio vocational education teachers' perceived use of student assessment information in educational decision making . Unpublished doctoral dissertation, The Ohio State University, Columbus.
Krejcie, R.V. & Morgan, D.W. (1970). Determining sample size for research. Educational and Psychological Measurements, 30 , 607-610.
Koretz, D., Linn, R., Dunbar, S., & Shepard, L. (1991). The effect of high stakes testings achievement . A paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
Lynch, R.L. (1993). Vocational teacher education in the U.S. colleges and universities and its responsiveness to the Carl D. Perkins and Applied Technology Education Act of 1990 . University of Georgia, School of Leadership and Lifelong Learning.
Mehrens, W. (1992, Spring). Using performance assessment for accountability purposes. Educational Measurement: Issues and Practice, 11 (1), 3-20.
Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory . New York: McGraw Hill.
Perlman, C. L., (1991, April).Developing a comprehensive assessment program. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. (ERIC Reproduction Document Service No. ED 330 715)
Schafer, W.D., & Lissitz, R.W. (1987). Measurement training for school personnel: Recommendations and reality . Journal of Teacher Education, 38 (3), 57-63.
Shepard, L., & Dougherty, K. (1991). Effects of high stakes testing on instruction . Paper presented at the annual meeting of the American Educational Research Association and the National Council on Measurement in Education, New Orleans, LA.
Smith, M.L., & Rottenberg, C. (1991). Unintended consequences of external testing in elementary schools . Educational Measurement: Issues and Practice, 10 (4), 7 -11.
Stecher, B.M., & Herman, J. (1997). Using portfolios for large scale assessment. In G.D. Phye (ed.), Handbook of classroom assessment. San Diego: Academic Press.
Stiggins, R. J., & Bridgeford, N. J. (1985). The ecology of classroom assessment . Journal of Educational Measurement, 22 (4), 271-186.
Stiggins, R. J., & Conklin, N. F. (1992). In teachers' hands: Investigating the practices of classroom assessment . Albany, NY: State University of New York Press.
U.S. Congress. (1917). Smith-Hughes Act of 1917 (P.L. 64-347). Washington, DC: Government Printing Office.
Warmbrod, J.R. (1993). Scholarship in agricultural education: It's essence and mood . Unpublished manuscript. The Ohio State University, Department of Agricultural Education, Columbus.
Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70 (9), 703-713.
Wolf, D. P. (1992). Good measures: Assessment as a tool for educational reform. Educational Leadership, 49 (8), 8-13.