Effectiveness of and Student Satisfaction with Web-Based Compared to Traditional In-service Teacher Education Courses
David J. Pucel
Thomas F. Stertz
University of Minnesota
Since the Smith-Hughes Act in the early 1900's, there has been a continuing need to prepare people who possess competence in occupational fields to teach in high schools, technical colleges, and community colleges. Most often these people begin teaching with little teacher education. They are hired first, and later receive in-service teacher education while they are teaching. Originally, most states offered these in-service courses through land grant institutions that were subsidized with funds from the Smith-Hughes Act. Itinerant teacher educators were hired to provide the new occupational teachers with the needed pedagogical instruction. As the fields of trade and industrial and technical education grew, the need for occupational teachers and, consequently, for in-service teacher education increased. In an effort to become more efficient and to provide more opportunities for new teachers to receive in-service training, there was a move to develop teacher education modules that could be offered to occupational teachers and be supported by local resource personnel and teacher educators. Some institutions, such as Ohio State University, continue to use modules as a key part of their career and technical education (CTE) teacher education programs. As the technology for teacher education became more sophisticated, some institutions began to provide instruction through interactive television.
More recently, there has been a move within teacher education in general, and career and technical education in particular, to provide Web-based instruction (WBI). Some institutions, such as Northern Arizona University, offer entire bachelor's and master's degree programs online. The National Research Center for Career and Technical Education (NCCTE) concluded the following based on a national study of CTE teacher education programs. "Increasingly, it appears that the profession is looking toward more distance education as a means to deliver education…In the next three years, the number of Web-based courses and interactive courses is expected to increase by at least one-third" (NCCTE, 2001, p. 49).
The movement toward online instruction has a number of driving forces. "In response to changing student populations and more competition among institutions, higher education is changing in a number of ways. One involves offering students more choices in ways which 'stretch the mold' of campus-based participation and traditional approaches to teaching and learning" (Collins, 2003, p. 2). The National Center for Educational Statistics (Sikora & Carrol, 2002) indicated that during the 1999–2000 academic year, 7.6% of undergraduates in the United States took some form of distance class. Many colleges and universities have been expanding their WBI. Harasim (2000) writes: "The 21st century thus begins with a paradigm shift in attitudes towards online education… Our new understanding of the very nature of learning has affected the definition, design, and delivery of education" (p.42).
Web-based instruction has several characteristics that tend to drive its increased use in education. First, Web-based delivery permits educational processes to be implemented at times and places that fit around working students' job and family responsibilities. Second, the variety of teaching tools available in Web-based delivery provides instructional options that are at times superior to those typically available within regular classroom instruction. Currently there are many examples of how online and Web-based instruction is being designed and used to meet educational needs around the world. The recent International Association of Science and Technology for Development (IASTED) International Conference on Computers and Advanced Technology in Education (CATE) held in Rhodes, Greece on June 30–July 2, 2003, included 141 papers contributed by individuals from 48 different countries. Many addressed how to develop instruction for a variety of purposes using various WBI formats or discussed how WBI was implemented (IASTED, 2003).
The impact of Web-based instruction on higher education is readily apparent. "Universities are witnessing a profound increase in the use of multimedia presentations, video teleconferencing, and, more currently, Web-based course instruction. These technological innovations have a direct impact on current university practices and policies and subsequently have the potential to alter our traditional definitions of education" (Daugherty & Funke, 1998, p 23).
Although the use of online WBI has been increasing, some educational institutions have been concerned that the quality and effectiveness of instruction may be compromised when providing instruction online rather than through traditional classroom-based instruction. The debate over the gain in flexibility and convenience for the student while taking online courses compared with the possible reduction in quality and effectiveness of instruction has been going on ever since online instruction became a possibility. Faculty also have major concerns about workload and curricular issues (Arnone, 2002). The Wall Street Journal (July 15, 1998) reported that a "survey of faculty at Florida Gulf Coast University— designed and built as a testing ground for Internet-based distance learning—found 54% of faculty disagreeing with the statement: 'At FGCU, distance learning is an effective alternative to traditional instruction.' "
Ultimately, it is not its numerous unique features that will determine the effectiveness of WBI as a teaching tool. WBI is only a vehicle for designing a learning context—one approach, one strategy, for conveying knowledge to individuals (Clark, 1994). The pedagogical soundness of this medium involves many factors that have yet to be investigated fully (Rovai & Barnum, 2003).
"It is apparent that the emergence of distance education in traditional higher education has altered and will continue greatly to alter teaching and learning contexts in our universities. However, there is little research on the effects of these changes. Studying various approaches for integrating technology and instruction, and the results of those efforts, will address the role of distance education in university instruction more responsibly and, we hope, ensure and protect academic quality for our students" (Daugherty & Funke, 1998, p. 24).
Thus, there is a continuing need for more research on the relative effectiveness of online WBI as compared with traditional classroom instruction. The two measures most often used to measure effectiveness found in the literature are student performance and student satisfaction.
The literature on student performance is mixed. Rovai & Barnum (2003) studied 19 graduate courses to determine how perceived learning varies by course. They found substantial variation. They concluded, "In the context of distance education, quality assurance seeks to balance course design, pedagogy, and technology with the needs of learners. Because the quality of educational programs is valued by school administrators, online courses should reflect a stable and repeatable process" (Rovai & Barnum, 2003, p. 68). They also concluded that, just as any other form of instruction, each course needs to be evaluated separately and, due to wide differences in courses, it is difficult to generalize regarding the effectiveness of WBI.
A number of studies have specifically addressed student satisfaction with WBI. Furst-Bowe (1997) compared students' reactions to a videoconferencing-based course to a traditionally delivered course. Her conclusion was that students found few differences between the two experiences. Day, Raven & Newman (1998) compared the attitudes of students towards writing in two versions of a technical writing course. The course was delivered traditionally to one group of students and through distance education to a second group. They found significant improvement in the attitude of the Web-based group toward technical writing compared with the traditional group (p. 75). The literature has shown mixed findings regarding student satisfaction. Therefore, Allen, Bourhis, Burrell, and Mabry (2002) did a meta-analysis summarizing the results of 24 studies that compared comparable courses taught through distance education and traditional methods. Their overall conclusion was that "…students find distance education as satisfactory as traditional classroom formats" (p. 93).
This study looked at student performance and student satisfaction in two courses offered by the University of Minnesota to in-service career and technical education teachers who were preparing to teach occupational programs at the high school or technical college level. The study compared the student performance and satisfaction in two WBI-enhanced classes with the performance and satisfaction in equivalent, traditional classroom versions of the same two classes. The study was undertaken to develop a generalizable model for evaluating other WBI courses and to determine if the WBI versions of the courses produced similar results to traditional versions.
This study was conducted during an in-service program to prepare career and technical education teachers at both the high school and two-year college level. Due to a continuing lack of qualified individuals who are both occupationally competent and possess teaching degrees, schools hire occupationally competent people and provide them with teacher education as they are teaching. The occupational teachers teach in fields such as communications technology, construction careers, creative design careers, early childhood careers, hospitality service careers, manufacturing careers, medical careers, and transportation careers. These new teachers are sparsely spread out throughout the state of Minnesota. It is often difficult for the teachers to come to a university campus to take the required pedagogical courses while they are teaching full time. It is also not always cost effective for teacher education institutions to offer courses for these teachers in remote off-campus locations because of the low density of the teachers. Therefore, the Minnesota Department of Education asked the University of Minnesota to develop online-enhanced WBI course alternatives for some of its basic pedagogical courses. The two 2-semester hour courses addressed in this study were History and Philosophy of Career and Technical Education and Instructional Methods for Business and Industry. It was hoped that offering the classes in a WBI format would reduce the amount of travel and in-classroom time and provide flexibility for students completing course requirements without compromising either student satisfaction or performance.
To ensure that the instruction in the WBI-enhanced versions of the courses was as nearly equivalent as possible to the past traditional versions, two instructors with Ph.D.'s in education who had long histories of teaching the courses in a traditional manner, and who were interested in online instruction, re-formatted the courses for WBI. Specialists in the development of online instruction from the university's College of Education and Human Development provided guidance in using WebCT to assemble the material for online delivery.
The instructors used the same course objectives, the same course assignments, and the same grading criteria as they had when teaching the traditional versions of the courses. They also used the same readings and basic texts. Each course had two 6-hour classroom sessions in which the students met the instructors, made presentations, delivered materials, and/or took examinations.
Two doctoral candidates, one familiar with the career and technical education content of the courses and one with expertise in developing online instruction, provided additional support. The instructors provided the basic content and instructional format information to these support personnel, who entered the information into WebCT.
Each WBI course contained a detailed syllabus, instructional modules designed to lead students through the learning activities and related resources, project guidelines and expectations, and exercises that required students to participate in cooperative learning activities. In addition, the course materials included Web links, video clips, audio clips, and PowerPoint presentations.Course Validation
After the courses were developed, the course material was reviewed by the curriculum coordinator for off-campus offerings, faculty with expertise and prior experience in teaching the same courses, representatives from the Minnesota Department of Education, and the Associate Dean from the College of Education and Human Development, who was responsible for overseeing the development of distance education. Once the courses were approved, each was offered online during a seven-week period between March 2003 and July 2003.Evaluation Criteria
Student satisfaction and student performance in both the WBI version and the traditional classroom version of each course were evaluated using the same criteria. The first measure addressed in the study was student satisfaction. The University of Minnesota requires that all courses be evaluated by a standardized student course evaluation process. Results are tabulated and reported by the University Measurement Services Center. The six dimensions used to evaluate each course are listed in Tables 4 and 5 below. The study made use of these standardized course evaluations to gauge student satisfaction. The study compared evaluation responses from the two WBI courses to evaluations from traditional classroom versions of the same courses taught by the same instructors. In addition to the standardized evaluations, students in the WBI-enhanced courses were asked to describe their satisfaction with the blend of WBI and traditional instruction used in the WBI courses. They were also asked to compare the relative time they spent on the WBI courses as well as the difficulty level of the WBI courses with their past experiences in traditional courses.
The second measure the study addressed, student performance, was evaluated based on four different assessments used in each of the courses: two projects, a final test, and an overall final grade. To complete the two projects for the philosophy course each student was to (a) react to the mission statement of his or her employing institution and suggest revisions to up-date it and (b) develop a survey of critical issues facing his or her field, administer the survey to his or her colleagues, and compile the survey results. For the methods course, the two projects required each student to (a) keep a learning log recording the various teaching methods used during the course instruction and how the teaching methods learned in the course might be implemented in their own classrooms and (b) deliver a micro-teaching presentation, using a variety of methods, over material from his or her content area. The same scoring rubrics were used in both the WBI and the traditional versions of the courses.Data Gathering
In-service career and technical education teachers across the state of Minnesota took the WBI courses during the spring and summer of 2003. Classroom sessions were held at off-campus locations. Grading information included student project scores, final test scores, and final grades gathered during each WBI course. Student satisfaction data were gathered at the end of the courses using the University Student Evaluation survey. Comparable data were assembled for the traditional versions of the courses from past course records.
The traditional version of the philosophy course was held during fall semester 2002. The traditional version of the methods course was held during summer 2001. There were 22 students in the WBI version of the philosophy course and 15 in the traditional version. There were 22 students in the WBI version of the methods course and 25 in the traditional version. Not all students completed the student satisfaction evaluations.Limitations
A major limitation of this study is the assumption that the students who took the earlier, traditional versions of the courses were similar to those who took the later, Web-based enhanced versions. However, this assumption seemed reasonable since all of the participants in the study had been hired to teach in high schools or post secondary career and technical education programs and were required to take the courses as part of their continuing employment. There seems no reason to believe the two groups were significantly different.
Although the instructors, assignments and grading criteria were the same, a further limitation of the study is the assumption that the two versions of each course were equivalent. As much as possible, every care was taken to ensure parity.
Students were asked to indicate how much time they spent on the course work outside of the two classroom sessions for the WBI versions of the two courses. The average amount of time reported was 28 hours outside of the classroom for the philosophy course and 29.3 for the methods course. In addition, students compared the relative amount of time they spent on the WBI versions of the courses to their past experiences with traditional courses. The results are shown in Table 1. Of the philosophy students, 91% thought they spent an equal amount or more time on the WBI course. In the methods course, 82% of the students thought the WBI course required an equal amount or more time. About one-half of the students reported that they spent more time on the WBI courses than in traditional courses.
Estimate of the Time Spent on WBI Compared to Traditional Courses Time estimate Philosophy Methods n % n % More time 10 48 13 59 Same amount of time 9 43 5 23 Less time 2 9 4 18 Total 21 100 22 100
Students also rated the relative difficulty of the WBI courses compared to their past experiences with traditional courses. Table 2 presents the findings. Sixty-seven percent of the philosophy students and 73% of the methods students thought the WBI instruction was easier or equal in difficulty to the traditional courses.
Estimate of the Difficulty of WBI Compared to Traditional Courses Difficulty Philosophy Methods n % n % Less difficult 10 48 6 27 Same difficulty 4 19 10 46 More difficult 7 33 6 27 Total 21 100 22 100
Table 3 presents the students' ratings of their satisfaction with the blend of WBI and traditional instruction in the WBI courses. Ninety-one percent of the methods students indicated satisfaction with two 6-hour classroom sessions as part of the WBI course while only 52% of the philosophy students expressed satisfaction. Forty-three percent of the philosophy students said they would have preferred more classroom instruction in contrast to 9% of the methods students.
Satisfaction with the Blend of WBI and Traditional Instruction Satisfaction Philosophy Methods n % n % Satisfied 11 52 20 91 More WBI 1 5 0 0 More traditional 9 43 2 9 Total 21 100 22 100
To further examine the student level of satisfaction in the WBI versions of the courses as opposed to that in traditional courses, the study compared the students' standardized course evaluations of the WBI course with those of equivalent traditional classroom versions for each of the two courses. Students rated the six evaluation descriptors on a scale of 1 to 7, with 1 indicating poor and 7 indicating exceptional. Mean values of the responses to each of the descriptors were calculated. In order to determine if the responses of the WBI students were significantly different from the responses of the students in the traditional versions of the courses, the t-test for independent samples was used to analyze the data. Tables 4 and 5 list the six descriptors on the standardized evaluation form and the statistical results of the students' responses.
Student Evaluation of Philosophy and Practice of Career and Technical Education (Traditional: N = 11; WBI: N = 20*; df =29*; Two-tailed significant at .05 = 2.05) Student Perception Traditional WBI M SD M SD t-value 1. Instructor's overall teaching ability? 6.5 0.7 6.2 0.8 1.042 2. Instructor's knowledge of the subject matter? 6.7 0.4 6.5 0.6 .988 3. How much would you say you learned from this course?* 6.2 0.6 5.2 1.6 1.982 4. Overall quality of tests and handouts? 5.5 1.1 5.0 1.3 1.079 5. Helpfulness of feedback given to you about your performance? 5.0 1.5 4.7 1.5 .516 6. Degree to which evaluation procedures measured your knowledge and understanding? 5.2 1.2 4.8 1.3 .841 * Item 3. Traditional: N =11; WBI: N = 19; df = 28
There were no statistically significant differences in student evaluations between the WBI versions of the two courses and the traditional versions. In all but two cases the means were either slightly higher or equal on the student evaluations of the traditional versions of the courses. The only exception was that the students in the methods course indicated they learned more from the WBI version.
Student Evaluation of Instructional Methods for Business and Industry (Traditional: N = 26; WBI: N = 22; df =46; Two-tailed significant at .05 = 2.01) Student Perception Traditional WBI M SD M SD t-value 1. Instructor's overall teaching ability? 6.4 0.6 6.3 0.9 .459 2. Instructor's knowledge of the subject matter? 6.6 0.5 6.6 0.5 .000 3. How much would you say you learned from this course?* 5.9 1.0 6.2 0.8 -1.133 4. Overall quality of tests and handouts? 6.0 0.9 5.9 0.8 .403 5. Helpfulness of feedback given to you about your performance? 6.0 0.9 5.7 1.4 .896 6. Degree to which evaluation procedures measured your knowledge and understanding? 5.5 1.0 5.1 1.3 1.204
The results of the comparison of student performance data between the WBI enhanced and traditional versions of the courses are presented in Tables 6 and 7. These tables compare the project grades, final test grades, and final course grades as achievement scores in the form of percentages. A student's raw score on each performance assessment was divided by the total points possible for that assessment to arrive at a percentage score. The means of the percentage scores for each of the four assessments in both the WBI classes and comparable traditional classes were calculated, and the t-test for independent samples was applied to determine if the differences in the means of the two groups were statistically significant.
Student performance on each of the assessments for the philosophy course was less for the WBI version as compared with the traditional version. In two cases the differences were statistically significant: 1) performance on the mission statement project, and 2) final grades.
In contrast, students in the WBI version of the methods course did better on three of the four student performance criteria than students in the traditional version of the course. Students in the WBI version of the methods course did significantly better on the final test. However, students in the WBI methods course did not do as well on the presentation project as students in the traditional course.
Student Assessment in Philosophy and Practice of Career and Technical Education (Traditional: N = 15; WBI: N = 22; df =35; Two-tailed significant at .05 = 2.03) Student Assessment Traditional WBI M SD M SD t-value Project 1: Mission .983 .004 .860 .007 4.779 Project 2: Survey .981 .003 .949 .007 1.261 Final Test .964 .004 .941 .004 1.063 Final Grade .959 .003 .908 .003 2.979
Student Assessment in Instructional Methods for Business and Industry (Traditional: N = 25; WBI: N = 22; df = 45; Two-tailed significant at .05 = 2.02) Student Assessment Traditional WBI M SD M SD t-value Project 1: Learning log .966 .010 .967 .011 -.029 Project 2: Presentation .906 .002 .881 .009 1.172 Final Test .918 .006 .957 .002 -2.003 Final Grade .926 .002 .937 .002 -.856
Conclusions and Implications
The majority of the students perceived that they spent more or an equal amount of time on the WBI courses as compared with past experiences in traditional courses. A majority of the students also indicated that the WBI instruction courses were easier or equal in difficulty to the traditional courses. In other words, students found they needed to spend an equal amount or more time on the WBI courses than on typical traditional courses, but they did not tend to find the WBI courses more difficult. No statistically significant differences were found between the two versions of each of the courses on the student satisfaction measures.
Forty-three percent of the philosophy students would have preferred more classroom-based instruction in the WBI-enhanced course in contrast to 9% of the WBI methods students. Upon examining why so many of the philosophy students might have preferred more traditional instruction, it was surmised that this may be due to the fact that the philosophy students were required to share and discuss their assignments with other students, an activity which would have been more easily done in a classroom.
The student performance results between the WBI versions and the traditional versions of the two courses were mixed. Performance in the WBI philosophy course tended to be worse than in the equivalent traditional course, while student performance in the methods course tended to be comparatively better than in the course's traditional version.
Although some statistically significant differences were found on some of the student performance measures, the practical differences between the WBI and traditional versions of the courses were not large. The courses were graded using a criterion referenced system in which students were not graded against one another but rather against a set of standards. Given this system, it is possible for all students to receive very high grades or very low grades depending upon their performance. This study revealed that in all cases but two, the average performance ratings on each of the four assessments for the WBI and traditional versions of the courses were above 90%, using the same scoring rubrics. The exceptions were that the average performance on the mission assignment for the WBI students in the philosophy course was 86%, and performance on the presentation project in the WBI version of the methods course was 88%. This suggests that although there were some differences in student performance between the WBI and the traditional versions of the courses, students were able to learn effectively within both versions of the courses.
The findings of this study are mixed and the results have limitations. It is virtually impossible to separate from the mix the impact of the instructors on the delivery of each format or to ensure that both versions of each course were exactly the same. It is also not possible to ensure that the students in the WBI and traditional versions of the courses comprised equivalent groups. However, it appears that students were well served by both versions of the courses. In addition, it appears that the general model used to evaluate these courses yielded meaningful results and is feasible to implement. Therefore, this model may provide an effective method of evaluating other similar WBI courses.
Allen, M., Bourhis, J., Burrell, N., & Mabry, E. (2002). Comparing student satisfaction with distance education to traditional classrooms in higher education: a meta-analysis. The American Journal of Distance Education, 16(2), 83-97.
Arnone, M. (2002). Many students' favorite professors shun distance education. Chronicle of Higher Education, 48(35), 39-40.
Clark, R. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21-29.
Collins, B. (2003, June 30-July 2, 2003). Stretching the mold: web applications as a tool for change. Paper presented at the IASTED International Conference on Computers and Advanced Technology in Education, Rhodes, Greece. ISBN: 0-88986-361-X, ISSN 1482-7905 Paper 402-807.
Daugherty, M., & Funke, B. L. (1998). University faculty and student perceptions of web-based instruction. Journal of Distance Education/Revue de l'enseignement a distance, 13(1), 21-39. ISSN: 0830-0445.
Day, T., Raven, M., & Newman, M. (1998). The effects of World Wide Web instruction and traditional instruction and learning styles on achievement and changes in student attitudes in technical writing in an agricommunication course. In Journal of Agricultural Education, 39(4), 65-75.
Furst-Bowe, J. (1997). Comparison of student reactions in traditional and videoconferencing courses in training and development. International Journal of Instructional Media, 24(3), 197-204.
Harasim, L. (2000). Shift happens: online education as a new paradigm in learning. In Internet and Higher Education, 3, 41-61.
IASTED (2003, June 30-July 2, 2003). Proceedings of the IASTED International Conference on Computers and Advanced Technology in Education, Rhodes, Greece. ISBN: 0-88986-361-X, ISSN 1482-7905.
NCCTE (2001). The Status of Career and Technical Teacher Education Teacher Preparation Programs. Minneapolis, MN University of Minnesota, National Research Center for Career and Technical Education. 78pp.
Reeves, T. C., & Reeves, P. M. (1997). Effective dimensions of interactive learning on the World Wide Web. In B. H. Kahn (Ed.) Web-based Instruction (59-67). Englewood Cliffs, N. J.: Educational Technology Publications.
Rovai, A. P., & Barnum, K. T. (2003). Online course effectiveness: an analysis of student interactions and perceptions of learning. Journal of Distance Education/Revue de l'enseignement a distance, 18(1), 57-73.
Sikora, A., & Carrol, D. (2002). A profile of participation in distance education: 1999-2000 (Research Report): National Center for Educational Statistics. Available: http://nces.ed.gov/pubs2003/2003154.pdf.
University's Cyber Classes Spark Faculty Concerns. (1998, July 15). The Wall Street Journal, p. F1.
Pucel is Professor and Stertz is Administrative Fellow in the Department of Work, Community, and Family Education at the University of Minnesota in St. Paul, Minnesota. Pucel can be reached at PUCEL001@umn.edu.