JITE v40n1 - Classroom Action Research: A Case Study Assessing Students' Perceptions and Learning Outcomes of Classroom Teaching Versus On-line Teaching
Classroom Action Research: A Case Study Assessing Students' Perceptions and Learning Outcomes of Classroom Teaching Versus On-line Teaching
Klaus Schmidt
Illinois State UniversityAction research has grown in popularity throughout the past two decades ( Harkavy, Puckett, & Romer, 2000 ; Fleming, 2000 ). It is becoming a more accepted tool for teachers to assess their own teaching strategies and reflect upon their effectiveness. McNiff (1999) defined action research as the name given to an increasingly popular movement in educational research that encourages teachers to be reflective of their own practices in order to enhance the quality of education for themselves and their students. McNiff continued that action research is a form of self-reflective inquiry that can be used in school-based curriculum development, professional development, and school-improvement schemes. Schmuck (1997) extended on teacher self-reflection and stated that "when educators strive to reflect on their past, present, and future actions and engage in solitary dialogue, their perspectives of work mature" (p. 8). McNiff concluded that action research actively involves teachers as participants in their own educational improvement.
Mettetal (2001) provided a seven-step outline to develop a classroom action research project. These steps included statement of the problem, review of literature, research strategy, data gathering, data analysis, taking action, and sharing the findings. The following sections discuss in detail how this author addressed these steps in a case study in which student perception of an on-line classroom environment and a traditional classroom environment were assessed along with the corresponding learning outcomes.
Statement of the Problem
As indicated by Mettetal (2001) , the statement of the problem for a classroom action research project should include a question related to student learning. Incorporating aspects of on-line and traditional classroom teaching could benefit both students and teachers if the learning outcomes are comparable. Little research exists on the evaluation of student perception of on-line versus traditional classroom learning environments and their corresponding learning outcomes, in particular, when the course material was to be delivered simultaneously by the same instructor.
In order to provide a meaningful integration of on-line tools into the traditional classroom environment, two questions were addressed. First, did students obtain the same learning outcomes on-line as they did in a traditional classroom setting? And second, did students perceive their on-line classroom environment to be comparable with a traditional classroom setting? Only when these questions can be answered positively can the incorporation of on-line tools be considered successful.
Review of Literature
The majority of higher education institutions offer courses on-line ( Beller & Or, 1998 ). An increasing number of faculty members across the country include teaching and learning tools provided by the World Wide Web. Ryan, Hodson Carlton, and Ali (1998) spoke of a shift in paradigm in higher education from traditional classroom settings to distance education program delivery via the World Wide Web. They further stated that distance education delivered via on-line technology was also becoming a viable and convenient alternative for students who are "not so distant." Of 609 students enrolled in one distance education program, 500 also were enrolled in traditional courses on campus ( Guernsey, 1998 ).
With the evolution of the World Wide Web, on-line teaching and learning has gained a tremendous amount of popularity. New web teaching and learning tools are created at a fast pace to provide better, more efficient, and easier access to learning communities.
In a typical on-line learning environment, each student is provided with access to a virtual classroom. The instructor posts lecture notes and related literature on the Web and organizes classroom discussions that are completed through Web conferencing. In addition, chat group sessions are held and student presentations are posted to websites. Liu and Thompson (1999) found that faculty members are more likely to use a wider variety of educational technologies when exposed to distance learning. For example, Powers, Davis, and Torrence (1998) enriched their on-line teaching and sense of learning community by expecting students to participate regularly and consistently in class discussions on the Web and by requiring responses from each student to their peers' on-line presentations.
Ryan, Hodson Carlton, and Ali (1998) observed that higher education is moving with deliberate speed toward the electronic classroom and that much has been published on faculty experiences with course delivery through the Web. In spite of the rapid expansion of on-line instruction, little research existed on the evaluation of student perception of on-line versus traditional classroom learning and their corresponding learning outcomes, in particular when on-line learning components are embedded in an otherwise traditional classroom learning environment. Sherry, Fulford, and Zhang (1998,) and Biner, Bink, Huffman, and Dean (1995) added that few evaluation models appear to have been formally assessed or developed in relation to distance education.
Student perception and the quality of on-line programs need to be continuously assessed in order to assure that learning outcomes are increased and do not suffer from using on-line technology. Sherry, Fulford, and Zhang (1998) discussed the positive relationships between students' satisfaction with instruction and their subsequent success in a course. They continued that the importance of efficiently assessing students' perception of their instructional environment is an integral role in student learning outcomes. Cheung (1998) added that student feedback is essential for improving the academic quality of on-line learning and helped provide comparative data across different courses to monitor the consistency of standards. However, Dasher-Alston and Patton (1998) stated that much of the faculty and student apprehension surrounding distance learning stems from uncertainty regarding quality. How can colleges and universities assure the quality of distance learning courses and programs? What safeguards can institutions employ to sustain the integrity of their academic programs and how can this nontraditional delivery system help an institution realize its stated educational goal and objectives? These questions created the need to further study outcomes of on-line learning.
Despite the fact that the literature seemed to agree that overall learning outcomes were similar between on-line and traditional classroom instruction (e.g., Spooner, Jordan, Algozzine & Spooner, 1999 ), quality of on-line learning environments seemed to be under more scrutiny than the quality of traditional classroom environments. Therefore, a deeper understanding of students' perceptions of on-line and traditional classroom learning and their corresponding learning outcomes was necessary to help improve and better facilitate on-line learning and to better integrate it into the classroom. Combining on-line learning with the traditional classroom could help to diversify teaching and learning alike, address a multitude of learning styles, and increase technological literacy of both faculty and students.
Research Strategy
Mettetal (2001) stated that both quantitative and qualitative methods were appropriate to assess the outcomes of a classroom action research project. Three major research designs could be used for classroom action research projects: pretest-posttest designs, comparisons of similar classes, and case studies. A case study was used to compare on-line teaching versus traditional classroom teaching and their corresponding learning outcomes. As is common in case studies, generalizability is left to the reader. It is up to the reader to determine whether or to what extent the findings may apply to a different context.
Subjects
All students enrolled in TEC 151 Introduction to Industrial Computer Systems in fall semester 2001 (N ª 35) were eligible to participate in the classroom action research project. The subjects for the study were rather homogeneous. Over 90% of the students were male; age and ethnicity were not assessed for this study. Students of TEC 151 were utilized because of the introductory nature of the course. Students' backgrounds in this course were more uniform than in higher-level classes offered in the same department. All participating subjects completed an informed consent form that had been reviewed and approved by the institution's human subjects review board, and all procedures for the protection of human subjects were followed. Students who chose not to participate in the study continued to attend the regular classroom sessions. Those students who volunteered to participate (n = 29) were randomly assigned to one of two cohorts, either Cohort A or Cohort B. Random assignment to Cohort A or Cohort B was necessary to prevent students from choosing a preferred method of instruction for a particular content matter.
Protocol
The case study extended over a six-week time period. During the first three weeks, Cohort A studied the first subject matter using an on-line learning method, while Cohort B studied the same subject matter for the same time period using the instructor-lead classroom method. During the second three-week section of the project, roles of the cohorts were reversed: Cohort A studied a new subject matter for three weeks using the traditional classroom method, while Cohort B studied the same subject matter for the same time period using an on-line learning method (Figure 1). The reversal of the groups provided each participating student in the class the opportunity to experience both on-line and off-line teaching and learning.
Figure 1
Design of Study
First three weeks:
Subject matter 1Second three weeks:
Subject matter 2Cohort A
Classroom
environment (15)
On-line environment
(14)Cohort B
On-line environment
(14)Classroom environment
(14)Total N= 29 N = 28
At the end of each three-week block, an instrument was administered to gather data on how students perceived their classroom environment. A test that covered the content of the three-week block was also administered at the end of each three-week block. The results of these tests were used as the basis to assess learning outcomes along with the assignments completed during each three-week block.
Data Gathering
Data-gathering strategies commonly used in classroom action research include the use of test scores, teacher evaluations, final course grades, and other progressive classroom assessment techniques. For this study, three instruments were employed to gather data. Instrument 1 measured student perception of classroom environment. A multiple-choice test was developed and administered at the end of each three-week block to assess learning outcomes for each cohort and served as Instrument 2. In addition to the tests, two exercises completed throughout each three-week time period were used to assess learning outcomes (Instrument 3).
Instrument 1
A questionnaire designed by Ryan, Hodson Carlton, and Ali (1998) was used to evaluate students' perceptions of their on-line and off-line classroom environment. Ryan, Hodson Carlton, and Ali determined the reliability of this instrument using Cronbach's alpha (ρ = .76 for the classroom scale and ρ = .82 for the Web module scale) and a test-retest procedure. The eight items included the following.
- Content covered topic.
- Interaction was evident.
- Participation was facilitated.
- Critical thinking was required.
- Time was appropriate for assignments.
- Faculty preparation and expertise was important.
- Required communication skills.
- Required technical skills.
The responses to the items were measured by a Likert-type scale ranging from 1 = strongly agree to 5 = strongly disagree. All participants were physically present to fill out the questionnaire on the last day of each three-week block. Three additional short-answer items asked the respondents what they liked and disliked about the classroom or on-line learning environments, and how those could be improved.
Instrument 2
In addition to students' perception of on-line and off-line learning, a 20-item multiple choice test was administered at the end of each three-week time period to assess learning outcomes. The tests to be completed were identical for both on-line and off-line learners. All test takers were allowed to use their notes and any literature they identified during the three-week block. Students were also allowed to access any information on-line and the appropriate software package during the time of the tests. Students assigned to the traditional classroom cohort took the quiz in a computer laboratory with the appropriate software packages installed and Internet access, and thus had the same access to information as did the on-line students.
Instrument 3
Two exercises per cohort were assigned during each three-week time period. The exercises to be completed were identical for both on-line and off-line learners. The exercises were problem-solving activities designed to address higher level thinking skills. Results of the tests and projects were the basis to assess learning outcomes and determine if they were statistically significant between on-line learners and off-line learners.
Data Analysis
Mettetal (2001) stated that the researcher should be looking for findings with practical significance when analyzing the data, in addition to statistical significance. She further suggested that simple statistical analyses of quantitative data, such as simple t-tests and correlations, were sufficient.
ANOVA's were used to identify statistically significant differences on the eight Likert-type items and on learning outcomes as measured by the tests and exercises. Qualitative responses provided by the short-answer items on Instrument 1 were analyzed for themes and insights.
To determine if there were statistically significant differences between students' perception of the two learning environments, the eight Likert-type items of Instrument 1 were analyzed with an alpha level of .05. One ANOVA was performed for the first three-week time period as students studied the first subject matter (Table 1). The analysis showed one statistically significant difference at the 0.05 alpha level: the item "Interaction was evident" was rated more favorably by off-line students (1.8 vs. 2.46, alpha: .033).
Table 1
First Subject Matter (A = Off-line, B = On-line)
Cohort A Cohort B Total n = 15 n = 14 N = 29 M SD M SD M SD
Content 1.87 .64 2.00 1.15 1.93 .90 Interaction 1.80 .56 2.46 .96 2.11 .83 Participation 2.00 .75 2.31 1.10 2.14 .93 Critical
thinking1.73
.79
1.38
.76
1.57
.79
Time
appropriate1.73
1.22
1.69
.94
1.71
1.08
Faculty
preparedness1.67
.61
1.46
.66
1.57
.63
Communication 2.20 .94 2.08 .76 2.14 .84 Technical skills 1.80 .67 2.08 .64 1.93 .66
The next ANOVA was conducted for the second three-week block as students studied a new subject matter (Table 2). Two statistically significant items were identified: the item "Content covered topic" was rated more favorably by off-line students (1.79 vs. 2.71, alpha .029), and the item "Communication skills required" was rated more favorably by on-line students (1.64 vs. 2.43, alpha .05).
Table 2
Second Subject Matter (B = Off-line, A = On-line)
Cohort B Cohort A Total n = 14 n = 14 N = 28 M SD M SD M SD
Content 1.79 .89 2.71 1.20 2.25 1.14 Interaction 2.00 1.17 2.50 1.28 2.25 1.23 Participation 2.14 1.16 2.50 1.28 2.32 1.21 Critical
thinking1.71 .91 1.86 1.16 1.79 1.03 Time
appropriate1.50 1.09 2.29 1.26 1.89 1.22 Faculty
preparedness1.79 1.05 1.93 1.07 1.86 1.04 Communication 2.43 1.08 1.64 .92 2.04 1.07 Technical skills 1.71 .825 1.64 1.15 1.68 .98
A third ANOVA was conducted on all on-line learners versus all off-line learners, regardless of the subject matter studied (Table 3). In order to obtain this data, the on-line data collected from Cohort A was combined with the on-line data collected from Cohort B; and the off-line data was combined respectively (Figure 2). This analysis revealed statistically significant differences for the item "Content covered topic." This item was rated more favorably by the off-line cohort (1.83 vs. 2.37, alpha 0.048), and the item "Interaction was evident" was rated more favorably by the off-line cohort (1.90 vs. 2.48, alpha 0.035).
The results from the multiple-choice test instrument indicated a mean score of 16.19 (SD = 5.17) for Cohort A and a mean score of 14.51 (SD = 6.43) for Cohort B. These scores did not reveal any statistically significant differences at a .05 alpha level. Results of the assignments evaluated also indicated a difference (Cohort A = 42.40; Cohort B = 37.71). However, some differences were implied, although not statistically significant; the mean scores for the assignments tended to be lower for on-line students, with a greater standard deviation. Additional data were obtained from Instrument 1 in three short-answer items. Thematic analyses of short answers were used to reflect more systematically on the teaching methods used during the time of the case study.
Table 3
Both Subject Matters Combined
Off-line On-line Total n= 29 n = 28 N = 57 M SD M SD M SD
Content 1.83 .75 2.37 1.21 2.09 1.03 Interaction 1.90 .90 2.48 1.12 2.18 1.04 Participation 2.07 .96 2.41 1.18 2.23 1.07 Critical
thinking1.72
.84
1.63
1.00
1.68
.91
Time
appropriate1.62
1.14
2.00
1.14
1.80
1.15
Faculty
preparedness1.72
.841
1.70
.91
1.71
.86
Communication 2.31 1.004 1.85 .86 2.09 .95 Technical skills 1.76 .739 1.85 .94 1.80 .84
Figure 2
Grouping On-line and Traditional Classroom Learners
On-line Traditional classroom First three
weeksCohort A
Cohort B
Second three
weeksCohort B
Cohort A
Total: N = 28 N = 29
The off-line students identified the following constructs: Many off-line students indicated that they enjoyed the face-to-face interaction with the professor and peers, and stated that it was easy to ask questions in the classroom. Some dislikes expressed by students included that the course material was covered too quickly in the classroom and that there was not enough lab time to complete the hands-on assignments. However, these dislikes were not reflected as statistically significantly different from on-line students as assessed by the item "Time available for assignments." A common suggestion for improving the traditional classroom environment was to allocate more time to laboratory exercises and less time to lecturing.
The most common construct on themes identified by the on-line cohort focused on the freedom students enjoyed in regard to the material to be studied. Students enjoyed working from home at their own pace, and the ability to review lectures as many times as they wanted. A few comments on dislikes were related to problems with the technology itself, such as problems using RealPlayer or slow modems; another concern included the lack of direct interaction with the faculty member and a longer wait to have questions answered that arose during the week. The lack of interaction with the faculty member was also reflected in the item "Interaction was evident", which on-line learners rated less favorably than off-line learners on Instrument 1.
The most common suggestion to improve the on-line experience was to include a time once a week or so in class or in the laboratory when students could directly interact with the faculty, rather than on-line.
Taking Action
How could the findings of the study be used to improve teaching strategies? The learning outcomes as measured by the tests and projects did not result in statistically significant differences between on-line and off-line learners. Based on the results of this study, neither teaching method appeared to be more effective than the other and thus does not lead to an obvious choice. Both teaching strategies seemed to be equally effective. Mettetal (2001) suggested in such a scenario that the teacher may choose the strategy that he or she prefers or the one that students prefer. Preferences are an important factor for a faculty member to decide whether or not to teach a course or certain portions of a course on-line. It will heavily depend on the faculty member's motivation, interest, and technological literacy to advance on-line teaching, in particular, since on-line teaching initially requires more faculty time and resources. Teacher self-reflection will be necessary when incorporating on-line technology into a course. Only then will teachers be successful in addressing diverse student learning styles and including student suggestions into the course development. Student suggestions obtained on the first instrument helped to identify a statistically significant difference for the item "Content covered topic" in favor of the traditional classroom method. This issue can be addressed in a revised version of the on-line learning environment. Although this study used the same presentations in both learning environments, the content will be revisited to further identify factors that might have caused the differences in the responses.
The item "Interaction was evident", which was more favorably rated by traditional classroom students for the first subject matter, will also be addressed in a future revised version of the on-line/off-line learning environments. Although on-line students were required to log on to a chat room (synchronously) twice a week for one hour and to use the asynchronous discussion tool at least twice a week, interaction seemed not to be sufficient for on-line learners to rate the item "Interaction was evident" more favorably. These lower ratings will be addressed by finding new and creative ways to use the chat and discussion tools, or by identifying new Web tools that better address student interaction.
Additional data analyses, research, and follow-up studies are needed to continue to successfully incorporate on-line learning into the classroom. Further research could include how previous exposure to computers and distance learning affects learning outcomes. Follow-up studies could assess how perceptions of on-line and off-line learning change over time as technical literacy increases. Additional research could investigate benefits of on-line teaching and learning for on-campus students. Yet other research may address how learning outcomes vary when students have a choice of their teaching and learning environment.
Sharing the Findings
Mettetal's (2001) last step included the sharing of findings. The case study presented in this paper did not only allow the author to reflect on his own teaching and learning style, but it also had the potential to impact faculty members in their efforts to incorporate on-line technology into their industrial teacher education classrooms. The dialog among colleagues throughout the department and the college, initiated by various presentations on campus, encouraged faculty to reflect on their teaching and learning style, and to enrich their teaching portfolio with on-line teaching and learning tools. The author will continue to use the Web as a teaching tool and further research and design successful web-enhancement models for traditional classroom environments.
Conclusions
The classroom action research project presented in this paper was the first of this kind for the author. In addition to branching into the on-line teaching and learning world, the author learned more about his own teaching style, not only in an on-line environment, but also in the traditional classroom. Based on the information provided by the students, the author can now address specific teaching issues in both the traditional classroom and in the virtual classroom. No statistically significant differences in learning outcomes were identified in this case study, thus indicating that students participating in this project learned as well on-line as they did in the traditional classroom setting. The incorporation of on-line teaching and learning tools in the traditional classroom can be considered successful, particularly in light of the similarity of learning outcomes and classroom perceptions. However, the author will continue to research the benefits of on-line teaching and learning and see if the findings of this study can be further corroborated. More research with larger student numbers should be conducted, including the use of variables such as learning style differences. For example, it will be important to investigate if students with certain learning styles do better in an on-line learning environment. The author will also continue to use the Web as a teaching and learning tool, and will try to identify additional creative ways to combine on-line and traditional classroom teaching and learning. What remains to be said is that on-line teaching and learning technology is manifesting itself in the classroom.
References
Beller, M., & Or, E. (1998) . The crossroads between lifelong learning and information technology: A challenge facing leading universities. Journal of Computer Mediated Communication , 4(2). Retrieved August 1, 2002, from http://www.ascusc.org/jcmc/vol4/issue2/beller.html
Biner, P.M., Bink, M.L., Huffman, M.L., & Dean, R.S. (1995) . Personality characteristics differentiating and predicting the achievement of television-course students and traditional-course students. The American Journal of Distance Education , 9(2), 64-70.
Cheung, D. (1998) . Developing a student evaluation instrument for distance teaching. Distance Education 19(1), 23-41.
Dasher-Alston, R,. & Patton, G. (1998) . Evaluation criteria for distance learning. Planning for Higher Education 27(3), 11-17.
Fleming, D. (2000) . The AEL guide to action research. Charleston, WV: Appalachia Educational Lab.
Guernsey, L. (1998) . Distance education for the not-so-distant. The Chronicle of Higher Education , 44(20), A29-A30.
Harkavy, I., Puckett, J., & Romer, D. (2000) . Action Research: Bridging service and research. Michigan Journal of Community Service Learning (special issue) , 113-118.
Liu, Y. & Thompson, D. (1999) . Teaching the same course via distance and traditional education: A case study . Commerce, TX: Texas A & M University-Commerce.
McNiff, J. (1999) . Action Research: Principles and Practice. London: Routledge.
Mettetal, G. (2001) . The what, why and how of classroom action research. The Journal of Scholarship of Teaching and Learning , 2(1), 6-13.
Powers, S., Davis, M., & Torrence, E. (1998) . Assessing the classroom environment of the virtual classroom. Paper presented at the annual meeting of the Mid-Western Educational Research Association (MWERA), Chicago, IL.
Ryan, M., Hodson Carlton, K., & Ali, Nagia (1998) . Evaluation of traditional classroom teaching methods versus course delivery via the World Wide Web. Journal of Nursing Education , 38(6), 272-277.
Schmuck, R. A. (1997) . Practical Action Research for Change. Arlington Heights, IL: IRI SkyLight Training and Publishing, Inc.
Sherry, A., Fulford, C., & Zhang, S. (1998) . Assessing distance learners' satisfaction with instruction: A quantitative and a qualitative measure. The American Journal of Distance Education , 12(3), 4-25.
Spooner, F., Jordan, L., Algozzine, B., & Spooner, M. (1999) . Student ratings of instruction in distance learning and on-line campus classes. The Journal of Educational Research 92(3), 132-140.
Schmidt is Assistant Professor in the Department of Technology at Illinois Sate University in Normal, Illinois. Schmidt can be reached at kschmid@ilstu.edu .