JOTS v42n2 - Flexible and Job-Embedded Professional Development for In-Service Technology, Design, and Engineering Educators

ABSTRACT

Technology, design, and engineering (TDE) education teachers have less access to
 quality professional development than other Science, Technology, and Mathematics
(STEM) educators. To address this need, the Transforming Teaching through Implementing Inquiry (T2I2) project created an online professional development system for TDE secondary educators. The online professional learning experiences, defined by National Board for Professional Teaching Standards (NBPTS), reinforce and introduce instructional practices that promote student learning. For this study, two groups of teachers, selected from five states (Illinois, Kentucky, Ohio, North Carolina, and Virginia), completed the T2I2 curricular units and submitted artifacts/evidence of practice. Analysis of the artifacts, using the non-parametric Wilcoxon-signed-ranks Test, provides evidence that the teachers within
the pilot studies demonstrated proficient abilities to manage, monitor, and adjust learning environments; contribute to a learning community; and increase their self-assessment following the completion of the curriculum. These results led the authors to suggest further use of the learning platform with in-service teachers in related STEM disciplines that face comparable pedagogical challenges.

INTRODUCTION

The importance of quality teacher learning opportunities cannot be overstated. Teacher quality is consistently noted as a critical factor that impacts student learning ( National Research Council [NRC], 2010 ). Effective professional development that affects teacher quality requires flexible, job-embedded, results-driven learning experiences, which are focused on content that integrates directly into classrooms and builds a community of learners ( Ernst, Segedin, Clark, & DeLuca, 2014 ; Garet, Porter, Desimone, Birman, & Yoon, 2001 ; National Staff Development Council [NSDC], 2001 ; Schlang, 2006 ; Weiss & Pasley, 2006 ). Changes in teacher practice require time, with some states mandating as many as 19 professional development days annually ( Ernst, Clark, DeLuca, & Bottomley, 2013 ; Jacob & McGovern, 2015 ). Such time is well spent when this work results in improving teaching skills and pedagogical content knowledge ( Li, Ernst, & Williams, 2015 ).

National STEM education initiatives call for high quality professional development for STEM educators; however, professional learning experiences for technology, design and engineering (TDE) educators pale in comparison to professional development for other STEM disciplines. Often these are characterized as less comprehensive and perceived to have little value ( DuBois, Farmer, Gomez, Messner, & Silva, 2009 ; Li, Ernst, & Williams, 2015 ; NRC, 2009 ). Professional development for these TDE educators is often found to be inadequate and limited ( National Academy of Engineering, 2009 ).

The lack of technology education National Board for Professional Teaching (NBPT) certified teachers, and an increasing shortage of TDE educators further accentuates the need for quality professional development and an enhanced pipeline for this group of educators (NBPT, personal communication, October 2012). To address the shortage, thirty-nine states (78%) utilize alternative routes, such as career-switcher programs, to licensing TDE educators ( Ndahi
 & Ritz, 2003 ). Targeted professional learning experiences and supported networks are needed to sustain and build teacher practices of newly qualified teachers.

This demonstrated need for professional development that focuses on improving TDE educators' teaching skills and pedagogical content knowledge was the impetus behind
 the development and implementation of the Transforming Teaching through Implementing Inquiry (T2I2) project. T2I2 is an online professional development system for grades
 6-12 TDE educators. The system content
 targets implementation and instructional practice, as defined by NBPTS, in support of quality classroom indicators for the promotion of student learning. T2I2 professional development is research-informed, interactive, and object- oriented, built upon professional learning frameworks developed and refined within prior studies such as Visualization in Technology Education (VisTE) and the Tech-Know Project ( Ernst & Clark, 2007 ; Ernst, Taylor, & Peterson, 2005 ). These frameworks utilize state-of-the-art course content management and collaboration software to provide clear, challenging, connected, and coherent professional learning experiences for educators that encourage critical reflection on practice and self-evaluation through “sustained opportunities over a substantial time interval” ( Mundry, 2007 ; NRC, 2011 ). Utilizing this web-based platform, T2I2 was designed to introduce, reinforce, and develop TDE educators' abilities in regard to the art and practice of teaching.

RESEARCH HYPOTHESES

This study's five investigational hypotheses address teachers in the pilot groups' abilities to manage, monitor, and adjust their learning environments; to develop reflective self-assessment strategies; and to increase contributions to the broader learning community.

Research Hypothesis 1: A teacher's ability to manage learning environments was deemed proficient following the use of the T2I2 professional development materials.

Research Hypothesis 2: A teacher's ability to monitor learning environments was deemed proficient following the use of the T2I2 professional development materials.

Research Hypothesis 3: A teacher's ability to adjust learning environments was deemed proficient following the use of the T2I2 professional development materials.

Research Hypothesis 4: A teacher's ability to contribute to the learning community was deemed proficient following the use of the T2I2 professional development materials.

Research Hypothesis 5: A teacher's ability to increase self-assessment was deemed proficient following the use of the T2I2 professional development materials.

The teachers' skills and abilities were documented through written and video artifacts, similar in design to artifacts developed for NBPT certi cation.

STUDY PARTICIPANTS AND METHODOLOGY

For the first year of the two-year pilot study
 (2012-2013), 190 applicants applied to participate from a five-state (Illinois, Kentucky, Ohio, North Carolina, and Virginia) list-serve recruitment. All candidates were middle or
high school teachers identified as not holding Technology Education NBPT certification.
From the applicant pool, eight middle school and eight high school teachers were randomly selected to participate in the first year of the
pilot study. For the purposes of this research, these sixteen teachers agreed to: (a) complete
 17 Learning Objects within the T2I2 curriculum and (b) submit artifacts/evidence of practice, upon the completion of this work. The 17 Learning Objects are clustered into the following four units: Demonstration Lesson, Fostering Teamwork, Assessment of Student Learning, and Documented Accomplishments. These units were based upon NBPTS’ expectations. Learning Objects are modular lessons that contain materials and information created by
a team of TDE NBPT-certified teachers,
 TDE teacher educators, and in-service veteran TDE K-12 educators. Learning Objects
provide a research-informed basis for each topic through the “Impact on Learning” sections, a step-by-step implementation approach through the “Procedures in the Classroom” sections,
 and specific methods to identify if the process has been successfully implemented through the “Determine Success” sections. As teachers finish each Learning Object they complete a five-question post assessment quiz to check
for understanding. Upon the completion of all Learning Objects within a unit, pilot teachers submitted written and/or video artifacts as evidence to document their abilities to implement newly learned practices. The post-assessment quizzes offered formative assessment to the research team. The assessment of the artifacts addressed the research hypotheses.

Teachers for the second year of the pilot study (2013 - 2014) were, once again, selected from the five project states. An additional sixteen pilot study teachers, eight middle school and eight high school, were randomly selected from 141 applicants. Teachers within this second pilot group agreed to complete the same tasks identified for the original group. For both pilot groups, teachers were introduced to the T2I2 website, resources, and project expectations through an introductory webinar run in early September of each academic year. Following the webinar, teachers were offered support from the T2I2 team through monthly email contacts and Skype of office hours. Work for each pilot group was targeted to be completed by March of each year.

Quantitative research methods were employed
to form the basis of this research using data
from both pilot groups. Data collected
includes the mean for each Learning Object’s post-assessment and average number of attempts. Data addressing the five research hypotheses was derived from teacher artifacts, four written commentaries and two video commentaries, scored by an NBPT-certified teacher using
an adapted rubric and four-point scoring system. Researchers used non-parametric statistical analysis to determine a teacher’s ability to manage, monitor, and adjust the learning environment in his/her classroom; contribute to a learning community; and increase self-assessment.

This study was initially proposed as a treatment and control study. However, after negotiation with the sponsoring entity, it was determined that the project would be better poised to increase the treatment group to broaden impact. Based upon this guidance, a directional study was planned to examine teacher proficiency.

INSTRUMENTATION

The pilot teacher outcome data, in the form of teacher artifacts, were measured by NBPTS criterion-referenced metrics, targeting the teachers’ abilities to manage, monitor, and adjust a learning environment to improve instruction; conduct self- assessment; and contribute to a learning community. The criterion-referenced metrics were organized around four entries where project Learning Object alignment has been achieved. The Learning Objects, grouped into units, are lessons that introduce and apply specific content, practices and pedagogy for participating teachers. A unit is a logical grouping of several individual Learning Objects. The pilot teachers were expected to complete all units, but, within the T2I2 system, the units do not have to be completed sequentially.

The scoring instances (n) varied depending upon the teacher artifacts submitted and determined to be complete by the project evaluation team. The research hypotheses, related units and Learning Objects, and NBPTS artifacts are found in Table 1. The first and fourth research hypotheses are addressed through evidence acquired from the written commentary and video artifacts submitted following completion of Learning Objects within the Demonstration Lesson unit. These Learning Objects introduce the following topics: Designing Standards-Based STEM, Lab and Class Management, and STEM Curricula. The second research hypothesis is addressed through evidence found within the written commentary and video artifacts following completion of the Fostering Teamwork unit that includes Learning Objects that introduce: Best Practices; Classroom Quality, Enhancing Classroom Creativity, Implementing Learning Activities Multiculturalism in the Classroom, and Working with Special Populations. Research hypothesis three is addressed following the teachers’ submission of the written commentary after completing the Assessment of Student Learning unit that contains Learning Objects focusing on Action Research, Adapting Instruction, Data Analysis, Formative Evaluation Techniques, and Initial Student Evaluation. The final research hypothesis was addressed by analyzing evidence submitted by teachers in the form of a description and analysis, following the teachers’ completion of the Documented Accomplishments unit that contains the Professional Organizations, School and Community, and Student Organizations Learning Objects.

Table 1: T2I2 teacher artifacts aligned with hypotheses, units, and learning objects

Hypotheses	Unit and Learning Objects	NBPTS Artifacts
Research Hypothesis 1: H0 - A teacher’s ability to manage learning environments was deemed proficient following the use of the T2I2 professional development materials.	Demonstration Lesson: Designing Standards Based STEM; Lab and Class Management; STEM Curricula	Entry 2.1: Video Capture
Research Hypothesis 2: H0 - A teacher’s ability to monitor learning environments was deemed proficient following the use of the T2I2 professional development materials.	Fostering Teamwork: Best Practices; Classroom Quality; Enhancing Classroom Creativity; Implementing Learning Activities; Multiculturalism in the Classroom; Working with Special Populations	Entry 3.1: Video Capture Entry 3.3: Written Commentary
Research Hypothesis 3: H0 - A teacher’s ability to adjust learning environments was deemed proficient following the use of the T2I2 professional development materials.	Assessment of Student Learning; Action Research; Adapting Instruction; Data Analysis; Formative Evaluation Techniques; Initial Student Evaluation	Entry 1.4: Written Commentary
Research Hypothesis 4: H0 - A teacher’s ability to contribute to the learning community was deemed proficient following the use of the T2I2 professional development materials.	Demonstration Lesson: Designing Standards Based STEM; Lab and Class Management; STEM Curricula	Entry 2.3: Written Commentary
Research Hypothesis 5: H0 - A teacher’s ability to increase self-assessment was deemed proficient following the use of the T2I2 professional development	Documented Accomplishments: Professional Organizations; School and Community; Student Organizations	Entry 4.1: Description and Analysis

An NBPT-certified assessor reviewed all of the submitted artifacts using an adapted four-point rubric ranging from (4) performance provides clear, consistent, and convincing evidence to (1) performance provides little or no evidence. The NBPTS metrics identifies teacher proficiency as (3) performance provides clear evidence. Teachers were provided written feedback from this review. Proficiency (3) was the level of performance identified within each directional research hypothesis.

Additional information and insight into the teachers’ impressions and views about the project was gathered through interviews with the participating teachers. Teachers were emailed to schedule a brief phone interview. Interviews were conducted with select pilot teachers – both teachers who had completed all Learning Objects and units, and those who had not. While not all teachers had joined the project with the intention of becoming Nationally Board Certified, all teachers interviewed reported clear alignment of the learning objectives with NBPT requirements and found this to be an attractive characteristic of the project. Another positive aspect of participating in the project, noted by interviewed teachers, was access to the comprehensive resources provided through the project website. Teachers reported using these resources in their classrooms throughout the year.

DATA AND ANALYSIS OF FINDINGS

Data was analyzed utilizing quantitative research methods. The two years of pilot data was collected from the assessment of the teacher artifacts and analyzed as a test of hypothetical value conducted using the non-parametric Wilcoxon-signed-ranks Test. The five research hypotheses were tested to determine the teachers’ abilities to monitor, manage, and adjust learning environments; contribute to learning communities; and increase self-assessment.
The specified parameter for this study was a median ≥ 3 with 3 indicating a proficiency level as described and determined by NBPTS. The results of the data analysis for each of the five research questions are displayed in Table 2.

Table 2: Research hypothesis examination using the Wilcoxon-signed-rank test

Research Hypothesis	n = scoring instance possible	n for test	Median Est.	Wilcoxon Stat.	p-value	Method
RH1	33	18	3.5	126	0.9476	Normal Approximation
RH2	33	24	3	88	0.9444	Normal Approximation
RH3	39	32	3	279	0.2377	Normal Approximation
RH4	37	26	3	67.5	0.9982	Normal Approximation

The Wilcoxon-signed-ranks Test was compared to the associated critical value based on the sample size of the participants. The participant data for the sample size was less than 50, denoting that no normal approximation with
the continuity correction was necessary and the reported p-value is exact. The critical alpha value was set at 0.05 for this investigation ( Noymer, 2008 ). The calculated p-values for the tests were determined to be larger than 0.05. The number of instances vary dependent on the number of constructs within each outcome variable.

All five research hypotheses were directional hypotheses described by the notation H1:
 θ ≥ 3. Analysis of the pilot data resulted
in the researchers failing to reject each positive directional hypothesis and suggests that participation in the T2I2 professional development sequence supports
the educator’s ability to monitor, manage,
and adjust the learning environment; contribute to the learning community; and increase the teacher’s self-assessment.

Although outside of the investigational hypotheses, teacher use and access data was also collected and analyzed as formative assessment and used for refinement of the Learning Objects within the four units. Teacher user data, seen
in Table 3, included assessment scores and teacher trials. Data were collected using analytics features of the T2I2 professional development system online architecture. End-of-unit quizzes were offered as teacher participant “self-checks” to identify areas of developing competency. Each quiz could be taken as many times as the teacher participant desired.

Table 3: T2I2 professional development system teacher user data

Teacher User Data
Units	Mean Quiz Scores	Average Number of Attempts
Assessment of Student Learning	94.50	4.50
Demonstration Lesson	100.00	3.83
Fostering Teamwork	98.46	3.15
Documented Accomplishments	97.78	3.22

Teacher access data focused on total unit view, average unique unit views per day, and average time spent on the unit. Teacher access data were also collected using analytics features of the T2I2 professional development system online architecture. This enabled the materials development team to supplementally identify potential problem areas or specific information that was presented in a complex or inefficient fashion, warranting recurrent access or elevated duration. This data for the pilot is seen in Table 4.

Table 4: T2I2 professional development system teacher access data

Teacher User Data
Units	Total Unit Views	Average Unique Unit Views per Days	Average Time Spent on Unit (seconds)
Assessment of Student Learning	1001	3.65	203.4
Demonstration Lesson	395	1.59	170.2
Fostering Teamwork	376	2.00	359.4

The summer following the second pilot
study was spent revising many aspects of
the curriculum, from the number of pilot teachers to the content of the Learning Objects. Concentrated efforts modified Learning Objects within two of the four units: Assessment of Student Learning and Documented Accomplishments. These two units were the basis of the Field Study that was conducted during the 2015-2016 academic year.

IMPLICATIONS

Data analysis indicates that the sample population of teachers who completed T2I2 professional development was supported in their ability to manage, monitor, and adjust learning environments. The pilot group also increased its ability for self-assessment and its contributions to the learning community. The anticipated end product of this initiative is an evidence-informed system that broadens TDE teachers’ instructional abilities.

Framing the coursework following coherent
and national standards-based topics purposefully produced units and Learning Objects appropriate for the broader STEM in-service teacher population. Mean quiz scores greater than 94% suggest teacher competency following the completion of the Learning Objects. Total unit views ranging from 200 to 1000 demonstrate the frequency of use and entry into the system, suggesting teacher diligence in attending to the completion of this professional development.

From this study, the research team has evidence that job-embedded and flexible professional development supports the needs of in-service teachers in TDE education, and may meet the needs of teachers in other STEM disciplines. Teachers within the sample demonstrated that asynchronous learning promoted self-reflection resulting in more robust analysis of their practice.

The development of the T2I2 platform provided a venue for easy delivery of professional development content reinforced through networking and collaboration. Digital tools and platforms, like the one developed for this project, allowed for continuous customization, real- time access, and delivery to select and targeted populations ( Zepeda, 2015 ). Teachers’ classroom and professional practices were reinforced
by leveraging the granular and repositionable teacher learning cyber infrastructure.

The first pilot year of the T2I2 project
yielded changes and improvements for the subsequent pilot year. The various data collected show connections between the implementation of T2I2 and positive teacher classroom practices, though the low number of teacher participants does not allow results to be generalized to wider populations.

CONCLUSIONS

Based upon this study, the authors recommend further development of this flexible professional development platform to not only address the busy schedules of in-service TDE teachers, but also to provide professional learning experiences for in-service teachers in related STEM disciplines. There are stark similarities to professional learning needs between technology and science education. Science educators face comparable pedagogical challenges and could benefit from similar professional development opportunities ( Bybee, 2001 ). Given these similarities, this model and infrastructure provides a venue and platform that could serve as a tool for STEM educators to interact with each other, focusing on topics with common objectives. This would result in a more holistic educational experience for students, clearly following the course set by the Next Generation Science Standards.

The T2I2 platform and units created a robust network of TDE teachers. The next step for
this networking may bring participating teachers’ students together for cross-state collaboration, offering another opportunity to implement key educational outcomes developed within the Learning Objects.

The authors recommend continued teacher needs’ assessments to identify additional topics for inclusion within the T2I2 units and Learning Objects. TDE educators come to the field with
a variety of prior experiences that shape their learning needs pertaining to content and practice. The authors also recognize this diversity and suggest tailoring future T2I2 units and Learning Objects to meet these varied needs.

The current study focused on the TDE teachers’ acquisition of the learning inherent within
the T2I2 curriculum, considering in-progress data collection gauging: (a) how teachers use knowledge of their students to design assessments; (b) how assessment relates to course learning goals; (c) how problem-solving can be incorporated into assessment design; (d) how instructional development further fosters teamwork of students while establishing a safe and encouraging learning environment; and (e) and participation in professional activities and individual accomplishment. Further study could advance the teachers’ implementation of acquired learning.

Note: This material is based upon work supported by the National Science Foundation under Grant No. 1156629.

Jeremy V. Ernst is an Associate Professor of Integrative STEM Education in the School of Education at Virginia Polytechnic Institute and State University, Blacksburg. He is a member of the Gamma Tau Chapter of Epsilon PI Tau.

Aaron C. Clark is a Professor and Director of Graduate Programs for the Department of STEM Education within the College of Education at North Carolina State University. He is a member of the Alpha Pi Chapter of Epsilon Pi Tau.

Sharon W. Bowers is a Senior STEM education specialist for the Center for Integrative STEM Education (CISE) at the National Institute of Aerospace (NIA). She is a PAEMST award winning educator.

REFERENCES

Bybee, R. W. (2001). Effective professional development for technology teachers. The Technology Teacher , 61(3), 26-29.

DuBois, B., Farmer, C., Gomez, L., Messner, R., & Silva, E. (2009). Increasing STEM capacity through “Learn-by-doing” pedagogy . In Proceedings: 2009 Educational Leadership STEM Symposium, 2. California Polytechnic State University: San Luis Obispo, CA.

Ernst, J. V., & Clark, A. C. (2007). Scientific and technical visualization in technology education. The Technology Teacher, 66(8), 16-20.

Ernst, J. V., Clark, A. C., DeLuca, V. W., & Bottomley, L. J. (2013). Professional development Proceedings of the American Society for Engineering Education Annual Conference and Exposition, Atlanta, GA, Session #6184.

Ernst, J. V., Segedin, L., Clark, A. C., & DeLuca, V. W. (2014). Technology, engineering, and design educator professional development system implementation: Initial pilot results . Published Proceedings of the American Society for Engineering Education Annual Conference and Exposition, Indianapolis, IN, Session #T247.

Ernst, J. V., Taylor, J. S., & Peterson, R. E. (2005). Tech-know: Integrating engaging activities through standards-based learning. The Technology Teacher , 65(2), 15-18.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective: Results from a national sample of teachers. American Educational Research Journal , 38(4), 915-945.

Jacob, A., & McGovern, K. (2015). The mirage: Confronting the hard truth about our quest for teacher development . New York, NY: Foundation Center.

Li, S., Ernst, J. V., & Williams, T.O. (2015). Supporting students with disabilities and limited English pro ciency: STEM educator professional development participation and satisfaction. International Journal of STEM Education , 2(20), 1-10.

Mundry, S. (2007). Professional development in science education: What works? Retrieved from http://www.conferences.ilstu.edu/NSA/papers/Mundry.pdf

National Academy of Engineering [NAE] & National Research Council [NRC] of the National Academies. (2009). Engineering in K-12 Education: Understanding the status and improving the prospects . Washington, DC: The National Academies Press.

National Research Council [NRC] (U.S.). Committee on a Conceptual Framework for New K-12 Science Education Standards. (2009, 2011, 2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas . Washington, D.C: The National Academies Press.

National Research Council. [NRC] (2010). Preparing teachers: Building evidence for sound policy . Washington, DC: The National Academies Press.

National Staff Development Council. [NSDC] (2001). NSDC standards for staff development. Retrieved February 2, 2004, from http://www.nsdc.org/standards/index.cfm.

Ndahi, H. B., & Ritz, J. M. (2003). Technology education teacher demand, 2002-2005. The Technology Teacher , 62(7), 27. Retrieved from http://login.ezproxy.lib.vt.edu/login?url=http://search.proquest.com.ezproxy.lib.vt.edu/docview/235301887?accountid=14826

Noymer, A. (2008). Alpha, Signi cance Level of Test. In Paul J. Lavrakas (Ed.), Encyclopedia of survey research methods . (p. 19). Thousand Oaks, CA: Sage Publications. doi: http://dx.doi.org/10.4135/9781412963947.n13

Schlang, J. A. (2006). Professional development that works: Impacting elementary science teachers learning and practice during the implementation of an inquiry-oriented science curriculum (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 3231081)

Weiss, I. R., & Pasley, J. D. (2006). University of Pennsylvania, Graduate School of Education. Scaling up instructional improvement through teacher professional development: insights from the local systemic change initiative. Philadelphia, PA: Graduate School of Education.

Zepeda, S.J. (2015). Job-embedded professional development . New York, NY: Routledge.