JITE v35n3 - Attitudes and Motivations of Vocational Teachers Regarding Program Improvement

Attitudes and Motivations of Vocational Teachers Regarding Program Improvement

James P. Greenan, Purdue University	Mingchang Wu, Purdue University
Ramlee B. Mustapha, Purdue University	Lisa B. Ncube Purdue University

The effectiveness and efficiency of vocational education programs have been a national concern over the last two decades ( Brown & Davison, 1991 ; Greenan, 1986 ; Nuebert, 1990 ; William T. Grant Foundation, 1988 ; Womble, Jones, & Ruff, 1995 ). Several reform initiatives at the federal and state levels have been designed to improve vocational education. Concern over its effectiveness and the quality of its graduates are evident in federal and state legislation. Among the primary intents of the Carl D. Perkins Vocational and Applied Technology Education Act (PL. 98-524) is the need to assist states in expanding, improving, modernizing, and developing quality vocational education programs in order to meet the needs of the nation’s existing and future workforce in the areas of marketable skills, improved productivity, and economic growth ( United States Statutes, 1984 ). Similarly, at the state level, a major focus has been to improve vocational education through adequate and appropriate state leadership, as well as administrative and program support services.

Despite a growing emphasis on improving vocational programs, minimal research exists with respect to the attitudes of vocational educators about program improvement. Attitude has been defined as a judgment of an object or event that prompts individuals to structure their complex social environments. which is to prepare persons for employment while assuming numerous initiatives.

The literature also shows that teacher motivation is a crucial factor in achieving educational goals. Motivated teachers display interest in program improvement activities, feel self-efficacious, expand efforts to succeed, persist at tasks, and use innovative strategies to accomplish educational goals ( Pintrich & Schunk, 1996 ). In addition to intrinsic motivation, external factors (e.g., environment) also play a vital role in creating positive change. The influence of parents, the local community, business, and industry on the school environment is significant. Parents have a powerful influence on their children’s learning and general school adjustment ( Ames, De Stefano, Watkins, & Sheldon, 1995 ).

In terms of community support, parents, the business community, and the nation as a whole are concerned over the quality and contribution of vocational education to society ( Brown, 1984 ). Parents’ perceptions of the responsiveness of schools in addressing vocational education purposes are crucial. Input and commitment of people in business and industry are important to students’ preparation for future employment.

Purpose of the Study

The purpose of this study was to identify the attitudes and motivations of vocational educators concerning the improvement of secondary vocational programs. In this study, program improvement was operationally defined as any process or activity designed to expand or improve vocational programs. The research questions were:

1. To what extent is program improvement important to vocational educators?
2. What are the motivations of vocational educators with respect to program improvement?
3. What are the factors that influence vocational educators’ perceptions of self-efficacy regarding program improvement?
4. To what extent do external supports influence vocational educators’ perceptions with respect to program improvement?
5. To what degree do vocational educators perceive that their professional training and development are beneficial for program improvement?
6. How do vocational educators perceive current vocational education reform issues in regard to program improvement?

Significance of the Study

This study was conducted to investigate current education issues from the perspective of vocational educators. It was designed to provide a base for making suggestions and recommendations for improving the effectiveness, efficiency, and quality of vocational programs and personnel. The results of this study could also assist in program development and improvement, state and district policy revision, and legislative updating.

Methodology

Population and Sample

The population included secondary vocational educators in the state of Indiana as identified from a directory of secondary vocational educators obtained from the Indiana Commission on Vocational and Technical Education (N=2,641). A random sample of 250 subjects was selected and was representative of the population’s gender, program area (agriculture, business and office, family and consumer sciences, health occupations, marketing, technical, technology, trade and industry), present position, type of instruction, highest degree, type of license, teaching experience, and administrative experience.

Instrumentation

The Program Improvement Survey was developed to measure the attitudes and motivations of secondary vocational educators regarding program improvement. The instrument’s items, format, and procedures were derived and constructed based on the existing research studies and literature related to educational reform, effective schools research, teacher preparation and professional development, curriculum and instructional delivery, and teacher attitudes with respect to vocational program development and improvement.

The first section of the survey contained a purpose statement, directions, and was designed to collect demographic information that included gender, primary program area, present position, type of institution, highest degree, type of license, years of teaching experience, and years of administrative experience.

The second section consisted of directions and 30, five-point Likert-scale items (i.e., Strongly Agree [5], Agree [4], Uncertain [3], Disagree [2], and Strongly Disagree [1]). The items were developed and validated based on the six research questions that were posited for the study. These items were clustered into the following categories (1) Importance to Teachers, (2) Teacher Motivation, (3) Factors of Self-Efficacy, (4) External Supports, (5) Professional Training, and (6) Vocational Education Reform.

Item rating point values for items numbered 3, 8, 13, 18, 23, and 28 were reversed because they were stated negatively. A high rating for these items indicated disagreement with a negative statement while a low rating indicated agreement with a negative statement. In addition, four open-ended items followed the Likert items to collect respondent suggestions and recommendations. Drafts of the instrument were reviewed by a panel of experts (i.e., teachers and administrators) in the field of vocational and technical education. Revisions were made based on their comments and recommendations.

The instrument was also pilot-tested on a small group (n = 10) ofvocational educators. An internal consistency reliability estimate was calculated using Cronbach’s Coefficient Alpha (a = 0.82). The items and procedures were subsequently revised using the quantitative and qualitative data that had been collected. Based on these data, the instrument was considered to have achieved an adequate and acceptable degree of content and face validity and internal consistency reliability.

Data Collection

The instrument and a cover letter were mailed to the 250 subjects. A cover letter explained the importance and purpose of the study and requested their assistance and cooperation. Three follow-up mailings were conducted at three-, six-, and nine-week intervals after the initial mailing. A total of 186 instruments were returned, which constituted a final response rate of 74.4%.

Data Analysis

The data were coded and analyzed using Statistical Analysis Software (SAS). Descriptive statistics, including frequencies, means, and standard deviations, were used to analyze the data in order to answer the research questions. The open-ended items were qualitatively analyzed and grouped into emerging categories ( Miles & Huberman, 1994 ) by four individuals. Descriptive codes or labels were used to assign units of meaning and simple frequencies, and rank-orders were used to describe the data. A reliability check was conducted using inter-rater reliability procedures through re-analysis and comparison of the data-sets by the same four individuals.

Results

The major focus of this study was to identify and assess the attitudes and motivations of vocational educators regarding the improvement of secondary vocational programs. The data were organized and analyzed around the study’s research questions.

The respondents were 58.6% female and 41.4% male. The majority (30%) of the respondents were in the Family and Consumer Sciences program area and over 90% of them were teachers. Most of the teachers were in comprehensive high schools and approximately two-thirds (68%) possessed a Master’s Degree. More than one-half (54%) of the respondents had obtained a professional license. Thirty-four percent had taught school for more than twenty years while approximately 5% had less than two years teaching experience. Further, a large majority of the respondents (73%) had zero to two years of administrative experience. The presentation of findings will be organized and prescribed according to the study’s research questions.

To what extent is program improvement important to vocational educators?

Not surprisingly, the respondents strongly agreed ( M = 4.76, SD = 0.44) that program improvement in vocational programs is important. This was also the case for gender, primary program area, present position, type of institution, highest degree, type of license, teaching experience, and administrative experience. For each level of the variables, the means were greater than or equal to 4.33 with standard deviations less than 0.62. This indicates that vocational educators support current program improvement initiatives related to educational reform. (This question was Item 1 on the instrument.)

What are the motivations of vocational educators with respect to program improvement?

The data revealed ( M = 3.77, SD = 0.51) some uncertainty about whether legislation, professional growth, individual student needs, community workforce needs, personal recognition and technology (Items 2-7) collectively, are reinforcers for vocational educators to engage in program improvement. However, when examining the individual items (see Table 1 ), professional growth ( M = 4.14, SD = 0.91), individual student needs ( M = 4.21, SD = 0.72), community workforce needs ( M = 4.05, SD = 0.73), and technology ( M = 4.10, SD =0.72) appeared to be the major motivational factors. However, the respondents were somewhat uncertain regarding whether legislation ( M = 2.98, SD = 1.03) or personal recognition ( M = 3.07, SD = 1.05) contributed to their motivation with regard to program improvement. The data were fairly homogeneous across the demographic variables with the exception of those with no administrative component, whose mean score ( M = 3.35) was considerably lower than those with administrative experience.

Table 1
Factors that Motivate Vocational Educators to Improve Vocational Programs
Item		n	M	SD
2.	Legislation encourages program improvement.	183	2.98	1.03
3.	Program improvement does not enhance professional growth.	185	4.14	0.91
4.	Individual student needs are served through program improvement.	186	4.21	0.72
5.	Program improvement addresses community workforce needs.	185	4.05	.073
6.	I receive recognition through program improvement activities.	185	3.07	1.05
7.	Program improvement addresses technological changes.	186	4.10	0.72

What are the factors that influence vocational educators’ perceptions of self-efficacy regarding program improvement?

The data ( M = 3.53, SD = 0.54) indicate general agreement among the respondents that their teacher training, self-confidence, and allocation of time were moderate contributing factors to vocational educators’ self-efficacy in implementing program improvement activities. They were uncertain ( M =3.36, SD =1.01), about whether their teacher training had prepared them to implement program improvement activities (see Table 2 ). There was strong support for the view that a supportive environment ( M =4.48, SD = 0.71) is an important contributor to self-efficacy. At the other extreme, they indicated that their work load prevents them from engaging in program improvement. These views were fairly consistent across the demographic variable categories.

Table 2
Factors Influencing Vocational Educators’ Perceptions of Self-efficacy
Item		n	M	SD
8.	A supportive enviroment is not needed for program improvement.	186	4.48	0.71
9.	Teacher training has provided me with adequate knowledge and skills to initiate program improvement.	186	3.36	1.01
10.	I am confident to carry out program improvement activities.	186	4.01	0.67
11.	My workload allows adequate time for me to plan program improvement activities.	186	2.27	1.12

To what extent do external supports influence vocational educators’ perceptions with respect to program improvement?

There was general agreement ( M = 3.28, SD = 0.54) within demographic categories (i.e., gender, primary program area, present position, type of institution, highest degree, type of license, teaching experience, and administrative experience) that external supports (i.e., school administrator, school resources, school evaluation, community and school organization) serve as moderate factors influencing teacher perceptions with respect to program improvement. The factor that was viewed as having the strongest impact was parental input ( M = 4.05). Overall, the data identify all of the resources, bureaucratic inefficiencies, and unpredictability as sources of concern related to program support. Parental support was viewed as an important influence of vocational educators’ perspectives (see Table 3 ).

Table 3
External Supports that Influence Vocational Educators' Perceptions with Respect to Program Improvement
Item		n	M	SD
12.	My school provides adequte resources (e.g., material, personnel) to facilitate program improvement.	186	2.82	1.12
13.	Administrators are not efficient in reducing unnecessary bureaucracy.	186	2.52	1.02
14.	Administrators encourage input from staff members in matters concerning school policies and programs.	186	3.42	1.04
15.	My school effectively disseminates and shares information, knowledge, or skills about program improvement.	186	3.10	1.06
16.	Changes in program policies and practices in my school are predictable.	186	2.98	0.96
17.	My school conducts ongoing program evaluation.	184	3.40	0.99
18.	Input from parents is not crucial for program improvement.	186	4.05	0.84
19.	Parents are supportive of program improvement activities.	186	3.55	0.77
20.	Community and school organizations are supportive of program improvement.	186	3.67	0.75

To what degree do vocational educators perceive that their professional training and development are beneficial for program improvement?

In general, the respondents tended to agree ( M = 3.88, SD = 0.47) that their professional training and development had been beneficial in helping them engage in program improvement. Professional training includes in-service workshops, teamwork, input from advisory committees, and networking. They were less certain ( M = 3.53, SD = 0.98) about the adequacy of these professional networks to provide the support necessary to engage in program improvement (see Table 4 ).

Table 4
Vocational Educators' Perceptions about the Benefits of their Professional Development to Support Program Improvement
Item		n	M	SD
21.	I have adequate opportunities for in-service and professional development.	186	3.42	1.01
22.	My in-service training and professional development is beneficial to program improvement.	186	4.00	0.78
23.	I do not work well with my colleagues in program improvement activities.	186	4.16	0.72
24.	Input  from advisory committees is crucial for program improvement.	186	4.09	0.80
25.	I have an adequate professional network to support program improvement activities.	186	3.53	0.98
26.	Networking with professional associations is important to improve my program.	186	4.08	0.65

How do vocational educators perceive current vocational education reform issues in regard to program improvement?

There was broad agreement ( M = 3.96, SD = 0.46) in all the demographic areas that reform issues (e.g., focusing on student learning styles, teaching critical thinking, and school-business partnerships) are important factors in program improvement activities. However, the respondents were uncertain (M = 2.87, SD = 1.13) about the effectiveness of their schools in facilitating collaboration between vocational and academic programs (see Table 5).

Open-ended Items

The final section of the instrument focused on identifying suggestions and recommendations of vocational educators related to program improvement. Four program improvement factors were addressed in this section, including planning factors, program improvement barriers, program improvement skills and knowledge, and other program improvement problems or issues.

The respondents were asked to identify the most important factors to be considered when planning program improvement activities. Eight major program improvement planning activity categories were identified. The most important factors to be considered when planning program improvement activities were: (a) identifying students’ needs; (b) enhancing support and collaboration; (c) identifying workforce and community needs; (d) clarifying the program’s rationale, goals, and objectives; (e) identifying funding and resources; and (f) enhancing teacher competency.

The most frequent response dealt with the factor, focusing on the identification and understanding of student needs such as abilities, interests, characteristics, and backgrounds. The second most frequently identified factor was support and collaboration, which focused on dimensions such as administrative support, state level support, input from advisory committees, communication with the public, and integration of academic and vocational curricula. Also identified were workforce and community needs, such as business trends and changes within the community. Some additional factors included the perceived relevance of program improvement, curriculum and leadership levels, enrollment patterns, and evaluation concerns (see Table 6 ). lack of motivation, lack of support, and cooperation and interest from colleagues and staff were specified. Lack of administrative and community support was the fourth most frequently identified barrier, and included administrative apathy, lack of parental support, and lack of external advice.

Table 6

Table 6
Influential Factors in Program Improvement Planning
Factors	f
Student Needs	71
Support and Collaboration	56
Workforce and Community Needs	48
Rationale, Goals, and Objectives	34
Funding	23
Resources	18
Pre-service and In-service Training	16
Time	15
Other	22

Barriers less frequently identified included inadequate resources, bureaucracy, and lack of communication, guidance and direction. Some additional barriers were reported including declining student enrollments, lack of in-service training, program relevance, commitment to program by personnel involvement, contractual constraints, rural settings, and lack of interest and ability of students (see Table 7 ).

The focus of the instrument was then directed toward identifying the important skills and/or knowledge that are required to improve programs. Content and technical knowledge were cited as the most important program improvement needs for faculty and staff. This includes items such as mastery of subject matter, technical competence in the field, remaining current with technology, and taking a proactive approach toward technological change. The second most frequently identified need had to do with better understanding the needs of the workforce. This knowledge includes an awareness of the needs of business and industry, an ability to link school curricula to the world of work, and an understanding of current job market trends. Public relations and interpersonal skills were perceived to be the third most important area of skills needed. These include the ability to communicate clearly with others, the skills to organize and coordinate diverse groups with different agendas, and an ability to share ideas, goals, and plans with others. The fourth category of skills focused on leadership and management aptitudes (e.g., organizing and networking, changing and adapting to the environment, and creatively solving problems.)

Table 7
Barriers to Program Improvement
Barriers	f
Inadeqate funding	73
Time constraints	63
Negative teacher attitudes	47
Lack of administration and community support	26
Inadequate resources (equipment, supplies, facilities)	22
Bureaucracy	22
Lack of communication, guidance and direction	13
Other	30

Knowledge of student needs including items such as learning styles, future needs of students, and realistic profiles of students was important, but less frequently identified. Other knowledge and skills mentioned were knowledge of legislative mandates, grant writing skills, and knowledge of successful or "exemplar" programs (see Table 8 ).

Table 8
Important Skills and/or Knowledge in Program Improvement
Skills and Knowledge	f
Content and technical knowledge	43
Knowledge of workforce needs	31
Communication, interpersonal, and public relations skills	25
Leadership and management skills	19
Knowledge of student needs	16
Other skills and knowledge	71

The final item on the survey asked respondents to identify any additional problems or issues related to program improvement that had not been addressed in the survey. Four items most identified included low student enrollment, lack of career guidance, students’ lack of skills and knowledge, and lack of good assessment practices (see Table 9 ).

Table 9
Additional Problems and Issues Related to Program Improvement
Additional problems and issues	f
Low student enrollment	3
Lack of career guidance	3
Students' lack of prerequisite skills and knowledge	3
Lack of assessment	2

Implications and Recommendations

The results of this study indicate that the majority of the respondents strongly support the need for program improvement in vocational programs. The factors that motivate teachers to improve their programs appear to include caring for students, concern for professional growth, and a desire to keep programs current with changing technologies. In addition, respondents clearly indicated that program improvement enhances professional growth. This implies that program improvement can play a significant role in furthering the professional growth of teachers. It is somewhat surprising that the respondents did not perceive legislative mandates and personal recognition as major factors that motivate them to improve their vocational programs. Rather, vocational educators in this study also cited the necessity of having supportive environments to facilitate program improvement.

Most respondents indicated that they were uncertain about the adequacy of their pre-service teacher training with regard to providing the required knowledge and skills needed to initiate and sustain effective program improvement activities. However, while most respondents were confident that they are capable of conducting successful program improvement activities, the majority of respondents disagreed that they had adequate time to plan for program improvement activities.

There appeared to be some disagreement regarding the extent to which external factors (i.e., schools and administrators) are effective in providing support for program improvement activities. For example, the respondents were skeptical about the ability of administrators to reduce unnecessary bureaucracy. Similarly, they were also less than positive about the ability of administrators at eliciting input from staff regarding school policies and the ability of schools to disseminate information and knowledge related to program improvement. There was some disagreement about the extent to which their schools are providing adequate resources to facilitate and support program improvement. They were also concerned about ambiguity related to policies and program changes that were being implemented in their schools. Sustained program evaluation was apparent only in some instances. The respondents agreed that input from the community and parents is essential to support successful program improvement activities.

In general, the respondents rated the role of professional development in program improvement relatively high . They agreed that in-service training, professional development, and networking with professional associations are important contributors to program improvement activities. They also agreed that input from advisory committees was crucial for program improvement. However, they were concerned with a perceived lack of adequate opportunities for in-service training and professional networking. Most respondents indicated that they are able to network and engage in teamwork effectively with their colleagues.

As expected, the teachers were supportive of new trends in education, such as emphasizing student learning styles, critical thinking, and problem-solving. They believed that these issues are crucial elements in achieving the objectives of program improvement. They also rated school and business partnerships very high. However, the respondents were uncertain about the effectiveness of schools in facilitating collaboration between vocational and academic programs. This suggests that efforts directed toward integrating academic and vocational programs need to be enhanced by all target audiences involved.

Similar factors were considered important when planning program improvement activities, especially: funding, school and community support, teacher competencies, collaboration between teachers and administrators, and student needs. Negative teacher attitudes, lack of clear direction, poor communication, and inadequate resources were identified as major barriers that impede efforts to initiate and/or maintain program improvement. In terms of essential knowledge and skills needed when planning and/or initiating program improvement, the respondents suggested interpersonal communication skills, content and technical knowledge, awareness of student needs, and leadership skills are important characteristics to possess. The additional issues related to program improvement identified were low student enrollments, insufficient assessment information, need for career guidance for students, students’ insufficient prerequisite skills and knowledge, and lack of incentives to conduct program improvement activities.

This study had several limitations. Specifically, it was restricted to secondary vocational educators and it did not deal with all potential definitions or descriptions of program improvement. Further, the study utilized an adequate but limited sample from which to generalize the results. However, these limitations are perceived as practical limitations and do not necessarily inhibit the methods used nor the findings and conclusions reported.

Therefore, based on the limitations and results of this study, several recommendations for practice and future research are offered:

1. Federal, state, and local legislation should provide clear directions to vocational educators regarding program improvement.
2. Supportive environments that include incentives, such as sufficient time and adequate funding should be fostered by school administrators to assist educators in initiating program improvement activities.
3. Colleges and universities should reassess their pre- and in-service teacher education programs in terms of the degree to which they focus on the development of professional development and program improvement competencies.
4. School administrators should eliminate or drastically reduce unnecessary bureaucracy, elicit input from staff, and disseminate information regarding program improvement activities.
5. Legislation and schools should provide resources and funding to facilitate program improvement activities.
6. Vocational educators need more opportunities for in-service and professional development training to support their program improvement activities.
7. More emphasis should be focused on interpersonal communications skills, critical thinking and problem-solving skills, and technical knowledge in teacher education and secondary programs.
8. Vocational educators must consider student needs in program improvement planning.
9. Future research should seek to expand and refine the concept of program improvement and use larger and more diverse samples.

Authors

Greenan is Professor and Chair, Vocational and Technical Education, Department of Curriculum and Instruction, School of Education, Purdue University, West Lafayette, Indiana.

Wu is Associate Professor, Teacher Education Center, National Yuhlin University of Science and Technology, Taiwan, R.O.C.

Mustapha and Ncube are Graduate Assistants, Department of Curriculum and Instruction, Purdue University, West Lafayette, Indiana.

References

Allport, G. W. (1935) . A handbook of social psychology . Chicago: Clark University Press.

Ames, C., De Stefano, L., Watkins, T., & Sheldon, S. (1995). Teachers’ school-to-home communications and parent involvement: The role of parent perceptions and beliefs . (ERIC Document Reproduction Service No. ED 383451)

Brown, B. F. (1984) . Crisis in secondary education . Englewood Cliffs, NJ: Prentice-Hall Publishers.

Brown, J. M., & Davison, L. J. (1991). Adapting vocational education teacher training programs to work place changes. Journal of Studies in Technical Careers , 13(3), 285-295.

Committee for Economic Development (1985) . Investing in our children: Business and the public schools . Washington, DC.

Fazio, R. H. (1986). How do attitudes guide behavior? In R. M. Sorrentino & E. T. Higgins (Eds.), The handbook of motivation and cognition: Foundations of social behavior . New York: Guilford Press.

Gagne, R. (1977) . The conditions of learning (3rded.). New York: Holt, Rinehart, and Winston.

Greenan, J. P. (1986). Curriculum and assessment in generalizable skills instruction. The Journal for Vocational Special Needs Education , 9(1), 3-10.

Greenan, J. P. (1994). The education reform movement and school-to-employment transition of youth. In A. J. Pautler (Ed.), High school to employment transition: Contemporary issues . Ann Arbor, MI: Prakken Publications, Inc.

Guilford, J. P. (1954) . Psychometric methods . New York: McGraw-Hill.

Layne, R. G., & Forester, G. (1991) . The cutting edge: Sharpening faculty skills in a changing world . Paper presented at the Thirteenth Annual International Conference on Teaching Excellence, Austin, TX. (ERIC Document Reproduction Service No. ED 336130)

McFarland, A. H. (1990). Bring back the qualified educator. Vocational Education Journal , 65(7), 94.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded source book . Thousand Oaks, CA: Sage Publications.

National Commission on Excellence in Education (1983). A nation at risk: The imperative for educational reform: A report to the nation on the Secretary of Education . Washington, D.C.: Superintendent of Documents.

Neubert, D. A. (1990). Vocational assessment: Effective intervention for meeting the vocational needs of rural youth with special needs. The Journal for Vocational Special Needs Education , 12(2), 17-22.

Pintrick, P. R., & Schunk, D. H. (1996). Motivation in education: Theory, research, and applications . Englewood Cliffs, N.J.: Merrill.

Remmers, H. H., & Gage, N. L. (1955) . Educational measurement and evaluation . New York: Harper and Brothers.

Thomas, C., Haskell, R., & McNelly, D. (1989). Tennessee vocational teacher needs assessment . Knoxville, TN: The University of Tennessee, Office of Research, Department of Technological and Adult Education. (ERIC Document Reproduction Service No. ED 317753)

United States Statutes At Large. (1984) . Carl D. Perkins vocational education act. (PL. 98-524) . Washington, DC: United States Government Printing Office.

William T. Grant Foundation Commission on Work, Family, and Citizenship. (1988). The forgotten half: Pathways to success for America’s youth and young families . Washington, DC: Author.

Womble, M. N., Jones, K. H., & Ruff, N. S. (1995). Employment readiness of urban youth: A study of perceptions of students enrolled in vocational courses. Journal of Vocational Education Research , 20(3), 51-83.

Reference Citation: Greenan, J., et al. Attitudes and motivations of vocational teachers regarding program improvement. Journal of Industrial Teacher Education , 35(3), 6-23.