Volume 16, Number 2

Spring 2005

Articles

Assessment of Project-Based Learning in a MECHATRONICS Context

Yaron Doppelt

Introduction

Project-based learning (PBL) that has authenticity in the pupils' world enables the teaching of science and technology to pupils from a variety of backgrounds. PBL has the potential to enable pupils to research, plan, design, and reflect on the creation of technological projects ( Doppelt, 2000 ). Imparting creative thinking within the design process of pupils' projects not only requires changing the teaching methods and learning environment, but also adopting new assessment methods such as student portfolios. Engineering education, which is common in Israel, has a unique structure in that it combines practical and theoretical knowledge, synthesizes vertical and lateral thinking, and creates a rich and flexible learning environment.

The CTT (Creative Thinking in Technology) program ( Barak & Doppelt, 1999 ) integrates Co.R.T. Thinking tools ( De Bono, 1986 ) into the technology curriculum using the LEGO-Logo learning environment for creating authentic projects. The program began in 1994. Pupils study lateral thinking tools in order to deal with different alternatives, to consider multiple factors, and to refrain from premature judgments on ideas. They use vertical thinking tools in order to document their design process and to calculate and to structure programming for the control of their projects. Earlier field research by Barak, Waks, & Doppelt ( 2000 ) showed that pupils prefer a learning environment that emphasizes planning and building activities and team projects. Pupils have stated that these aspects of a learning environment contribute to creating challenges, curiosity, imagination, and success in studying technological subjects ( Doppelt & Barak, 2002 ). As the CTT program evolved, a Creative Thinking Scale (CTS) was developed in order to assess pupils' portfolios ( Barak & Doppelt, 2000 ). The study reported herein extends research conducted on the CTT program and proposes a Creative Design Process (CDP) that can be assessed using the CTS.

Theoretical Background

The theoretical background of this study cuts across three areas of inquiry: engineering education, the infusion of creative thinking into the design process, and the assessment of project-based learning.

Engineering Education: A MECHATRONICS Context

De Vries ( 1996 ) claimed that we should help pupils integrate knowledge from science and other disciplines into the design processes. It is evident that there is a role for science education and that science education remains a crucial part of general education, even where technology education has gone beyond the "technology is applied science" paradigm. Technology education is an equally valuable subject to science education, and both subjects should be taught ( Gardner, 1997 ).

Engineering education in Israel is part of the comprehensive high school curriculum. At the end of junior high school, pupils have to choose one or more areas as a "major," such as the Sciences, Humanities or Technology. Within Technology is Engineering. The Engineering curriculum for high school in Israel contains several core subjects that are related to physics and mathematics, such as civil engineering, computers and electronics, and mechanics and control systems. MECHATRONICS is a new sub-major in the mechanics and control area. Pupils study MECHATRONICS for three years, 10th – 12th grades. They study physics, system control, mechanics, and programming for five hours per week, with a total time devoted to all classes of 20 hours per week. This syllabus is about half of their weekly schedule.

The MECHATRONICS syllabus has been implemented in nine schools since 2000/2001. In 2003 there were fifteen schools that chose to advise their pupils to take this syllabus. The MECHATRONICS curriculum suggests that educators create rich learning environments filled with real world applications. The assessment processes of learning outcomes in a rich learning environment have had an important impact on the learning process ( Doppelt & Barak, 2002 ). In addition, the perspective of pupils on the most influential characteristics of the learning environment is important for the teaching-learning process and for designing the learning environment ( Doppelt, 2004 ). This study suggests a way of infusing creative thinking into the design process during project-based learning (PBL).

Infusing Creative Thinking into the Design Process

The design process is similar to the process of problem solving. Many general structures for design processes appear in the literature. A generalized approach to structuring the design process might include six stages: defining the problem and identifying the need, collecting information, introducing alternative solutions, choosing the optimal solution, designing and constructing a prototype, and evaluating and correcting the process. The design process might also include four thinking levels:

1. Understanding existing systems
2. Systematic and functional understanding of engineering systems
3. Applying set procedures for analysis and synthesis
4. A controlled design process that assures that the above steps are applied by the students.

( Mioduser, 1998 )

Teaching technology using a generalized design process has been criticized by researchers who have claimed that it is difficult for pupils, and even for teachers, to learn how to use it ( McCormick & Murphy, 1994 ). This is not surprising given that different technology disciplines go about designing in different ways ( Hill & Anning, 2001 ). In order to avoid the pitfalls of a generalized design process, teachers should assist pupils to integrate the knowledge of the various disciplines and learn the applicable standards and rules, as well as the underlying scientific principles and economic concepts ( De Vries, 1996 ). This article will describe a design process that applies these suggestions and is sensitive to the authenticity of the pupils' projects. This design process was based initially on the PISCO framework ( De Bono, 1986 ). PISCO stands for: Purpose, Input, Solutions, Choice and Operations. Infusing the teaching of thinking skills into a specific disciplinary course may provide a rich learning environment that will contribute not only to the development of thinking skills but also to a better understanding of the discipline under study ( Glaser, 1993 ; Ennis, 1989 ; Zohar & Tamir, 1993 ).

De Bono ( 1986 ) differentiates between two types of thinking: lateral thinking , which refers to discovering new directions of thinking in the quest for a wealth of ideas, and vertical thinking , which deals with the development of ideas and checking them against objective criteria. Lateral thinking and vertical thinking are quite different processes. It is not a matter of one process being more effective than the other, as both are necessary. Rather, in order to be able to use both effectively, one must appreciate their differences. Lateral thinking is a central, but not singular, component of creative thinking. Waks ( 1997 ) pointed out that during work on a technological project, lateral thinking initiates the learning process while pupils seek alternatives and examine different solutions. Vertical thinking is essential at the stage of choosing a solution and developing it. Vertical thinking and lateral thinking complement each other, and both are the essential elements of creative thinking ( De Bono, 1986 ).

Assessing Creative Thinking in PBL

Technological systems that are controlled by a computer provide a rich learning environment and expose the learner to a variety of representations and configurations, such as realistic models, simulations, mathematical models, algorithms, graphics, and animations. One of the better-known examples of such a rich computer-based learning environment is the LEGO-Logo system ( Doppelt & Armon, 1999 ; Jarvinen, 1998 ). Resnick and Ocko ( 1991 ) stated that this learning environment puts children in control because they formulate their own designs and experiments and work on projects that they care about personally instead of reproducing someone else's experiment.

Project based learning in technology encourages pupils to work in teams ( Barak & Maymon, 1998 ; Denton, 1994 ). In this way, pupils combine "handson" activities with what Papert ( 1980 ) has termed "heads-in" activities. Projectbased learning (PBL) could be used as a tool to develop pupils' competencies by working on integrated projects ( Barlex, 2002 ). Project-based learning through authentic issues taken from the pupils' world enables pupils from various backgrounds to study science and technology in a way that makes sense to them ( Seiler, Tobin & Sokolic, 2001 ). An authentic project deals with real life situations and, by definition, is integrative in nature. This approach has been implemented throughout the world and past researchers have shown interesting findings regarding the opportunities afforded by PBL ( Barak & Doppelt, 1999 , 2000 ; Barak, Eisenberg & Harel, 1995 ; Barlex, 1994 ; Doppelt, 2003 ; Doppelt & Barak, 2002 ; Hill, 1998 ; Resnick & Ocko, 1991 ). One of the key elements for success in PBL is engaging the pupils in the assessment process.

Imparting creative thinking within science and technology education not only requires changing the teaching methods and learning environment, but also adopting new assessment methods, such as portfolio assessment, which is based on records of pupils' activities. The portfolio might consist of such items as written material, computer files, audio and video media, sketches, drawings, models, and pictures. The portfolio reflects what pupils have learned, how they question, analyze, synthesize, solve problems, and create new ideas, and how then design and build useful products or systems. The portfolio also shows how pupils interact intellectually, emotionally, and socially with others ( Collins, 1991 ; Wolf, 1989 ).

Barak and Doppelt ( 1999 ) have shown that pupils were able to cope with complex problems and they developed solutions that depended on creativity, supporting the notion that they applied lateral and vertical thinking. Pupils created portfolios containing documentation of their creative thinking and the learning processes in which they were engaged. Over a period of several years, each class developed criteria for assessing their portfolios. On the basis of these experiences, a Creative Thinking Scale (CTS) was developed and was formally applied to the assessment of pupils' portfolios ( Barak and Doppelt, 2000 ).

The suggested assessment scale of creative thinking can help educators strive for a gradual development of higher-order thinking skills in two main areas. The first is the choice of a project topic that that pupils make. This includes dimensions of complexity, originality, and creativity on the one hand and the extent of mathematical, logical, and scientific thinking on the other. The second area considers the thinking and learning processes applied as the pupils developed their project and includes problem solving, teamwork, and reflective thinking. Thus, learning through the designing and implementation of technology projects based on portfolio assessment and directed towards a systematic development of vertical and lateral thinking may promote teaching and learning that assist the successful integration of the pupil into the dynamic and changing world outside of the school once they graduate.

Background and Precursors

Intervention Program

The intervention program included in-service training of teachers for a duration of 112 hours. This consisted of twenty-eight workshops of four hours per meeting during the school year. The program was aimed at introducing three main topics: the tutoring process, the Creative Design Process (CDP), and assessment using the Creative Thinking Scale (CTS) criteria. Eighteen teachers from nine schools who had taught MECHATRONICS since 2000 were involved in the project.

Six Stages of the Creative Design Process (CDP)

The creative thinking framework was originally developed by De Bono ( 1986 ). He based this framework on five thinking steps that he named "PISCO" - Purpose, Input, Solutions, Choice and Operations. The Creative Design Process (CDP), which is presented in this article, adopts this creative thinking framework and extends it to project-based learning (PBL) in technology education.

The creative thinking tools that are suggested in the CDP are part of the Co.R.T thinking program ( De Bono, 1986 ). They include P.M.I (Plus, Minus, Interesting), C.F.A (Consider All Factor), Rules, C&S (Consequence and Sequel), O.P.V (Other People's View) and F.I.P (First Important Priorities).

First stage: Design Purpose .The first step in the design process is defining the design problem. The pupils need to set the design goals. These goals must fit the definitions of the problem. The achievement of these goals will be under the restrictions that the designer has set forth and include budget, availability, equipment and tools, schedule, and so on. Three steps are recommended to the pupils for the documentation of this first stage:

1. The problem and the need – Pupils describe the reasons that motivated them to choose their project. They also define the problem and define the needs that their solution will address.
2. The target clientele and restrictions – Pupils describe the target clientele and define the restrictions that they must take into consideration. The Rules thinking tool can help you consider rules, standards, and other restrictions.
3. The design goals – The pupils define the necessary demands they expect to be met by the system.

Second stage: Field of Inquiry . The second stepin the design process is to define the field of inquiry in which their problem resides. This is founded on the problem definitions and goals from the first step. Pupils must research and analyze existing systems that are similar to the one they are developing. Pupils need to organize the documentation of their inquiry. This includes the following main areas:

1. Information Sources - Books, professional magazines, manufacturers' catalogs, and Internet sites.
2. Identification of Engineering, Scientific, and Societal Aspects - Engineering concepts, scientific concepts, societal and environmental aspects, cultural values, and potential issues and dilemmas.
3. Organization of the Information and its Assessment – Arrangement of the information according to the goals and restrictions of the problem. Pupils need to summarize the information gathered so that the design problem and approach are informed by it. Pupils should express their opinions regarding the match of the information they have gathered to their design problem. They must also provide a rationale for why their evolving design is a better alternative to systems that already exist.

Third stage: Solution Alternatives. The third step in the Creative Design Process involves the consideration of alternative solutions to the design problem. This is a lateral thinking stage and includes three components: Ideas, Factors, and Opinions (of other people). Pupils need to be educated so that they feel freedom in their thinking and are not discouraged by the judgments that other pupils and their friends might make. This strategy will increase the likelihood that numerous design possibilities will be considered, with the intent of inspiring creativity and arriving at an idea that no one else has developed. There are no bad ideas at this stage. The pupils are presented the following suggestions and guidelines:

1. Ideas Documentation – The use of the P.M.I ( P lus, M inus, I nteresting) thinking tool is recommended to the pupils so that they consider as many ideas as possible and formally evaluate them, considering all aspects. The aspects of the ideas are rated as positive, negative, or interesting, meaning that they have promise but more investigation is needed.
2. C onsider A ll F actors ( C.A.F ) – Pupils are asked to write down all the factors related to the system they are designing. They must consider the perspectives of the consumer, the designer, the manufacturer and the marketer.
3. C onsequence and S equel ( C&S ) – Pupils must consider and document all of the consequences of each of their ideas on such elements as the environment, society, and the individual. Both short term and long term consequences must be considered.
4. O ther P eople's V iew ( O.P.V ) : - Pupils must seek the opinions of others about their ideas and they must document what they find.

Fourth stage: Choosing the Preferred Solution. - The fourth step in the design process is choosing the preferred solution. The choice is made from the various ideas that were documented in the third stage. The solution chosen will possibly fulfill the following criteria:

1. Have the largest amount of plus and/or interesting points and the least minus points
2. Considers as many factors and viewpoints as possible.
3. Is adequate in both the short and long term.
4. Appears to be a good solution in the minds of others.
5. Meets all the requirements set forth in the definition of the problem.

Table 1 is designed to assist pupils in choosing the best solution.

Table 1

An evaluation scheme for evaluation pupil ideas

	Very Weak (1)	Weak (2)	Average (3)	Good (4)	Very Good (5)
Positive Points
Interesting Points
Factors Involved
Viewpoints
Short/Long Term
Other People View
Necessary Demands
Desirable Demands

The use of the F.I.P ( F irst I mportant P riority) thinking tool is recommended to pupils in order to help them set priorities. This thinking tool could assist in choosing the optimal solution.

Fifth stage: Operation Steps. The fifth step in the design process is planning the operational ways to implement the chosen solution. Planning must be consider the following points:

• Sometimes the chosen solution is a complex system. Dividing it into subsystems may assist in defining the steps that are needed to develop the solution.
• In many cases choosing the ideal materials, parts, and mechanisms are a central part of the design process.
• The sketches and drawings (computer-generated as well as traditional) are important to the presentation of a design.
• Choosing machines, tools, and manufacturing processes are necessary steps to creating a prototype.
• Planning how to make the prototype is critical to success. Planning activities include developing a timeline of tasks and making sure the necessary materials, parts, machines, human resources, and so forth are available and will be available when needed.

Six stage: Evaluation. The last step is to evaluate the overall process and product. This is a summative evaluation and relates back to the formative evaluation steps that were done as the idea was developing. In this final stage the pupils need to document:

• What difficulties were encountered and what methods were used to overcome them?
• Does the system actually provide a viable solution to the problem?
• Does the prototype fulfill all its performance requirements?
• In what way could the prototype be improved?
• What are the implications for further development?

The Creative Thinking Scale

In this study the Creative Thinking Scale (CTS) was used to assess the suggested Creative Design Process (CDP). The CTS was introduced to the teachers during the workshops mentioned earlier. It consists of the four thinking levels that De Bono ( 1996 ) defined:

1. Awareness of Thinking
   The first level deals with developing an awareness that thinking is a skill that can be developed. Pupils are taught how to prepare to engage in thinking about something, how to conduct inquiry, and how to listen to and evaluate the opinions of others.
2. Observation of Thinking
   The second level deals with observing the consequences of action and choice, considering other people's views, and comparing alternatives.
3. Thinking Strategy
   The third level deals with the directed use of some thinking tools, organizing thinking as a sequence of steps, and using thinking to define goals.
4. Reflection upon Thinking
   The fourth level deals with a systematic use of thinking tools, clear awareness of the need for reflective thinking, self-evaluation of thinking, designing thinking tasks, and methods to implement these tasks.

The CTS evaluates the pupils' portfolios across two dimensions. The first considers the design, construction, and evaluation of the product or system. Evidence of lateral thinking, including originality, authenticity, usefulness, and unique design is sought. Likewise, evidence of vertical thinking is also sought and includes functionality, reliability, accuracy, geometric structure, and the application of scientific principles.

The second dimension considers the processes of learning and includes thinking, problem solving, and teamwork. Evidence is sought of individual and group efforts in problem solving, collaborative decision-making, and leadership. Table 2 presents the CTS, the development of which was detailed elsewhere ( Barak & Doppelt, 2000 ).

Table 2

The creative Thinking Scale(CTS)

	Portfolio's Components
Achievement Levels	Design, construction, and evaluation of the system or product	Learning, thinking, and problem-solving activities
Level 1: Awareness The awareness to consider thinking as a skill which can be developed; prepare to think about something; prepare to inquire; prepare to listen to other people opinions.	Standard diagram of a system or product taken from available literature. Basic explanation of the model and its construction. Description of the model by means of pictures or sketches.	An example ofsolving a simple problem in planning and construction. Division of tasks among the team members. A few examples of using lateral and vertical thinking tools.
Level 2: Observation The observation of consequences of action and choice; consider the views of others; compare alternatives.	Original schematic diagram of system or product designed by the pupil. Detailed drawings of the model. Specification of planning and construction stages including calculations, specifications or computer programs.	Justified examples of choices among a number of alternatives. Information exchange and reciprocal help in the team. Various examples of using thinking tools.
Level 3: Strategy The directed use of some thinking tools; organizing the thinking as a sequence of steps; using thinking to define goals.	Original system functional block diagrams, structural tree or flow chart. Description of a number of iterations in the planning and construction of the model. Comparison among possible models and choosing from them.	Examples of the contribution of individuals and teamwork to solving complex problems. Evidence of the planned use of the thinking tools, open-mindedness, and postponing decision making (lateral thinking); setting priorities, goals and criteria (vertical thinking).
Level 4: Reflection A systematic use of thinking tools; clear awareness of reflective thinking; self-evaluation of thinking; designing thinking tasks and methods to implement these tasks.	Examination of the final product's features, compared to the set goals. Conclusions on successes or difficulties during the development process. Suggestions for improvement in the planning and construction process.	Conclusions drawn from the influence of the team's collaboration on the completion of the project. Pupils' view on the influence of the team's functioning on thinking and learning processes. Assessment of the selected solution compared to the goals.

Method

This study aimed first to investigate the way pupils design their projects. The second aim was to describe the assessment of Project-Based Learning (PBL). The third aim was to explore the ways teachers set goals for their pupils according to the CDP and the CTS. A qualitative approach was used in order to foster collaboration with teachers and to gain close interaction with pupils, teachers, principals, and supervisors from the Israeli Ministry of Education.

In 2003 there were nine schools that offered MECHATRONICS as a major during the 10th-12th grades (16-18 years old). Eighteen teachers were involved in delivering the MECHATRONICS courses. Approximately 180 pupils have graduated from high schools with MECHATRONICS as their major. Pupils' projects are examined through a matriculation examination at the end of the 12th grade. In these examinations a supervisor arrives at each school and the pupils are required to present a portfolio of their design process and the final product or system that resulted. Team projects are assessed the same as single projects. The examination supervisor asks questions of each of the pupils in the team.

A national contest was organized for all the Israeli pupils in the MECHATRONICS programs. All the pupils knew at the outset of the contest that they would be required to design, construct, and to program a system that would assist humans. Criteria for assessing the projects were developed with the teachers. The teachers were instructed to introduce the criteria to their pupils six months prior to the contest. The assessment process during the contest itself focused on evaluating the design process and resultant products based upon a presentation made by the pupils.

Agreement was reached among the teachers on the criteria for assessing the pupils' work during a meeting six months before the contest. These criteria were validated in advance of this meeting through the agreement of five senior teachers from schools that were not participating in the contest and five researchers from the Technion – Israel Institute of Technology. These ten individuals were invited at the contest to serve voluntarily as members of the assessment committee.

Participants

In all nine schools the teachers assessed the projects using the developed criteria. Each school could send three teams to the contest. Fifty-four pupils were chosen by their teachers to participate in an Israeli national MECHATRONICS contest, which was held at the Technion. The pupils were in the second semester of their 12th grade and they had been learning MECHATRONICS according to the syllabus that was described earlier. The teachers of these pupils, eighteen from the nine participating schools, applied PBL in their classes during the 12th grade.

Data Collection and Analysis

As mentioned earlier, the CDP and the CTS had been introduced to the teachers during a workshop. The author actively participated in the workshop. During the workshop, criteria for assessing PBL were agreed upon among the teachers. These criteria were used to assess the pupils' projects in the national contest. Six month prior to the contest, the teachers and pupils became familiar with the assessment criteria. The pupils were required to present their projects during the contest. By way of these presentations, the researcher was able to investigate the implementation of the CDP and the impact of the CTS upon pupils' projects.

Findings

The findings are presented in three categories. First, projects are briefly described, showing the variety and authenticity of the pupils' work. Second, a representative example of the assessment process is presented. Finally, the judges' assessment of the MECHATRONICS projects in the national competition is presented.

The Projects

Fifty-four pupils were eligible to enter the national MECHATRONICS contest, representing 18 projects. Of these, three teams were unable to bring their projects to the final level needed to actually compete in the contest. In the end, forty five pupils representing 16 projects were in the competition. Only one project was done by a student working alone.

The Judging Process

The pupils presented their projects to an assessment committee consisting of teachers from the participating school, senior teachers from schools that were not participating in the contest, and researchers from the Technion - Israel Institute of Technology. Procedures were used to assure that a given project was not judged by a teacher of the pupils who produced it.

There were two stages to the assessment process. In the first stage, the projects were presented to teams of three judges. Approximately three projects were evaluated by each team of judges. From this procedure, six projects moved on to the second, or final, stage of the evaluation process. The descriptions in Table 3 show that the projects were authentic and varied.

Table 4 presents a representative project, which demonstrates the assessment process. This example is project No. 7 from the final assessment presented in Table 3. These findings show a high-level of agreement among the independent assessments of each criterion as scored by the judges. The scale for the scoring was 10 percent for the first and the last criteria, and 20 percentages for each of the other criteria.

Table 3

MECHATRONICS Projects

Project No.	Project Description	Pupils on Team
1	Automated control system for a hoisting machine	4
2	Automated, multi-player basketball game	3
3	Simulator for jogging	2
4	Automated system for changing and playing compact discs	1
5	Computerized Scanner	2
6	System for coaching Ping-Pong players	6
7	Simulation of riding a bicycle	2
8	Automated system for finding and collecting tennis balls	2
9	Computerized system for playing chess.	3
10	Computer controlled model of a detention facility	2
11	Computer controlled system for coaching boxers	5
12	Automated system for replacing wheels	3
13	System to assist pupils in learning about computer control	2
14	System for coaching tennis players	3
15	Automated system for identifying and neutralizing bombs	3
16	Computer controlled system for pumping water	2
Total		45

Table 4

Independent judgment according to agreed criteria

Criteria	Judge No.1	Judge No.2	Judge No.3	Mean
Presenting the needs and the system's goal	9	8	10	9.0
Presenting alternative and creative solutions	19	15	18	17.3
Analyzing the chosen system	18	15	18	17.0
Performance of a working controlled prototype	12	13	12	12.3
Sophistication of the control program	20	17	18	18.3
Presentation of design stages	8	7	8	7.7
Total	86	75	84	81.7

Judge 1 was a senior teacher who had not participated in the intervention but was familiar with the criteria. The correlation between judges 1 & 2 was 0.957; between judges 1 & 3 it was 0.990 and between judges 2 & 3 it was 0.944. Similarly, high correlations were found among all of the other assessments, which were scored by the other judging teams. The assessments at the same time and there was no interaction between one team and the other teams. This shows that the judges have a shared perception of the criteria.

The Final Assessment of the Projects

Table 5 presents the findings from the final assessment process. In the final stage the pupils presented the six projects, shown in bold in the table, to the committee in a large auditorium. In the auditorium 160 guests watched the final contest and included families, teachers, school principals, supervisors from the Ministry of Education, and researchers from the Technion. The assessment committee watched each presentation and assessed the six projects according to the developed criteria, as mentioned previously. The six projects were an automated controlled system for lifting a hoister weight, an automated multiplayer basketball game, a mini football game, a jogging simulator, an automated system for changing and playing compact discs, a computerized scanner, and a system for coaching ping-pong players.

Table 5
Final scoring of each participating project

1	10.0	17.7	20.0	20.0	19.7	8.7	96.0
2	10.0	17.7	19.7	16.7	20.0	10.0	94.0
3	10.0	16.7	18.3	20.0	18.3	10.0	93.3
4	10.0	18.0	17.7	17.3	20.0	9.3	92.3
5	8.7	14.7	17.7	20.0	19.3	9.0	89.3
6	9.0	18.3	16.3	16.3	18.7	8.7	87.3
7	9.0	17.3	17.0	12.3	18.3	7.7	81.7
8	9.7	15.3	17.7	14.7	14.0	9.0	80.3
9	7.7	15.3	15.0	18.3	15.3	8.3	80.0
10	8.0	13.3	14.3	16.0	20.0	8.0	79.7
11	8.0	13.3	15.0	20.0	15.0	8.0	79.3
12	6.3	13.3	12.3	20.0	17.7	8.0	77.7
13	7.0	11.3	16.7	17.7	19.0	5.3	77.0
14	8.3	15.0	14.3	13.3	16.7	8.3	76.0
15	10.0	13.3	12.3	16.0	11.7	10.0	73.3
16	6.7	13.3	12.3	15.0	15.0	7.0	69.3

Discussion

Seven years of experience in implementing the CTS in order to assess pupils' projects demonstrates how this methodological assessment can help educators develop and evaluate learning assignments aimed at fostering creative thinking in technology ( Doppelt & Barak, 2002 ). Through the CDP and systematic reflection on it, pupils can develop awareness of their internal thinking processes and document them. The purpose is not to educate pupils to design according to some generalized procedure, external to them, for constructing their ideas, solutions, and products ( De Vries, 1996 ). This is counter to creativity. Rather, it is an educational goal to teach them to document their thinking properly and thereby enable them to reflect on their creations and how they developed them. Research was recently conducted in this area by Doppelt, Mehalik, and Schunn ( 2005 ).

Pupils are expected to internalize their adaptation of the design process, to use it in their own way, to apply it to new situations, and to demonstrate general patterns of lateral and vertical thinking in their technology projects. No less important is fostering pupils' meta-cognition, or 'thinking of thinking'. The way pupils commence, progress, and complete their project demonstrates that creative thinking in technology is a combination of vertical and lateral thinking ( Waks, 1997 ; Barak & Doppelt, 2000 ).

In addition, the projects reported herein show that pupils in high school can create, design, implement, control, and document authentic, real-life projects instead of solving well-defined problems prescribed by the teacher. In fact, the criticism of current engineering education is that there is an overemphasis on solving well-defined, closed-ended problems ( NSPE, 1992 ). Furthermore, pupils have proven through their projects that they are capable of dealing with the "large definition of DESIGN" – that the DESIGN activity does, in fact, encompass the entire process of planning, designing, constructing, and managing the development of a product ( De Vries, 1993 ; Hill, 1998 ).

The CTS has enabled teachers and researchers to set goals for the pupils (and for the teachers) during the PBL. The consistency of the judges' scores and the successful application of the criteria developed by the teachers strengthened their validity. The findings of the assessment process indicate that the CDP and the CTS are useful and can be implemented by teachers who have participated in a suitable in-service training. The assessment of technology education can serve as a highly integrative element in technology education that allows pupils to combine and integrate various knowledge and skills ( De Vries, 1997 ).

Conclusions

The in-service teacher workshop assisted the teachers in discussing tutoring issues, design stages, and assessment criteria. The independent judging showed a high-level of agreement among the judges. These assessments were similar to the researchers' and senior teachers' assessment in the final stage of the contest. The researchers and senior teachers had not taken an active part in the tutoring process during the school year. They were familiar, however, with the criteria and had agreed upon them. This model can be adopted in other relevant issues regarding collaboration between field practice and academic research.

This article introduced the Creative Design Process (CDP) and the creative thinking scale (CTS). The CDP is aimed at assisting pupils in documenting the design process. The CTS could be used as a guideline for teachers during their tutoring and for pupils during the development of creative solutions to problems. The findings showed that pupils learned to document their design process according to the CDP. Finally, teachers turned into better tutors after they became familiar with the CTS.

The implementation of the CTS concerning the outcomes of the CDP has important consequences for the professional development of teachers and for the development of pupils' skills. Teachers can use the CTS as the goal of their teaching. If the CTS is introduced together with the CDP to pupils, they can develop their competencies according to various learning styles. The methodological assessment used in this study during the intervention program with the teacher and during the contest with teachers, senior teachers and researchers can be used in other science and technology domains. The contest was also found to be a useful instrument to enhance collaboration between researchers and teachers and among schools. This research could add a relevant body of knowledge to the assessment of technology education.

Acknowledgements

The author wishes to acknowledge Mr. Doron Edelding and the Technion Science Technology Youth Center for assisting in the administration of the Mechatronics contest. In addition, thanks are due to Dr. Nadav Betzer and Mr. Ron Eizenberg from the Mechatronics Department in the Ministry of Education for their continuous efforts in promoting technology education in Israel.

References

Barak, M. & Doppelt, Y. (1999). Integrating the CoRT program for creative thinking into a project-based technology curriculum. Research in Science and Technological Education , 17(2), 139-151.

Barak, M. & Doppelt, Y. (2000). Using portfolios to enhance creative thinking. Journal of Technology Studies , 26(2), 16-24.

Barak, M., Eisenberg, E. & Harel, O. (1995). 'What's in the calculator?' An introductory project for technology studies. Research in Science and Technological Education , 12(2), 147-154.

Barak, M., & Maymon, T. (1998). Aspects of teamwork observed in a technological task in junior high schools. Journal of Technology Education , 9(2), 3-17.

Barak, M., Waks, S. & Doppelt, Y. (2000). Majoring in technology studies at high school and fostering learning. Learning Environment Research , 3, 135-158.

Barlex, D. (1994). Organising project work. In F. Banks (Ed.), Teaching Technology (pp. 124-143). London: Routledge.

Barlex, D. (2002, March). The relationship between science and design and technology in the secondary school curriculum in England. In I. Mottier & M. J. De Vries (Eds.), Proceedings of the PATT12 Conference (pp. 3-12).

Collins, A. (1991). Portfolio for biology teacher assessment. Journal of School Personnel Evaluation in Education , 5, 147-167.

De Bono, E. (1986). The CoRT thinking program (2nd ed.). Oxford: Pergamon Press.

De Bono, E. (1996). Master Workshop of de Bono's Thinking Course . Jerusalem: Branco Weiss Institute for the Development of Thinking.

De Vries, M. J. (1993). Design methodology and relationships with science: Introduction. In M. J. deVries; N. Cross; & D. P. Grant (Eds.), Design methodology and relationship with science (pp. 1-14). Boston: Kluwer

De Vries, M. J. (1996). Technology education: Beyond the "technology is applied science" paradigm. Journal of Technology Education , 8(1), 7-15.

De Vries, M. J. (1997, April). Technology assessment and the assessment of technology education. In I. Mottier and M. J. De Vries (Eds.), Proceedings of PATT8 Conference: Assessing Technology Education (pp. 373-378). Eindhoven, The Netherlands: PATT Foundation - Eindhoven University of Technology.

Denton, H. (1994). The Role of Group/Team Work in Design and Technology: Some Possibilities and Problems. In F. Banks. (Ed.), Teaching Technology (pp. 145- 151). London: Routledge.

Doppelt, Y. (2000, June). Developing pupils' competencies through creative thinking in technological projects . Paper presented at the 28th Israel Conference on Mechanical Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel.

Doppelt, Y. (2003). Implementaton and assessment of project-based learning in a flexible environment. International Journal of Technology and Design Education , 13, 255-272.

Doppelt, Y. (2004). The impact of the characteristics of science-technology learning environment: Pupils' perceptions and gender differences, Learning Environment Research .

Doppelt, Y., & Armon, U. (1999, August). LEGO/Logo (Multi-Techno-Logo) as an authentic environment for improving learning skills of low-achievers, Paper presented at the Euro-Logo Conference , Sofia, Bulgaria.

Doppelt, Y. & Barak, M. (2002). Pupils identify key aspects and outcomes of a technological learning environment, Journal of Technology Studies , 28(1), 12-18.

Doppelt, Y; Mehalik, M. M., & Schunn, D. C. (2005, April). A close-knit collaboration between researchers and teachers for developing and implementing a design-based science module. Paper presented at the annual conference of the National Association for Research in Science Teaching, Dallas, TX.

Ennis, R. H. (1989). Critical thinking and subject specificity: Clarification and needed research, Educational Researcher , 18(3), 4-10.

Gardner, P. (1997). The roots of technology and science: A philosophical and historical view. International Journal of Technology and Design Education , 7, 13-20.

Glaser, R. (1993). Education and thinking: The role of knowledge. In R. McCormick, P. Murphy, & M. Harrison (Eds.), Teaching and Learning Technology (pp. 91-111). Wokingham, England: Addison-Wesley in association with The Open University.

Hill, A. M. (1998). Problem solving in real-life context: Alternatives for design in technology education. International Journal of Technology and Design Education , 8(3), 203-220.

Hill, A. M. & Anning, A. (2001). Comparisons and contrasts between elementary/primary 'school situated design' and 'workplace design' in Canada and England. International Journal of Technology and Design Education , 11(2), 111-136.

Jarvinen, E. (1998). The learning environment in technology education: An experiment in a Finnish context. Journal of Technology Education , 9(2), 47-59.

McCormick, R. & Murphy, P. (1994). Learning the processes in technology . Paper presented to the British Educational Research Association Annual Conference, Oxford University, England.

Mioduser, D. (1998). Framework for the study of cognitive and curricular issues of technological problem solving. International Journal of Technology and Design Education , 8, 167-184.

National Society of Professional Engineers. (1992). Engineering education issues: Report on surveys of opinions by engineering deans and employers of engineering graduates on the first professional degree (NSPE Publication No. 3059). Alexandria, VA: The author.

Papert, S. (1980). Mindstorms, Children, Computers and Powerful Ideas . New York: Basic Books.

Resnick, M. & Ocko, S. (1991). LEGO/Logo: Learning through and about design, In: I. Harel & S. Papert (Eds.). Constructionism (pp. 141-150). Norwood, NJ: Ablex.

Seiler, G.; Tobin, K.; & Sokolic, J. (2001). Design, technology and science: Sites for learning, resistance, and social reproduction in urban schools. Journal of Research in Science Teaching , 38(7), 746-767.

Waks, S. (1997). Lateral thinking and technology education. Journal of Science Education and Technology , 6(4), 245-255.

Wolf, D. (1989). Portfolio assessment: Sampling pupil's work. Educational Leadership , 45(4), 35-39.

Zohar, A., & Tamir, P. (1993). Incorporating critical thinking within a regular high school biology curriculum. School Science and Mathematics , 93(3), 136-140.

---

Yaron Doppelt ( yaron@pitt.edu ) was the Academic Coordinator of the Science and Technology Youth Center, Technion - Israel Institute of Technology , Haifa, Israel. At the time this was published he was serving as a Research Associate at the Learning, Research, and Development Center (LRDC) at the University of Pittsburgh in Pennsylvania.