JOTS v26n2 - Using Portfolios to Enhance Creative Thinking

Using Portfolios to Enhance Creative Thinking

Moshe Barak and Yaron Doppelt

Education has long emphasized imparting cognitive competencies, such as logical-mathematical thinking, problem solving, and creativity, along with social and personal competencies. Infusing metacognition thinking skills into any course may provide a rich learning environment while also contributing to a better understanding of the discipline under study ( Ennis, 1989 ; Glaser, 1993 ). The constructivist learning approach also emphasizes these principles: Learning is an active process; thelearner absorbs information from the environment and derives meaning from it; learning needs to relate to pupils' daily lives; meaningful assignments place responsibility with the pupil and gives him or her freedom; and activity-based practice involves planning and constructing products and systems in an environment outside the school ( Brooks & Brooks, 1993 ).

Computerized technological systems can provide a rich learning environment which can expose the learner to a variety of experiences such as true modeling, simulations, building models that represent formulas, algorithms, graphics, and animation. One of the better known examples of such a rich computerized learning environment is the LEGO-Logo system. Resnick and Ocko ( 1991 ) believe this learning environment puts children in control since they formulate their own designs and experiments, and work on projects that they care about personally. Project learning also encourages pupils to work in teams ( Barak & Maymon, 1998 ; Barak, Maymon, & Harel, 1998 ; Denton, 1994 ). In this way, pupils combine "hands-on" activities with what Papert ( 1980 ) has termed "heads-in" activities.

Despite the increasing recognition of the educational importance of rich learning environments and projectbased learning, many educators do not have sufficient tools to realize the potential of technology education in fostering the development of higher order intellectual skills. Thus, it's crucial to delineate the higher order thinking skills we wish to inculcate as we search for ways to manifest them in pupils' work.

Creative Thinking as a Synthesis Between Lateral and Vertical Thinking

De Bono ( 1970 ) differentiated between two types of thinking: lateral thinking, which refers to discovering new directions of thinking in the quest for a wealth of ideas, and vertical thinking, which deals with the development of ideas and checking them against objective criteria. Vertical thinking is selective and sequential; it moves only if there is a direction in which to move. Lateral thinking is generative; it can make jumps and moves in order to generate a new direction. Lateral thinking does not have to be correct at every step and does not use fixed categories, classifications, or labels. Vertical thinking selects the most promising approach to a problem while lateral thinking generates many alternative approaches. According to De Bono, the processes of vertical and lateral thinking are both essential. Creative thinking is a synthesis of lateral thinking and vertical thinking, each complementing the other.

This view of creative thinking differs from the traditional approach in which curricula and research address creativity, mathematical-logical thinking, and critical thinking as separate entitities. Waks ( 1997 ) claimed that education-for-all programs should introduce lateral thinking habits in addition to the traditional vertical thinking ways stressed in the past. Scientific and technological changes in everyday life call for the ability to handle new situations.

Portfolio Assessment

These changes in teaching methods and learning environments necessitate new methods for assessing students' achievements. The concept of alternative assessment includes a variety of methods including performance assessment, open-ended problems, interviews, journal writing, exhibitions, oral examinations, and peer evaluations. All these evaluation tools are intended to promote learners' competencies in the cognitive area as well as the metacognitive area, interpersonal arena as well as personal development ( Gredler, 1995 ). These complicated competencies do not lend themselves to assessment by methods based on question and answer tests ( Berenson & Carter, 1995 ). Alternative assessment is an integral component of the teaching and learning process (rather than a concluding stage); it focuses on the learning process (rather than just the product); it tests understanding and thinking (rather than rehearsal and memory); and it is related to teamwork and the individual's contribution to the team. A dialogue takes place between teacher and pupil about the goals of assessment, the manner of its performance, and its conclusions. A learner's reflection about learning is a significant component of his/her success. And portfolio assessment is a major component of many alternative assessment methods.

A portfolio is a record of a pupil's learning process: what a student has learned and how he or she has gone about learning; how he or she thinks, questions, analyzes, synthesizes, produces, and creates; also, how one interacts intellectually, emotionally, and socially. Important ingredients of the portfolio are the learners' reflections at different times, the progress in their development, and future goals. This metacognition or "thinking about thinking" enhances what they learn since learners are often not aware of their internal thinking processes. Through reflection they think about their learning processes, learn to direct their own thinking, and subsequently plan their learning processes.

The Research

Objectives

This study explored:

1. The process pupils undergo while accumulating and completing their portfolio as an integral part of their technological project.
2. The role of one's portfolio for fostering higher-order cognitive skills.
3. A methodological scale for assessing pupils' creative thinking based on their portfolios.

Method

This study was part of a comprehensive study aimed at investigating the impact of project-based technology studies on pupils' self-confidence, self-image, motivation to learn, thinking competencies, and academic achievements ( Barak & Doppelt, 1998 ). Data were collected during ongoing work with the pupils, using documentation of class activities; discussions; informal talks or semi-designed interviews with pupils, parents, other teachers, and school staff; copies of pupils' portfolios; and examples of their projects in different stages. This article focuses on the process of developing and assessing pupils' portfolios.

Subjects

The subjects of the study were 10 ^th -grade pupils in a high school in northern Israel. The intervention program ran from 1994 to 1998. Fifty-six pupils participated in this program (9 to 24 pupils each year). All pupils had profiles of low academic achievement at the end of junior high school and most were deemed inadequate for studies at the level required to receive a matriculation certificate upon graduation from high school.

Intervention

The program, entitled "Creative Thinking and Technology" ( Barak & Doppelt, 1998 ), encompassed two hours of study each week during an entire school year. During the first semester of the school year (about 15 weeks), the class learned thinking tools from the CoRT thinking program developed by De Bono ( 1986 , 1994 ). Thinking tools such as PMI (Plus, Minus, Interesting), CAF (Consider All Factors), and APC (Alternatives, Possibilities, Choices) were studied.

After drawing on examples from the pupils' daily lives, learning focused on the process of constructing mechanical systems, such as a car or a robot, by means of the LEGO-Logo system. For example, all pupils constructed identical cars according to a given LEGO design, compared their features, and suggested improvements while using the CAF and APC thinking tools. In the course of this process, the pupils also became familiar with the LEGO-Logo system and the computer interface and with writing basic programs in the Logo language.

During the second semester (about 15 weeks), the pupils chose and performed original technological projects; for example, a robot that moved in forward or circular motions and cleared obstacles on the floor; an automatic conveyor belt that received, identified, and counted items loaded off a truck; a crane that scanned an area, collected objects that were randomly distributed, and delivered them onto a train; and a chocolate drink machine that filled powder into a glass, mixed it with milk, and delivered the glass onto a conveyor. The pupils coped with complex problems and found solutions that depended on creative thinking by synthesizing lateral and vertical thinking.

Data Collection

Pupils constructed 35 portfolios over the five years of the program. The difficulties they encountered while solving their problems, designing the construction, and programming the computers, and the teacher's hesitations about how to deal with various situations (such as how much to interfere or help with the pupils' work) were all documented.

Results

Process of Constructing a Portfolio

Assessment methods . After the pupils had experienced the application of thinking tools from the CoRT series while designing and constructing computerized systems in a LEGO-Logo environment, a discussion occurred at the end of the first semester, prior to the second semester. The teacher introduced the pupils to the following new principles for assessing their work: assessment refers to the work process and not just the final product; both peer assessment and self-assessment would take place; and the final grade would be awarded cooperatively by the teacher and pupils, according to predetermined weighting. The teacher and pupils cooperated in preparing examples of elements that may enter a folder, including sketches, drawings, calculations, flow charts, computer programs, photographs of the models in various stages of development, the thinking process that led to the model's design, the problems encountered by the pupils, and how they managed to overcome the problem.

Since the students were accustomed to receiving grades for turned-in assignments, it was hard to explain the value of drafts, documenting the difficulties they encountered, as well as the purpose of photos or sketches of initial models. To convince them that continuous documentation of their work was necessary, the teacher presented them with final models of pupils' work from previous years, along with the portfolio prepared by the pupils. Thus they saw that sometimes the final product was completely different from the first model. Previewing portfolios along with finished projects illustrated how previous pupils sometimes encountered complex problems that they managed to overcome. Pupils were persuaded that the final model alone without documentation of the different stages does not give an opportunity to see the design and construction process, the efforts invested by the pupils, and the pupils' achievements. Despite this, the pupils were not easily convinced, and each year the same question arose: "Why is it necessary to document all stages of the design and construction process?"

Weighting of the portfolio elements . About three weeks after the pupils had chosen their project topics, a class discussion took place, during which the pupils formulated the criteria for assessing their work and the weight of each criteria in the final grade. Table 1 reflects the pupils' lateral thinking (such as originality and creativity) and vertical thinking (such as usefulness, quality, and complexity). Criterion 7 reflects high-level skills in both vertical and lateral thinking to achieve a high performance level.

Teamwork and peer assessment . The teacher fostered an atmosphere of teamwork, knowledge sharing, and reciprocity in class. The richness and flexibility of the learning environment allowed each group to begin work on the project from a different point: planning, construction, calculation, or programming. As a result, some of the pupils acquired more knowledge or expertise in certain areas than other pupils. This created a basis for true cooperation, information transfer, and reciprocal help in problem solving among the pupils. Sometimes a pupil from one group would help pupils working on a different project. As a result, the pupils were familiar with their peers' work, the difficulties they had overcome, and the efforts invested by each one. This made peer assessment more valid. One year, the pupils decided that the weight of peer assessment in the final grade would be 70% while that of the teacher's assessment would be 30%. At the end of the school year, the pupils presented their work to their peers. They prepared a presentation that summarized their work in retrospect, reconstructed their management of difficulties, and demonstrated special achievements. This event also served as the formal stage of peer assessment. An example of peer assessment scores is also shown in Table 1.

Table 1
An Comparative Example of Peer and Teacher Assessment.
Criteria in Portfolio of Group 1 Weight Group 2 Group 3 Group 4 Group 5 Teacher 1.Originality 10% 9 8 8 10 10 2.Usefulness 25% 24 18 23 20 24 3.Considering all factors 15% 10 10 9 10 12 4.Computer program feasibility 20% 19 18 18 20 19 5.Technical quality, graphic editing and design 10% 6 7 7 10 6 6.Interesting subject 10% 8 7 7 10 7 7.Complexity number of subsystems / procedures 10% 4 7 5 10 5 Total 100% 80 75 75 90 83

The scores reported in Table 1 show a high internal consistency among peers' and teacher's scores (reliability coefficient alpha = 0.987). This indicates that the pupils and the teacher reached a strong consensus. In this example, the final grade was 81% (70% the average score of the four groups of pupils and 30% the teacher's score).

Discussion

Two Domains for Assessing Creative Thinking

In analyzing the pupils' portfolios, two domains stood out in assessing the pupils' creative thinking. The first domain relates directly to the development process of the product or the system planned and constructed by the pupils. Higher order thinking levels were expressed in the portfolios, for example, by describing unique system features, findings of conducted tests, performance attained in comparison to the original plan, difficulties the pupils encountered, and how they managed to overcome them. Examples of lower order thinking levels include a standard schematic diagram (taken from a book, for example) or a basic explanation about the purpose of the system and its mode of action.

The second domain relates to pupils' thinking and learning processes, teamwork, and cooperation in class. Pupils who reach a high level of lateral and vertical thinking may express these processes in their portfolios. They can mention, for instance, their hesitation in choosing among different alternatives while planning or problem solving, how they shared their work among themselves, or how they turned to members of other groups for help. Also, learners are not always aware of their internal thinking processes. But, constructing the portfolio caused pupils to reexamine the processes they went through. Pupils who reach higher order thinking may illustrate in the portfolio the manner in which they used the thinking tools they learned.

Developing an Assessment Scale

After identifying the above-mentioned domains for characterizing the level of creative thinking on the basis of the portfolio elements, we developed an assessment scale of the creative thinking level achieved by the pupils, based on the elements they included in their portfolios. De Bono ( 1996 ) defined four achievement levels of creative thinking skills development:

Level 1: Awareness of thinking . General awareness of thinking as a skill. Willingness to think about something. Willingness to investigate a particular subject. Willingness to listen to others.

Level 2: Observation of thinking . Observation of the implications of action and choice, consideration of peers' points of view, comparison of alternatives.

Level 3: Thinking strategy . Intentional use of a number of thinking tools, organization of thinking as a sequence of steps. Reinforcing the sense of purpose in thinking.

Level 4: Reflection on thinking . Structured use of tools, clear awareness of reflective thinking, assessment of thinking by the thinker himself. Planning thinking tasks and methods to perform them.

Table 2 offers characteristics of portfolio elements for each level in the two domains: (a) system or product design, construction, and evaluation, and (b) learning and thinking activities.

The scale presented in Table 2 is clarified below, including several genuine examples from pupils' portfolios.

Domain A-Level 1: Presenting a system pictorially accompanied with basic explanations. This kind of documentation is graded relatively low on the creative thinking scale since it expresses mainly the pupil's awareness of the need to present his or her work before others, labeling its parts, and providing basic explanations.

Domain A-Level 2: System documentation by schematic electrical or mechanical drawings and computer programs. This type of documentation is ranked at the second level of the creative thinking scale since the pupils have to show how they observe the implications of choice, such as using specific components or programming algorithms.

Domain A-Level 3: System outline by block diagram and flow chart structural tree chart. These elements of the portfolio correspond to the third level on the creative thinking scale entitled "thinking strategy," since the pupils must choose a strategy and coordinate among various explanations in their work. They have to decide what level of detail is required and how to present the sequence of action or logical conditions of the system's action. Pupils reported it was easier for them to build a system or write a computer program than to describe their work using systematic flow charts or block diagrams.

Additional portfolio elements at this level include a description of the number of iterations and problem solving.

The machine is controlled by time, but it has a number of problems…the glass gets stuck or goes too fast and thus the mixer does not come down in time…we added a sensor which controls the action of the mixer…the machine started to act as required.

This example shows that the pupils had independently discovered one of the basic principles in the action of control systems: feedback control is preferable to openloop control.

In the portfolios of the wind turbine project, we found the following statements:

After changing our machine many times we succeeded to produce 3 volts. But we knew that we needed more power so we sat together and thought how to improve our machine. One idea to increase speed was to build wider wings. The other idea was to change the mechanical trans mission. We decided to work on both ideas and the result was very good; we produced 6 volts this time.

This example shows reflection upon the whole process and the problem the pupils had in terms of planning and constructing the optimal model.

The following passage appeared in one pupil's portfolio:

Table 2
A Creative Thinking Scale Derived From Portfolio Elements of Technology Projects
Portfolio Domain	Level 1 Awareness of Thinking	Level 2 Observation of Thinking	Level 3 Thinking Strategy	Level 4 Reflection on Thinking
A. System or product design, construction, and evaluation Originality, authenticity, usefulness, unique design (lateral thinking products). Functionality, reliability, accuracy, geometric structure, scientific principle (vertical thinking products). B. Learning and thinking activities Individual and group efforts in project development and problem solving. Processes of teamwork decision making and leadership.	Standard diagram of a system or product taken from available literature. Basic explanation of the model and its construction. Description of the model by means of pictures or sketches. An example of solving a simple problem in planning and construction. Division of tasks in the team. Few examples of using lateral and vertical thinking tools.	Original schematic diagram of system or product designed by the pupil. Detailed drawings of the model. Specification of planning and construction stages including calculations, specifications, or computer programs. Justified examples of choices among a number of alternatives. Information exchange and reciprocal help in the team. Various examples of using thinking tools.	Original system functional block diagrams, structural tree, or flow chart. Description of a number of iterations in the planning and construction of the model. Comparison among and choosing from a number of models. Examples of the contribution of individuals and teamwork to solving complex problems. Evidence of planned use of the thinking tools, openmindedness, and postponing decision (lateral thinking); setting priorities, goals, and criteria (vertical thinking).	Examination of the final product's features, compared to the set goals. Conclusions on successes or difficulties during the development process. Suggestions for improvement in the planning and construction process. Conclusions drawn from the influence of the team's functioning on the completion of the project. Pupils' views on the influence of the team's functioning on thinking and learning processes. Assessment of the selected solution compared to the goals.

One pupil concentrated on programming, drawing preparation, and the functional description, while the other pupil was responsible for constructing the model and the mechanical calculations.

In another project, the pupils wrote:

To maintain a certain working order, we decided that each of the group members will work on a different part of the system. Of course, everyone will also have the right to make suggestions and improve parts on which he isn't working personally.

In this example, the pupils spontaneously used the PMI (Plus, Minus, Interesting) thinking tool, taken from the CoRT program, studied six months earlier.

The mechanism that moves the digger is the most complicated part of the project. We encountered the problem that the track would move very quickly. The positive side of this: everything went faster. The negative: the tractor would fall apart because of the great speed. What was interesting was that the motor had a strong power to move the lever quickly, and thus we decided to change the transmission coming from the motor to the track.

The two highest levels in Domain B on this creative thinking scale manifest themselves in the portfolio as examples of using thinking methods to solve complex problems, how teamwork affected their work, or how decisions were made by the team. Only scant and indirect references to strategy and reflection-thinking processes in the portfolio were, in fact, observed. In the portfolio, the pupils reported retrospectively what had been achieved over weeks and months. The difficulties encountered were mentioned only briefly, although pupils invested much effort and made several attempts prior to reaching the described solutions. Sometimes the pupils encountered periods of crises and despair or, conversely, periods of enthusiasm and working late at night, but this was not generally documented. The pupils directed much more attention to describing the system they constructed and its features (Domain A), since they perceived this as the main task, rather than their reflection on the process itself.

One Project:

A team of three pupils built a machine aimed to squeeze large-size junk into a small piece. The pupils' portfolio was comprised of a description of the machine, the process of design, construction, programming, and improvement, as shown in the following authentic examples.

• Example 1: Machine description.

  The machine contained five subsystems, as seen in Figure 1.

  Pupils' explanation of their system: A loading truck: carries the junk to the conveyor at the reception station. Download is assisted by a pneumatic piston

  A conveyor: moves the junk to the robot station.

  A robotic arm: moves the junk from the conveyor to the squeezing station.

  A squeezing machine: presses the junk by means of four pneumatic pistons.

  An uploading fork: takes the compressed junk out.

• Example 2: System structure presented by a tree chart.

  The pupils' portfolio contained a chart of the machine structure.

  This kind of chart is original and shows pupils' understanding of the system structure and functioning.

• Example 3: Computerized control - A Logo program.

  Part of the system was controlled via the LEGO-Logo interface and programming language. An example of a computer Logo program is demonstrated below.

  To start: key

  Listen-to [sensorA? sensorB]
  If sensorA? [stop]
  If sensorB? [stop]
  If (ascii :key) = 328 [motorA-right]
  If (ascii :key) = 336 [motorA-left]
  If (ascii :key) = 333 [motorB-right]
  If (ascii :key) = 331 [motorB-left]
  If (ascii :key) = 139 [graphic]
  If (ascii :key) = 138 [go-forward]
  If (ascii :key) = 137 [go-backward]
  If (ascii :key) = 136 [auto-program]

  Start read-char
  End

  The program demonstrates structured programming. In every row there is a specific procedure such as "motor-right."

• Example 4: Functional block diagram.

  The pupils used an industrial programmable logic controller (PLC) to control the robotic arm and the electropneumatic valves.

• Example 5: Technical planning.

  An important part of the pupils' work was technical calculations and computerized drawing of the parts and subsystems.

  The above five examples demonstrate different aspects of pupils' work on the technological project. Table 3 provides an assessment of their work using the CTS.

Finally, most of the portfolio elements at the end of 10 ^th grade correspond to Levels 2 and 3, although occasionally Level 4 elements and level elements were present. In a conventional learning situation, De Bono ( 1996 ) expected an average achievement between 1 and 2, and in learning that stresses thinking development, an average achievement between 2 and 3 should be expected. A higher rate of items in Levels 3 and 4 was found when the pupils continued their projects in the 11 ^th and 12 ^th grades. The achievements of the pupils in this study are particularly significant, since they were considered low achievers at the onset of their high school studies.

Figure 1. A machine description: A computerized machine for squeezing junk.

Conclusions

In view of the growing influence of technology on the individual and society, technology education is increasingly becoming an integral component of education for all age groups. This study highlighted the role of portfolio assessment in technology education and its contribution to promoting higher order thinking skills in school graduates. The perception of creative thinking as a synthesis of lateral and vertical thinking emphasizes the cognitive implications of technology education and, in particular, project-based learning. Technology presents many opportunities for promoting imagination and a wealth of ideas and for developing new products to fulfill human needs and realize human aspirations. Dealing with these issues at school may engender growth in lateral thinking skills. However, realization of the ideas, turning imagination into reality, designing and constructing instruments or systems that perform the expected function and fulfil the requirements of quality and reliability necessitates the activation of mathematical-logical tools, knowledge of laws of nature, systematic planning, and consideration of limitations and constraints. These require more vertical thinking. Lateral and vertical thinking complement one another, and technology education via projects constitutes a basis for experience in, and the promotion of, both types of thinking. This perspective necessitates changes in educational perceptions and curriculum planning. For decades education has stressed vertical thinking over lateral thinking, particularly in mathematics and science studies.

Table 3
Examples of Methodological Assessment of Portfolio Elements Through the Creative Thinking Scale (CTS)
Portfolio Element	Rank on CTS	Interpretation
Example 1: Machine description	1	Description of a system by a picture and listing of system components is ranked low on the scale. However, this kind of documentation is recommended.
Example 2: Presenting system's structure by a tree chart	3	An authentic structural tree of a technological system is ranked relatively high on the scale because it demonstrates pupils' ability to utilize system approach and present their unique design intelligently and insightfully.
Example 3: Computerized control-a Logo program	2	Although the given example shows original programming, using subroutines, it provides specific solutions and concrete system's operations.
Example 4: Functional block diagram	3	The original functional block diagram shows how the pupils understand and apply the three basic functions of a feedback control system: measuring, comparing, and correcting the controlled variable.
Example 5: Iterations in technical design	3	A display of systematic iterations of technical design is graded higher than simple use of given formulas.

However, fulfillment of the existing potential in technology education for promoting higher order competencies does not happen spontaneously. This study shows that introducing "thinking lessons" into technology curricula helps to develop an awareness of thinking among the teachers and pupils and gives them new tools for observing, thinking, and reflecting on thinking. Emphasizing the promotion of thinking processes within technology education should also express itself in assessment methods at school. Traditionally, teachers and pupils engaged in technological projects directed most of their efforts to completing the task and documenting the final product. While preparing a portfolio, pupils are encouraged to express the wealth of means they used, modes of action adopted, and the processes that the pupil and team go through during designing, constructing, and improving the technological system. Pupils do not tend to keep records of their work, document their experiments, or report their difficulties. Thus, it is essential for the teacher to discuss and cooperate with the pupils in determining the criteria for assessing their work.

The suggested assessment scale of creative thinking can help educators strive for a gradual development of higher order thinking skills in two main areas. The first is choosing the project topics for the pupils, their complexity, level of expectation for originality and creativity on the one hand, and the extent of using mathematicallogical and scientific thinking on the other hand. The second area of gradual progress is developing learning and thinking processes in class, problem solving, teamwork, and reflection on thinking. Thus, learning through technology projects based on portfolio assessment and directed towards a systematic development of vertical and lateral thinking may promote teaching and learning that assist the school graduate's successful integration into a dynamic and changing world.

Dr. Moshe Barak is a senior lecturer in the Department of Education in Technology and Science Technion, Israel Institute of Technology in Haifa, Israel. He is a Member-At-Large of Epsilon Pi Tau.

Dr. Yaron Doppelt is a research assistant in the Department of Education in Technology and Science Technicion, Israel Institute of Technology in Haifa, Israel.

References

Barak , M., & Doppelt, Y. (1998, September). Promoting creative thinking within technology education . Paper presented at the International Workshop for Scholars in Technology Education, WOCATE, George Washington University, Washington, DC.

Barak , M., & Maymon, T. (1998). Aspects of teamwork observed in a technological task in junior high schools. Journal of Technology Education , 9 (2), 317.

Barak , M., Maymon, T., & Harel, G. (1998). Teamwork in modern organizations: Implications for technology education. International Journal of Technology and Design Education , 3 (7), 2133.

Berenson , S. B., & Carter, G. S. (1995). Changing assessment practices in science and mathematics. School Science and Mathematics , 95 (4), 182185.

Brooks , J. G., & Brooks, M. G. (1993). In search of understanding: The case for constructivist classrooms . Alexandria, VA: Association for Supervision and Curriculum Development.

De Bono , E. (1970). Lateral thinking . New York: Harper & Row.

De Bono , E. (1986). The practical teaching of thinking using the CoRT method. Special Service in Schools , 3 (1-2), 33-47.

De Bono , E. (1994). De Bono's thinking course (Rev. ed.). New York: Facts On File.

De Bono , E. (1996). Master workshop of de Bono's thinking course . Jerusalem: Branco Weiss Institute for the Development of Thinking.

Denton , H. (1994). The role of group/team work in design and technology: Some possibilities and problems. In F. Banks (Ed.), Teaching technology (pp. 145-151). London: Routledge.

Ennis , R. H. (1989). Critical thinking and subject specificity: Clarification and needed research. Educational Researcher , 18 (3), 4-10.

Glaser , R. (1993). Education and thinking: The role of knowledge. In R. McCormick, P. Murphy, & M. Harrison (Eds.), Teaching and learning technology (pp. 91111). Wokingham, England: Addison-Wesley.

Gredler , E. M. (1995). Implications of portfolio assessment for program evaluation. Studies in Educational Evaluation , 21 (4), 431-437 .

Papert , S. (1980). Mindstorms, children, computers and powerful ideas. New York: Basic Books.

Resnick , M., & Ocko, S. (1991). Learning through and about design. In I. Harel & S. Papert (Eds.), Constructionism (pp. 141-150). Norwood, NJ: Ablex.

Waks , S. (1997). Lateral thinking and technology education. Journal of Science Education & Technology , 6 (4), 245-255.

Wolf , D. (1989). Portfolio assessment: Sampling pupil work. Educational Leadership , 45 (4), 35-39.

TS