The GRID _C Project: Developing Students' Thinking Skills in a Data-Rich Environment

V. William DeLuca and Nasim Lari

The purpose of this study was to determine the impact of using renewable energy data, obtained from a comprehensive data acquisition system, on improving students' learning and developing their higher-order learning skills. This study used renewable energy data available through a data acquisition system installed and tested by the G reen R esearch for I ncorporating D ata in the C lassroom (GRID _C ) project. The purpose of GRID _C is to develop curriculum to teach science, technology, engineering, and mathematics (STEM) concepts using data collected from renewable energy technologies at the North Carolina Solar House (NC Solar House), located on the campus of North Carolina State University (NC State). This project enhances instruction and improves learning while addressing a highly relevant social issue-renewable energy. The GRID _C project gives professors, instructors, and their students the opportunity to study and evaluate the value of renewable energy systems through the use of real-time renewable energy data.

Throughout the years, researchers have shown the value of using real world data to enhance instruction in mathematics, science, and social studies ( Drier, Dawson, & Garofalo, 1999 ; Gordin, Polman, & Pea, 1994 ). Climate and environmental databases, such as the Quantitative Environmental Learning Project website ( Langkamp & Hull, 2002 ), are available to educators to support instruction. Curricula that are based on the performance data of renewable energy technologies provide students with valuable knowledge and skills that can be used for professional growth and decision making. Data-driven decision making is a critical skill used in engineering and education ( Diane, Johnson, & Mistry, 2004 ; ">Mandinach, Honey, Light, Heinze, & Rivas, 2005 ), and as technological and social systems become more complex, the aptitude for data-driven decision making becomes even more critical.

In order to develop students' higher order thinking skills in the context of a data-rich learning environment, the researchers considered that students must understand factual, conceptual, and procedural knowledge; apply their knowledge to learn by doing; and then reflect on the process that led to the solution ( Bransford, Brown, & Cocking, 2000 ; Anderson, Krathwohl, Airasian, Cruikshank, Mayer, Raths, & Wittrock, 2001 ).

Factual and conceptual knowledge includes an understanding of the systems, subsystems, and components of the technology being studied. In other words, what is the basic design, how does it function, and what are the expected outputs? This knowledge, gained through lecture, readings, or personal research, forms the basic understanding needed before proceeding with the design and problem-solving process ( Lumsdaine, Shelnutt, & Lumsdaine, 1999 ).

Procedural knowledge includes an understanding of the engineering design and/or problem-solving processes that lead to innovative solutions. The processes and strategies used to solve problems and make decisions must be understood ( Schweiger, 2003 ; Woods, 2000 ). These processes include equations used to calculate system performance, transform data, and make predictions and problem-solving processes, such as troubleshooting and project management, that help engineers, designers, and technicians reach solutions.

However, in order to develop higher order thinking skills, students must have the opportunity to apply their content and process knowledge ( Bonanno, 2004 ; Moriyama, Satou, & King, 2002 ; DeLuca, 1992 ) and learn from errors ( Mathan & Koedinger, 2005 ). Performance data from the variety of renewable energy systems proposed for this project provide opportunities for students and teachers to analyze and evaluate system variables within the context of their disciplines.

Bransford, Brown, and Cocking (2000) discuss the importance of making students' thinking visible. The nature of the data collected and used in this study supports the development of thinking skills and allowing students to reflect on their thought process. Students have the opportunity to analyze, evaluate, and predict while applying concepts in a variety of situations. Reflection also includes looking back on the processes that led to decisions ( Quintana, Zhang, & Krajcik, 2005 ). The GRID _C project team and participating professors and instructors developed instructional units grounded in these concepts while incorporating the use of the renewable energy data collected through GRID _C resources into the units.

The core of the GRID _C data acquisition system is located at the NC Solar House and gathers renewable energy data from the house and other units (e.g., garage and research annex) on the grounds. The NC Solar House was first opened to the public in 1981 and is one of the most visible/well-known and visited solar buildings in the United States today.

The monitoring system records meteorological data (i.e., irradiance, ambient and module temperature, wind speed and direction, module temperature, relative humidity, rain gauge, barometric pressure), photovoltaic data (i.e., AC/DC power, current, voltage, and energy, panel temperature), hot water data (i.e., flow rate, in/out temperate, energy), and hydrogen fuel cell data (i.e., in/out power, current and voltage, energy).

Data from these systems is collected and uploaded to an online data acquisition system, where daily, monthly, and yearly information may be viewed graphically or downloaded in a spreadsheet format. The aggregated GRID _C data, available on the project's website (www.GRID _C .net), is used by professors and instructors to develop instructional units to be implemented in various undergraduate and graduate level courses.

Method

Participants

The sample consisted of 118 individuals. Student data was collected from a variety of undergraduate and graduate courses at NC State and a course at Pitt Community College. The research team gathered student data through each course's professor or instructor and assigned a number to each student, which was subsequently used in data analysis. This allowed for full student confidentiality. Students were selected based on their enrollment in engineering, STEM education, or construction courses that addressed topics in renewable energy. Specifically, the students were enrolled in one of the following courses:

• Construction Technology (TED 221 - Undergraduate Course - Department of Mathematics, Science, and Technology Education, College of Education, NC State): This course provides an overview of residential and commercial structures and their construction. Students use drawings and models completed in a laboratory environment to simulate construction methods.
• Current Trends in Technical Graphics Education (TED 532 - Graduate Course - Department of Mathematics, Science, and Technology Education, College of Education, NC State): This graduate level course discusses the current trends in technology, techniques, and theories relating to technical graphics education. The course is centered on assigned readings and student-researched presentations on topical subjects; readings are drawn from journals and texts, on-line databases and articles, and current news media sources.
• Instructional Science Materials (EMS 373 - Undergraduate Course - Department of Mathematics, Science, and Technology Education, College of Education, NC State): This course teaches students to develop and select teaching materials that reflect concepts of content, with an emphasis on middle and secondary school science. The course provides an overview of experimental and laboratory approaches, including the use of microcomputer and video technologies.
• Design of Solar Heating Systems (MAE 421 - Undergraduate Course - Department of Mechanical and Aerospace Engineering, College of Engineering, NC State): This course involves the analysis and design of active and passive solar thermal systems for residential and small commercial buildings. The course provides an overview of solar insulation, flat plate collectors, thermal storage, heat exchanges, controls, performance calculations, suncharts, and photovoltaics.
• Selected Topics in Energy Efficient Building and Design (CST 293 - Construction and Industrial Technology Division, Pitt Community College): This course familiarizes students with building principles that form the basis of energy efficient building and design. Students will be exposed to passive solar design, thermal analysis, indoor air quality, and studying the house as a system.

Given the mix of community college students and university students enrolled in lower and upper level courses, subjects varied in age and class rank. The instructional modules developed were reviewed to ensure that they broadened opportunities and enabled the equitable participation of women, nontraditional age groups, underrepresented minorities, and persons with disabilities.

North Carolina's Community College System has, throughout its history, served nontraditional age groups through its successful outreach to adults seeking education, training, and retraining for the workforce, including basic skills and literacy education, as well as occupational and pre-baccalaureate programs. The 58 North Carolina community colleges reported over 810,000 curriculum and continuing education student enrollments for the 2007-2008 academic year. Among the nearly 300,000 curriculum student enrollees, females outnumbered males approximately 2 to 1 ( NCCCS, 2008a ). Racial diversity is also noteworthy: 24.9% of the student population is black, 1.5% American Indian, 2.1% Asian, and 3.6% Latino. At Pitt Community College, with over 9,000 curriculum students enrolled, approximately 31% are black, 0.5% American Indian, 1.1% Asian, and 2.1% Latino ( NCCCS, 2008b ).

Instruments

Each instructional unit was developed and implemented by the professor or instructor assigned to the course. The GRID _C project team provided individual training sessions for the professors and instructors involved in curriculum development and design. Each session included a detailed description of the project's curriculum design goals and involved discussions on factual, conceptual, and procedural knowledge; knowledge application; and student reflection. Handouts were provided on methodology, instrumentation, procedure, and assessing learning outcomes. The sessions gave professors and instructors a good opportunity to ask questions. Instructional units were designed to use the GRID _C renewable data, presenting students with problems pertaining to renewable energy issues. Students were exposed to the website and required to download and manipulate data to answer questions.

To determine if the desired learning objectives were achieved, the following research method was employed. Each unit began with a pretest consisting of general renewable energy knowledge items and a metacognitive inventory. With the introduction of each unit, students were instructed on the unit's learning objectives and required activities. During the unit, students kept a journal. Upon completion of each unit, the posttest knowledge questions and the metacognitive inventory were administered. Data collected with pre-/post-tests, journals, forums, and activities requiring knowledge application were archived for statistical analysis and reporting.

Thus, three instruments were designed and used to measure knowledge, application, and reflection. Knowledge gained was measured through pre- and post-test analysis. Alternative versions of a multiple choice test were developed by a panel of content experts. Each test consisted of a set of core questions (i.e., common questions across disciplines) as well as discipline-specific questions.

Application of knowledge gained in the units developed was measured through certain activities, and rubrics were developed to measure student performance on assigned activities. Once again, a panel of content experts was used to develop the rubrics, and a separate panel was used to validate the measure. Post-analysis was done to determine reliability and to ensure continuous improvement. Finally, to measure reflection, quantitative and qualitative analyses were conducted on student journals.

Students' awareness of their cognitive processes as they approach and solve problems was evaluated using the metacognitive inventory. The Metacognitive Inventory (MI) was developed using 6 items from the Problem-Solving Inventory (PSI) and 20 items from the State Metacognitive Inventory (SMI), with slight modifications ( Heppner, 1994 ; O'Neil & Abedi, 1996 ). This inventory was designed such that it may be used in varied situations in which the developed curricula are implemented. The items cover the six categories of approach-avoidance, awareness, cognitive strategy, confidence, planning, and self-checking . The Appendix provides a list of items within each category; items derived from the PSI are marked accordingly. The PSI is a 35-item test, which uses the Likert scale response options to assess individuals' awareness of their style of solving life problems such as relationship conflicts and career choices ( Heppner, 1994 ). The SMI, a 20-item test which also makes use of Likert scale response options, is used to assess the extent to which students are aware of thinking skills they use to complete tests ( O'Neil & Abedi, 1996 ).

Results

The first unit was implemented in the fall semester of 2008. Since then, units have been implemented and data gathered from five other classes, providing 118 observations. Several observations were deleted for certain analyses; these deletions are detailed on the next page.

Renewable Energy General Knowledge Outcomes

In one course, the instructor failed to administer the renewable energy general knowledge posttest questions, leaving researchers with a base of 112 observations. Table 1 provides descriptive statistics for the renewable energy general knowledge pre- and post-tests. The tests were graded out of 12 possible points.

Table 1
Descriptive Statistics for the Renewable Energy General Knowledge Pre- and Post-Tests

	Mean	Std Deviation	Minimum	Maximum	Median
General Knowledge Pretest	6.33	2.06	1.71	11	6.6
General Knowledge Posttest	8.25	1.85	2.4	11.4	8.57

Table 2 presents the results of the Shapiro-Wilk test for normality. The null hypothesis is that the data are normally distributed.

Table 2
General Knowledge - Normality Assumption Checks (Results of Shapiro-Wilk Test)

	Statistic (W)	df	Sig.
Difference in General Knowledge Pre- and Post-Tests	0.986	97	0.391

The null hypothesis was not rejected, and the normality assumption was satisfied. A paired t-test is used for the analysis. The results indicate significant gains in posttest renewable energy general knowledge scores (t (96) = 9.41, p < 0.001).

Metacognitive Inventory Outcomes

Table 3 (next page) provides descriptive statistics for the MI pre- and post-tests. Administration error resulted in the loss of 50 of the 118 observations in the analysis of the MI and its individual items.

Table 3
Descriptive Statistics for the Metacognitive Inventory (MI) Pre- and Post-Tests

	Mean	Std Deviation	Minimum	Maximum	Median
MI Pretest	3.98	0.41	2.85	4.96	3.96
MI Posttest	4.07	0.45	2.92	5	4.04

Table 4 presents the results of the Shapiro-Wilk test for normality. The null hypothesis was that the data were normally distributed.

Table 4
MI - Normality Assumption Checks (Results of Shapiro-Wilk Test)

	Statistic (W)	df	Sig.
Difference in General Knowledge Pre- and Post-Tests	0.987	59	0.784

The null hypothesis was not rejected, and the normality assumption was satisfied. A paired t-test was used for the analysis. The results indicated significant gains in metacognitive performance, as measured by the MI (t (58) = 2.19, p < 0.001).

A Wilcoxon signed-rank test was performed on each of the 26 MI items. The MI made use of 5-point Likert scale response options. Six items showed significant gains in student perceptions, primarily in items from the category of "self-checking." Table 5 provides descriptive statistics for the items found significant under the category of "self-checking."

Table 5
Descriptive Statistics for Significant "Self-Checking" Items

Item		Mean	Std. Deviation	Min	Max	Median
After I solve a problem, I analyze what went right or what went wrong.	Pre	3.95	0.76	1	5	4
	Post	4.17	0.68	2	5	4
I almost always know how much of an assignment I have left to complete.	Pre	3.86	0.66	2	5	4
	Post	4.06	0.77	3	5	4
I check my accuracy as I progress through assignments.	Pre	3.69	0.78	2	5	4
	Post	3.96	0.76	1	5	4

Table 6 presents the results of Wilcoxon signed-rank tests. In the category of "self-checking", the items "After I solve a problem, I analyze what went right or what went wrong," "I almost always know how much of an assignment I have left to complete," and "I check my accuracy as I progress through assignments" showed significant gains from pre- to post-tests.

Table 6
Wilcoxon Signed-Rank Test Results for "Self-Checking" Items

Item	Signed Rank
Where * indicates significance at p < 0.05 and ** indicates significance at p < 0.01.
After I solve a problem, I analyze what went right or what went wrong.	76.5**
I almost always know how much of an assignment I have left to complete.	99.0*
I check my accuracy as I progress through assignments.	122.0**

Table 7 provides descriptive statistics for the items found significant under the categories of "confidence," "cognitive strategy," and "awareness."

Table 7
Descriptive Statistics for Significant "Confidence," "Cognitive Strategy," & "Awareness" Items

Item		Mean	Std. Deviation	Min	Max	Median
I am usually able to think up creative or effective alternatives to solve a problem.	Pre	4	0.71	2	5	4
	Post	4.20	0.67	3	5	4
I think through the meaning of assignments before I begin.	Pre	3.5	0.93	2	5	4
	Post	3.86	0.81	2	5	4
I check my accuracy as I progress through assignments.	Pre	3.69	0.69	2	5	4
	Post	4	0.73	2	5	4

Table 8 (next page) presents the results of Wilcoxon signed-rank tests. The following items from the category of "awareness" also indicated significant gains: "I am usually able to think up creative or effective alternatives to solve a problem" from the category of "confidence," "I think through the meaning of assignments before I begin" from the category of "cognitive strategy," and "I am aware of which thinking techniques and strategies to use and when to use them."

Table 8
Wilcoxon Signed-Rank Test Results for "Confidence," "Cognitive Strategy," & "Awareness" Items

Item	Signed Rank
Where * indicates significance at p < 0.05, ** indicates significance at p < 0.01 and ***indicates significance at p < 0.001.
I am usually able to think up creative or effective alternatives to solve a problem.	82.0**
I think through the meaning of assignments before I begin.	147.0**
I am aware of which thinking techniques and strategies to use and when to use them.	182.0***

Table 9 provides descriptive statistics for the significant item under "awareness."

Table 9
Descriptive Statistics for "Awareness" Item

Item		Mean	Std. Deviation	Min	Max	Median
I am aware of the need to plan my course of action.	Pre	4.39	0.66	2	5	4
	Post	4.13	0.70	3	5	4

Surprisingly, the following item from the category of awareness showed a decrease in perceived frequency of use: "I am aware of the need to plan my course of action."

Table 10 presents the results of the Wilcoxon signed-rank test.

Table 10
Wilcoxon Signed-Rank Test Results for "Awareness" Item

Item	Signed Rank
Where * indicates significance at p < 0.05
I am aware of the need to plan my course of action.	(-) 96.0*

Reliability of Metacognitive Inventory Items

The MI consists of six categories. The categories of "awareness," "cognitive strategy," "planning," and "self-checking" consist of six items each, and the categories of "approach/avoidance" and "problem-solving confidence" consist of three items each. Cronbach's alpha was used in subsequent analyses to estimate the internal consistency for each of the categories. Alpha coefficients for the categories of "awareness," "cognitive strategy," "planning," and "self-checking" indicate a good scale (α ≥ 0.75). Cronbach's alpha decreases as the number of items in the category decreases, which may explain the lower alpha values for the categories of "problem-solving confidence" and "approach/avoidance style," 0.57 and 0.63, respectively. However, given the smaller number of items in these categories, alpha for "approach/avoidance" still proves adequate.

Discussion

The present analyses show significant gains in posttest renewable energy general knowledge scores. This indicates that the use of real-time renewable energy data was effective in instruction, providing students with valuable knowledge and skills that can be used for decision making. The results confirm the claims of previous studies that using real world data enhances instruction in various fields.

The researchers also found significant gains in metacognitive performance, as measured by the metacognitive inventory. The metcognitive inventory makes the thinking process visible, thereby allowing researchers to see the significant increase in students' reflections on their thought processes. This outcome is of particular importance, as research on technological problem solving, critical thinking, novice/expert performance, and metacognition has shown that students must understand factual, conceptual, and procedural knowledge; apply their knowledge to learn by doing; and then reflect on the process that led to the solution ( Bransford, Brown, & Cocking, 2000 ; Anderson, Krathwohl, Airasian, Cruikshank, Mayer, Raths, & Wittrock, 2001 ).

Detailed analyses of the MI showed significant gains for certain items. The majority of gains were in the category of "self-checking." Students were found to check the accuracy of their work as they progressed through assignments and reflect on problems, analyzing what went right or what went wrong. Further, they developed a better understanding of how much of an assignment they had left to complete.

Significant gains were found in other MI categories as well. Students reported a greater ability to think up creative or effective alternatives to solve a problem, which showed a significant increase in the area of "confidence." They also reported thinking through the meaning of assignments before beginning, showing development of a "cognitive strategy." Finally, in the category of "awareness," students reported becoming more aware of which thinking techniques and strategies to use and when to use them. However, within the same category of "awareness," students showed a decrease in awareness of their need to plan a course of action. Collection of more data will allow for a deeper evaluation of these statements and explorations of how general knowledge and MI outcomes may differ among various demographic groups.

To this end, GRID _C researchers are actively recruiting professors and instructors from various NC State departments, local colleges and universities, and K-12 teachers to help develop and implement GRID _C curricula. In an effort to obtain quality data with a maximum number of usable observations, steps have been taken to ensure that professors and instructors are aware of the importance and value of proper data collection.

In addition to gathering more student data, the future brings new opportunities for collaboration with various companies within the energy and transportation industries. Such collaboration will expand GRID _C 's data acquisition system to include transportation data, as well as wind energy data. Broadening the data acquisition system will further enhance students' opportunities to conduct comparative analysis and aggregate data for decision making.

Finally, refinements to the curriculum will be introduced to demonstrate the effectiveness of an integrated, data-rich curriculum to teach STEM concepts and develop metacognitive skills. Through the various courses offered among the partnering institutions, this curriculum will reach a sizeable and diverse population of science, engineering, and technology students, better enabling students to learn about renewable energy technologies by understanding the variables and variable relationships that are controlled by the technologies' design and function. Additionally, students will learn how the disciplines of science and mathematics are used in the design and optimization of systems. As the results suggest, the GRID _C research project has national implications for improving STEM education and will provide a platform for continued research and development of instructional materials that improve STEM education.

---

V. William DeLuca ( william_deluca@ncsu.edu ) is an Associate Professor in the Department of Mathematics, Science, and Technology Education at North Carolina State University. Nasim Lari ( nlari@ncsu.edu ) is a Research Consultant in the Department of Mathematics, Science, and Technology Education at North Carolina State University.

References

Appendix

Awareness

I am aware of the need to plan my course of action.
I am aware of my ongoing thinking processes.
I am aware of my own thinking.
I am aware of my trying to understand assignments before I attempt to solve them.
I am aware of which thinking techniques and strategies to use and when to use them.

Cognitive Strategy

I think through the meaning of assignments before I begin.
I use multiple thinking techniques or strategies to complete an assignment
I attempt to discover the main ideas in assignments.
I select and organize relevant information to complete assignments.
I ask myself how the assignments are related to what I already know.

Planning

I try to determine what assignments require.
I make sure I understand just what has to be done and how to do it.
I determine how to solve assignments.
I try to understand the goals of assignments before I attempt to answer or solve.
I try to understand assignments before I attempt to solve them.

Self-Checking

I almost always know how much of an assignment I have left to complete.
I keep track of my progress and, if necessary, change my techniques or strategies.
I check my work while I am doing it.
I check my accuracy as I progress through assignments.
I correct my errors.

Problem-Solving Confidence

I trust my ability to solve new and difficult problems. (PSI)
I am usually able to think up creative or effective alternatives to solve a problem. (PSI)
When I become aware of a problem, one of the first things I do is to try to find out exactly what the problem is. (PSI)

Approach/Avoidance Style

After I solve a problem, I analyze what went right or what went wrong. (PSI)
When confronted with a problem, I stop and think about it before deciding on a next step. (PSI)
In trying to solve a problem, one strategy I often use is to think of past problems that have been similar. (PSI)