Current Editor: Chris Merrill, email@example.com
Previous Editors: Mark Sanders 1989-1997; James LaPorte: 1997-2010
As an open access journal, the JTE does not charge fees for authors to publish or readers to access.
The Development and Validation of a Test of Industrial Technological Literacy Michael A. Hayden It is often stated that technology af- fects every person on this planet, is growing exponentially, and is becoming increasingly complicated. With technology being such a force in our lives, it seems logical that we should be knowledgeable about it. This know- ledge of technology has been labeled-- technological literacy (TL). Several reports on education have addressed the TL issue and have all called for an increase in the level of TL exhibited by our students. Maley (1985) noted that, "in 1984 alone, ten major studies of education were reported...each one calling for changes in school to prepare our students to live in a technological society" (p. 16). Since 1984 an increasing amount of research concerning TL has been conducted. A large number of educators and special- ists in several disciplines are concerned about the level of TL displayed by the Ameri- can public. They voice an opinion similar to Ley (1987), namely, "the level of technolog- ical literacy to which educators and others are able to bring the general population will determine the future world in which humankind will exist" (p. 7). A set of very pertinent questions has recently been on the minds of many concerned leaders. These questions include: 1. As a public, how technologically literate are we? 2. How do we become technologically liter- ate, or become more so? To answer these questions we must first answer such questions as: 1. What is technological literacy? 2. Does it really exist, i.e., is it worthy of study by itself or is it part of some- thing else? 3. Assuming TL exits, can we measure it with utility? If TL is a viable concept and can be measured reliably, then studies can be con- ducted to determine the most advantageous ed- ucational environment for the attainment of such a characteristic. A test of TL with sufficient validity could be used to identify and diagnose those students in need of greater technological knowledge, as well as evaluate the effectiveness of programs teach- ing the construct. PURPOSE OF THE STUDY The purpose of the study was to investi- gate the validity of the construct of TL. The definition of TL will not be expounded upon here since it has been exhaustively cov- ered by this and many other authors (see Hayden, 1989b). In this study the following definition of technological literacy was used: technological literacy is having the knowledge and ability to select, properly ap- ply, then monitor and evaluate appropriate technology given the context. Following are general questions the study investigated. 1. Does the empirical model (an instrument measuring TL) exhibit satisfactory psychometric properties? 2. Does the empirical model exhibit multiple dimensions? 3. How do the possible correlates (independ- ent variables) of general achievement, grade level, gender, parental contact with technology, and prior industrial course exposure affect TL? THEORY OF TECHNOLOGICAL LITERACY For the purpose of this study, technol- ogy was defined as: the process by which hu- mans utilize available resources to extend their potential or alter their environment (Hayden, 1989b). Based on this definition it was theorized that technology is a learned phenomenon. Hence, being literate about technology should also be a learned phenome- non. Technological literacy is not skill in directly applying specific technology, i.e., technical literacy. Technological literacy for the purpose of this research was viewed as general knowledge, abilities and behaviors concerning technology. Furthermore, techno- logical literacy was not viewed as being re- stricted to general skills in applying technology, i.e. technological operacy. Therefore, TL was assumed to be more closely related to achievement than aptitude. Being related to general achievement, a TL instrument should perform psychometrically similar to other measures of general achieve- ment. If TL is in part related to achieve- ment, a person's standing on the attribute should increase with age due to developmental factors. Also, a person's score on a TL in- strument should increase with additional learning and exposure related to the attri- bute. Additional exposure could come from several sources, e.g., parental/student interaction, school curriculum and hobbies. RESEARCH QUESTIONS To investigate the questions of the study, 29 hypotheses were tested, the major- ity of which related to eight dependent vari- ables (total test and three subtests for male and female). A synopsis of the research questions follow. 1. Is the instrument unidimensional? 2. To what extent do the independent vari- ables account for unique variance in in- strument score. 3. Can the independent variables be used to predict scores on the instrument? 4. Does the instrument behave psychometrically in a similar fashion to other measures of general achievement? 5. Does the score on the instrument have a linear relationship with grade level? 6. Do males and females score differently? 7. Is there any interaction between gender and grade level on the instrument score? METHODOLOGY INSTRUMENTATION The instrument was developed from a rig- orous test plan which included a detailed ra- tionale and definition of the domain including limitations, delimitations, the target population, uses and constraints of the test. A blueprint was created that de- tailed the format and scoring procedures. The test plan also included generation of items and revising algorithms and adminis- tration instructions. Clearly, technological literacy is a broad term. Many authorities maintain that it encompasses operacy, i.e, what we can do with the technology, in addition to typical literacy concepts. Many (see Fleming, 1989) see technological literacy as a type of empowerment enabling us to do a very broad range of things. Additionally, general lit- eracy concerning technology would cover all technology, e. g., agricultural, medical mil- itary, etc. Lastly, literacy about technol- ogy would encompass affective, cognitive and psychomotor domains. The time and monetary limitations placed on this project necessitated a tightly fo- cused test blueprint. The following delimi- tations and their accompanying rationale were used to focus the instrument. 1. Focus on the industrial strata because it was assumed that if industrial technolog- ical literacy was a valid concept then the more encompassing concept of TL was also valid; this strata was the research- er's area of expertise; prior research had most clearly explained this strata; and it was perceived that an instrument focused on industrial technology would have the most future utility for use in industrial programs. 2. Maintain a primary focus on the cognitive domain because it readily lends itself to item writing, test administration, scor- ing and data analysis techniques which are relatively unambiguous and can be used with a large number of items and subjects in a timely fashion. 3. Delimit the content of the items to gen- eral principles of technology or recent technological innovations/impacts because it was assumed that if an individual un- derstood and was literate about these then they would know or could easily learn about other time periods. The instrument was titled "Industrial Techno- logical Knowledge" (ITK). Trained item writ- ers developed and refined the items in the instrument. The writers were all faculty or doctoral students of the Industrial Education and Technology or Research and Evaluation De- partments at Iowa State University. The mean number of years of industrial/consulting and teaching experience of the writers was nine and four years respectively. The items underwent several iterations of qualitative and statistical selection and revision. There were several item tryouts in addition to a pilot study. Parental contact with technology was measured dichotomously for father and mother. Students responded to the question, "Does your parent through work, major home respon- sibilities or hobbies work with tools and/or machines?" Industrial course exposure was measured by recording the number of semesters of industrial/technical/vocational (I/T/V) courses taken. This included the current se- mester. General achievement was measured by the students' Iowa Test of Educational Develop- ment (ITED) subscores of Natural Science (NS), Quantitative (Q), Reading Total (RT) and Social Science (SS). DATA ANALYSIS Exhaustive analyses were conducted. Only the results will be summarized in this paper. For most analyses, cases with missing data were deleted case-wise, i.e., only sub- jects for whom complete information on the independent variables was obtained were used in the analyses. All correlational and t- tests were two-tailed and performed at the .05 level. Several post hoc analyses were conducted. Many of these were outside of the realm of the study and were of small sample size. However, they play a significant role in interpretation of the results and recomm- endations for future inquiry. Primary analysis procedures included factor analysis to determine the dimensional- ity of the instrument and correlational, analysis of variance and regression proce- dures to investigate variance and predict- ability questions. The instrument's characteristics were compared with other in- struments measuring general achievement to assess its psychometric properties. Visual inspection of plotted cell means and variable correlations were also used to assess statis- tical procedure assumptions such as linear relationships. Descriptive statistics of the sample, the ITK scores, and the individual item char- acteristics were generated by a variety of techniques including one parameter logistic scaling, analysis of variance, and correlational methods. POPULATION AND SAMPLE The population of this study was high school students in the state of Iowa. The sample was comprised of volunteering school buildings. Within the schools, intact classes were targeted for participation in the study. The classes selected were as het- erogeneous as possible. Most classes se- lected were required courses. Approximately 25% of the subjects were from each of the grade levels 9 - 12. Within grade levels, males and females were represented approxi- mately equally. Virtually all students had some I/T/V course exposure. Industrial Tech- nological Knowledge test scores were col- lected from 826 subjects. INTERPRETATION OF RESULTS Only those results most significant to interpretation and recommendations will be discussed. For detailed hypotheses and re- sults of all statistical tests see the ori- ginal study (Hayden, 1989a). PSYCHOMETRIC PROPERTIES INSTRUMENT STATISTICS. The internal consistency reliability of the total test was .83. The internal consistency of the ITED subtests range from .86 to .96 (The Univer- sity of Iowa, 1987). Application of the Spearman-Brown formula indicated that if the ITK instrument were the same length as the ITED composite instrument, the ITK's esti- mated reliability would be .975. The average interitem correlation of the ITK was rather low, being only .10. Average item/total correlations were also small, be- ing .28. It is possible that these corre- lations could be increased by revising the items and increasing sample size. If the av- erage interitem correlation could be in- creased to .15 (the average interitem correlation of subtest #3), the reliability estimate for the 45 item instrument would be .89, uncorrected. There was no floor or ceiling effect on the ITK instrument. The distribution of scores was nearly normal with an average item mean of .5. This allowed for maximum vari- ance. However, due to the potential for guessing, a slightly easier test would have given more information about test takers. The 2.94 standard error of measurement of the ITK instrument is proportionately large compared to the ITED composite. The standard error of measurement for the 361 item ITED composite is approximately 1.2. It is possible that further item revision could decrease the ITK's standard error of measure- ment. Table 1 summarizes the instrument's psychometric properties. TABLE 1 INSTRUMENT STATISTICS - MALE AND FEMALE COM- BINED --------------------------------------------- Total test H.S. Number of items 45 Cronbach's alpha a .83 Minimum score 2 Maximum score 44 Mean 22.23 Median 23 Mode 24 Standard deviation 7.12 Standard error of measurement 2.94 Average item/total score correlation .28 Skewness of distribution -0.13 Kurtosis of distribution -0.55 Average inter-item correlation .10 Average item mean .50 --------------------------------------------- a Calculated using uncorrected point-biserial correlation. The data collected from the main phase of this study is equivalent to the main field testing phase of a research and development project. It is the view of this researcher that the ITK instrument exhibits satisfactory psychometric properties to be worthy of re- vision and advancement to the operational field testing stage of development. ITEM STATISTICS. Among items and be- tween gender there was much fluctuation of item means. Females do better on some items, while males do better on others. These fluc- tuations are much more pronounced for items which are more difficult. Females tended to perform better on items requiring reading comprehension. There was no consistent pat- tern of item means between males and females except that males scored higher on most items. DIMENSIONALITY. Three factors or sub- tests were extracted from the ITK instrument. Factor #1 primarily contained items pertain- ing to the human adaptive systems. The two highest loading items on this factor and their loadings were: #18 Why can computers lead to greater pro- ductivity in industry? REDUCTION OF HUMAN ERROR, REDUCTION OF COST, GREATER SPEED, AND INCREASED ACCURACY. Loading of .81. #29 The average worker has to be retrained because NEW METHODS OF DOING THINGS HAVE BEEN DEVELOPED. Loading of .80. Factor #2 primarily contained items per- taining to the application of other disci- plines (most notably math and science) to the solution of technological problems. The two highest loading items on this factor and their loadings were: #15 One of the most promising uses of microbes is for CLEANING THE ENVIRON- MENT. Loading of .69. #31 Cooking in a microwave oven is done by EXCITING THE ATOMS IN THE FOOD. Load- ing of .64. Factor #3 primarily contained items dealing with the interpretation of written and graphical material having a technological theme. The two highest loading items on this factor and their loadings were: #37 Technology can best be defined as PEO- PLE USING TOOLS, RESOURCES, AND PROC- ESSES TO SOLVE PROBLEMS OR EXTEND THEIR CAPABILITIES. .76. #45 Consider this situation. The design, manufacturing, marketing, and manage- ment of product XX was performed by hundreds of individuals. After using product XX, many consumers were in- jured. In a latter court case, injured person John Doe testified that he thought product XX was unsafe even be- fore he used it. Several employees that helped make and sell the product also testified that they had always thought XX was dangerous. The Judge asked the injured person why he went ahead and used the product. He also asked the employees why they went ahead and made and sold the product even though they thought it was unsafe. The situation presented above could be ac- counted for because INDIVIDUALS, WHEN GIVEN DIRECTIONS BY PEOPLE IN AUTHOR- ITY, OFTEN DO THINGS WITHOUT THINKING ABOUT THE POSSIBLE OUTCOMES. Loading of .66. PREDICTING SCORES It was found that the score on the ITK instrument could be predicted. Table 2 shows the regression equations for males and fe- males. Table 3 further summarizes the pre- diction findings. TABLE 2 REGRESSION EQUATION AND SUMMARY STATISTICS FOR PREDICTION OF ITK TOTAL SCORE ***(Available as a graphic only)*** TABLE 3 SIGNIFICANTLY PREDICTING VARIABLES ------------------------------------------------------ Total #1 #2 #3 Grade M,F F F Gender x a x x x Father M M M Mother Father/mother combined M M Indust./tech./vocat. course F F Q F F SS F NS M,F M,F M M,F RT M M Q, SS, NS and RT combined M,F M,F M,F M,F ------------------------------------------------------ a Gender does not make a significant contrib- ution to ITK prediction when used in conjunc- tion with the other independent variables. However, gender affects the statistical re- sult of almost every hypothesis tested. Notes: M - indicates significant predictor for males. F - indicates significant predictor for females. CONSTRUCT VALIDITY EVIDENCE GRADE LEVEL EFFECTS. This author theo- rized that grade level, being closely related to cognitive development, should correlate positively with the construct. Grade level accounted for unique variance in ITK total score for males but not females. However, for either gender there is no evidence that score on the ITK instrument has a positive relationship with grade level for high school students. A post hoc analysis was conducted to in- vestigate grade level effects. By interview and observation, the amount of elapsed time since taking a course that fit the definition of Technology Education (TE) was ascertained. The following was discovered. It is required of virtually every student (male and female) in Iowa to take 1 or 2 semesters of technol- ogy type education in grades 7 or 8. These courses most closely fit the definition of TE. They are general in nature, do not focus on skill development and stress the impacts of technology. It was found that the more recent the student had taken a TE course the higher their score. This was after factoring out the effect of the other variables. GENDER EFFECTS. Only for subscore #2 did gender account for unique variance in score. It was hypothesized that males would receive higher ITK total scores because of stereotypical male roles in society. While mean scores for males and females were not significantly different on the ITK total test, gender did play a significant role in testing hypotheses. Males and females per- formed very differently on most items. The total variance accounted for in ITK total score by the other independent variables was roughly two-thirds for females of what it was for males. For males the independent vari- ables explain 52% of the variance in ITK total score; for females the figure is 35%. For both sexes, but especially for fe- males, there is much unexplained variance at- tributable to sources other than the variables investigated. The source of the unexplained variance could be attributable to various factors. However, from talking to students and observing instruction, it seems tenable that the student's level of literacy about industrial technology is related to specific course content learned and what the student reads, watches on television, etc. Other possible sources of variance include hobbies, part-time jobs, and interest/aptitude areas. PARENTAL EFFECTS. It was hypothesized that parental contact with tools or machines should correlate positively with the con- struct. For males, father contact with tools or machines as perceived by the subject cor- relates positively with ITK total score. The unique variance accounted for in ITK score by this variable was attributable to subtest #1. The two highest loading items in subtest #1 concern working in industry. For females, father contact with tools or machines played no part in ITK score. Mother contact with tools or machines played no part in ITK score for males or females. COURSE EXPOSURE EFFECTS. Central to this writer's theory of TL and especially the industrial strata is that exposure to I/T/V courses should increase a person's level of the attribute. For males this was not found to be the case. However, for females such a positive relationship was exhibited. Two findings clarify this gender difference. First, the mean number of I/T/V courses taken by males was over three times that of fe- males. Second, after one or two semesters of more general I/T/V courses (which are closest in content to the intended curriculum of TE) the additional courses taken are often more of a craft or vocational nature (which often deviate significantly from the intended con- tent of TE). An examination of the plot be- tween the semesters of I/T/V courses taken by males and their ITK score reveals a curvilinear relationship. When the ITK score is recoded into the categories of 0, 1, 2, and 3 or more semesters, the correlation is significantly positive. Based on course ex- posure effects, it is concluded that there is substantial support for the existence of in- dustrial TL. ITED SUBSCORE EFFECTS. The proposed theory of TL states that the construct is a partial subset of general achievement. It was hypothesized that TL or more narrowly the industrial strata should intercorrelate and behave psychometrically in a similar manner to other subsets of general achievement. In this research project the other measures of general achievement were ITED subscores. The ITK instrument was found to intercorrelate in a similar fashion with ITED subtest intercor- relations. However, the ITK should not cor- relate too highly with other subsets of achievement if it is to be worthy of being partitioned from the encompassing set of at- tributes. Correcting for attenuation, the shared variance between ITK score and ITED subscores ranges from 36% to 56%. These per- centages are large enough to infer a meaning- ful relationship between the latent trait being measured by the ITK and that being measured by the ITED. However, the shared variance is not so large as to negate the ITK instrument as an additional means of measur- ing general achievement. It is concluded that the ITK measures a part of general achievement, only part of which is also meas- ured by ITED subscores. SUMMARY The results of the research give support for the existence of the construct of indus- trial technological literacy; and by extrapo- lation, technological literacy. Results also indicate that TL can be measured reliably. It is concluded that TL is a subset of the general achievement domain but is worthy of separate consideration. Table 4 summarizes the author's interpretation of the construct validity evidence. TABLE 4 STRENGTH OF CONSTRUCT VALIDITY EVIDENCE --------------------------------------------- Total #1 #2 #3 Grade 0,2a 0 0 0 Gender 1b 0 0 0 Father 2 2 0 0 Mother 0 0 0 0 Indus./tech./vocat. courses 2 1 2 1 Q 1 0 0 0 SS 1 0 0 0 NS 2 2 0 2 RT 1 0 0 0 --------------------------------------------- a Substantial evidence when courses are recoded as to their TE content. b Gender has less of an affect when courses are recoded as to their TE content. Notes: 0 = little or no construct validity sup- port. 1 = partial construct validity support. 2 = substantial construct validity sup- port. Based on responses to items in the in- strument it is this researcher's conclusion that there is widespread misunderstanding of technology on the part of many high school students. Based on observation, interviews, and the post hoc analyses it is this re- searcher's conclusion that the most appropri- ate content and pedagogy to increase TL would be those of TE. RECOMMENDATIONS Based on the results and the insights gained from conducting this research the fol- lowing additional research projects are re- commended. 1. Revision and operational field testing of the ITK instrument. The benefits of this would be improvement of psychometric properties and gathering additional va- lidity evidence. 2. Replication of this research for cross validation purposes. 3. Investigation of strata other than indus- trial, e.g., agricultural, biomedical, etc. 4. Development of alternate forms of the ITK instrument so that alternate forms reli- ability can be calculated. 5. Development of other means of measuring TL and subsequent multi-method multi- trait studies. 6. Research aimed at investigating what type of content and/or methodology contributes to and/or increases TL most efficiently. There is much corroborating research (Baker, 1989; Croft, 1990; Hameed, 1988; Hatch, 1985) which defines and gives evi- dence for the validity and utility of measuring TL. This author believes this recommendation to be most important. It seems logical that before we try to in- crease a student's TL we first know the best way or ways to do so. ---------------- Michael A. Hayden is Assistant Professor, De- partment of Technology and Education, Mississippi State University, Mississippi State, MS. REFERENCES Baker, D. (1989). Technological literacy: The essential criteria for a definition. DISSERTATION ABSTRACTS INTERNATIONAL, (University Microfilms No. 89-00392) Croft, V. E. (1990). TECHNOLOGICAL LITERACY: CHARACTERISTICS OF A HIGH SCHOOL GRADUATE. Paper presented at the International Technology Education Confer- ence, Indianapolis, Indiana. Fleming, R. (1989). Literacy for a techno- logical age. SCIENCE EDUCATION, 73(4), 391-404. Hameed, A. (1988). Development of a test of technological literacy. DISSERTATION AB- STRACTS INTERNATIONAL, (University Micro- films No. 88-24509) Hatch, L. O. (1985). TECHNOLOGICAL LITERACY: A SECONDARY ANALYSIS OF NA- TIONAL ASSESSMENT OF EDUCATION PROGRESS SCIENCE DATA. Unpublished doctoral dis- sertation, University of Maryland, College Park. Hayden, M. A. (1989a). The development and validation of a test of technological lit- eracy. DISSERTATION ABSTRACTS INTERNA- TIONAL, (University Microfilms No. 89-20136) Hayden, M. A. (1989b). What is technolog- ical literacy? BULLETIN OF SCIENCE, TECH- NOLOGY AND SOCIETY, 9(4), pp. 228-233. Ley, C. (1987). Technological Literacy and Investigative Imperatives for vocational education researchers. JOURNAL OF VOCA- TIONAL EDUCATION RESEARCH, 12(1), 1-10. Maley, D. (1985, October). ISSUES AND TRENDS IN TECHNOLOGY EDUCATION. Paper presented at the Technology Education Sym- posium VII, California, Pennsylvania. The University of Iowa (1987). MANUAL FOR TEACHERS, ADMINISTRATORS AND COUNSELORS: IOWA TESTS OF EDUCATIONAL DEVELOPMENT: FORMS X-8 AND Y-8. Iowa City: Author. Permission is given to copy any article or graphic provided credit is given and the copies are not intended for sale. Journal of Technology Education Volume 3, Number 1 Fall 1991