JTE v3n1 - The Development and Validation of a Test of Industrial Technological Literacy

Volume 3, Number 1
Fall 1991

The Development and Validation of a Test of Industrial Technological Literacy 
                        Michael A. Hayden
               It is often stated that technology af-
          fects every person on this planet, is growing
          exponentially, and is becoming increasingly
          complicated.  With technology being such a
          force in our lives, it seems logical that we
          should be knowledgeable about it.  This know-
          ledge of technology has been labeled--
          technological literacy (TL).  Several reports
          on education have addressed the TL issue and
          have all called for an increase in the level
          of TL exhibited by our students.  Maley
          (1985) noted that, "in 1984 alone, ten major
          studies of education were reported...each one
          calling for changes in school to prepare our
          students to live in a technological society"
          (p. 16).  Since 1984 an increasing amount of
          research concerning TL has been conducted.
               A large number of educators and special-
          ists in several disciplines are concerned
          about the level of TL displayed by the Ameri-
          can public.  They voice an opinion similar to
          Ley (1987), namely, "the level of technolog-
          ical literacy to which educators and others
          are able to bring the general population will
          determine the future world in which humankind
          will exist" (p. 7).
               A set of very pertinent questions has
          recently been on the minds of many concerned
          leaders.  These questions include:
          1.  As a public, how technologically literate
              are we?
          2.  How do we become technologically liter-
              ate, or become more so?
               To answer these questions we must first
          answer such questions as:
          1.  What is technological literacy?
          2.  Does it really exist, i.e., is it worthy
              of study by itself or is it part of some-
              thing else?
          3.  Assuming TL exits, can we measure it with
               If TL is a viable concept and can be
          measured reliably, then studies can be con-
          ducted to determine the most advantageous ed-
          ucational environment for the attainment of
          such a characteristic.  A test of TL with
          sufficient validity could be used to identify
          and diagnose those students in need of
          greater technological knowledge, as well as
          evaluate the effectiveness of programs teach-
          ing the construct.
               The purpose of the study was to investi-
          gate the validity of the construct of TL.
          The definition of TL will not be expounded
          upon here since it has been exhaustively cov-
          ered by this and many other authors (see
          Hayden, 1989b).  In this study the following
          definition of technological literacy was
          used:  technological literacy is having the
          knowledge and ability to select, properly ap-
          ply, then monitor and evaluate appropriate
          technology given the context.
               Following are general questions the
          study investigated.
          1.  Does the empirical model (an instrument
              measuring TL) exhibit satisfactory
              psychometric properties?
          2.  Does the empirical model exhibit multiple
          3.  How do the possible correlates (independ-
              ent variables) of general achievement,
              grade level, gender, parental contact
              with technology, and prior industrial
              course exposure affect TL?
               For the purpose of this study, technol-
          ogy was defined as:  the process by which hu-
          mans utilize available resources to extend
          their potential or alter their environment
          (Hayden, 1989b).  Based on this definition it
          was theorized that technology is a learned
          phenomenon.  Hence, being literate about
          technology should also be a learned phenome-
          non.  Technological literacy is not skill in
          directly applying specific technology, i.e.,
          technical literacy.  Technological literacy
          for the purpose of this research was viewed
          as general knowledge, abilities and behaviors
          concerning technology.  Furthermore, techno-
          logical literacy was not viewed as being re-
          stricted to general skills in applying
          technology, i.e. technological operacy.
          Therefore, TL was assumed to be more closely
          related to achievement than aptitude.
               Being related to general achievement, a
          TL instrument should perform psychometrically
          similar to other measures of general achieve-
          ment.  If TL is in part related to achieve-
          ment, a person's standing on the attribute
          should increase with age due to developmental
          factors.  Also, a person's score on a TL in-
          strument should increase with additional
          learning and exposure related to the attri-
          bute.  Additional exposure could come from
          several sources, e.g., parental/student
          interaction, school curriculum and hobbies.
               To investigate the questions of the
          study, 29 hypotheses were tested, the major-
          ity of which related to eight dependent vari-
          ables (total test and three subtests for male
          and female).  A synopsis of the research
          questions follow.
          1.  Is the instrument unidimensional?
          2.  To what extent do the independent vari-
              ables account for unique variance in in-
              strument score.
          3.  Can the independent variables be used to
              predict scores on the instrument?
          4.  Does the instrument behave
              psychometrically in a similar fashion to
              other measures of general achievement?
          5.  Does the score on the instrument have a
              linear relationship with grade level?
          6.  Do males and females score differently?
          7.  Is there any interaction between gender
              and grade level on the instrument score?
               The instrument was developed from a rig-
          orous test plan which included a detailed ra-
          tionale and definition of the domain
          including limitations, delimitations, the
          target population, uses and constraints of
          the test.  A blueprint was created that de-
          tailed the format and scoring procedures.
          The test plan also included generation of
          items and revising algorithms and adminis-
          tration instructions.
               Clearly, technological literacy is a
          broad term.  Many authorities maintain that
          it encompasses operacy, i.e, what we can do
          with the technology, in addition to typical
          literacy concepts.  Many (see Fleming, 1989)
          see technological literacy as a type of
          empowerment enabling us to do a very broad
          range of things.  Additionally, general lit-
          eracy concerning technology would cover all
          technology, e. g., agricultural, medical mil-
          itary, etc.  Lastly, literacy about technol-
          ogy would encompass affective, cognitive and
          psychomotor domains.
               The time and monetary limitations placed
          on this project necessitated a tightly fo-
          cused test blueprint.  The following delimi-
          tations and their accompanying rationale were
          used to focus the instrument.
          1.  Focus on the industrial strata because it
              was assumed that if industrial technolog-
              ical literacy was a valid concept then
              the more encompassing concept of TL was
              also valid; this strata was the research-
              er's area of expertise; prior research
              had most clearly explained this strata;
              and it was perceived that an instrument
              focused on industrial technology would
              have the most future utility for use in
              industrial programs.
          2.  Maintain a primary focus on the cognitive
              domain because it readily lends itself to
              item writing, test administration, scor-
              ing and data analysis techniques which
              are relatively unambiguous and can be
              used with a large number of items and
              subjects in a timely fashion.
          3.  Delimit the content of the items to gen-
              eral principles of technology or recent
              technological innovations/impacts because
              it was assumed that if an individual un-
              derstood and was literate about these
              then they would know or could easily
              learn about other time periods.
          The instrument was titled "Industrial Techno-
          logical Knowledge" (ITK).  Trained item writ-
          ers developed and refined the items in the
          instrument.  The writers were all faculty or
          doctoral students of the Industrial Education
          and Technology or Research and Evaluation De-
          partments at Iowa State University.  The mean
          number of years of industrial/consulting and
          teaching experience of the writers was nine
          and four years respectively.  The items
          underwent several iterations of qualitative
          and statistical selection and revision.
          There were several item tryouts in addition
          to a pilot study.
               Parental contact with technology was
          measured dichotomously for father and mother.
          Students responded to the question, "Does
          your parent through work, major home respon-
          sibilities or hobbies work with tools and/or
               Industrial course exposure was measured
          by recording the number of semesters of
          industrial/technical/vocational (I/T/V)
          courses taken.  This included the current se-
               General achievement was measured by the
          students' Iowa Test of Educational Develop-
          ment (ITED) subscores of Natural Science
          (NS), Quantitative (Q), Reading Total (RT)
          and Social Science (SS).
               Exhaustive analyses were conducted.
          Only the results will be summarized in this
          paper.  For most analyses, cases with missing
          data were deleted case-wise, i.e., only sub-
          jects for whom complete information on the
          independent variables was obtained were used
          in the analyses.  All correlational and t-
          tests were two-tailed and performed at the
          .05 level.  Several post hoc analyses were
          conducted.  Many of these were outside of the
          realm of the study and were of small sample
          size.  However, they play a significant role
          in interpretation of the results and recomm-
          endations for future inquiry.
               Primary analysis procedures included
          factor analysis to determine the dimensional-
          ity of the instrument and correlational,
          analysis of variance and regression proce-
          dures to investigate variance and predict-
          ability questions.  The instrument's
          characteristics were compared with other in-
          struments measuring general achievement to
          assess its psychometric properties.  Visual
          inspection of plotted cell means and variable
          correlations were also used to assess statis-
          tical procedure assumptions such as linear
               Descriptive statistics of the sample,
          the ITK scores, and the individual item char-
          acteristics were generated by a variety of
          techniques including one parameter logistic
          scaling, analysis of variance, and
          correlational methods.
               The population of this study was high
          school students in the state of Iowa.  The
          sample was comprised of volunteering school
          buildings.  Within the schools, intact
          classes were targeted for participation in
          the study.  The classes selected were as het-
          erogeneous as possible.  Most classes se-
          lected were required courses.  Approximately
          25% of the subjects were from each of the
          grade levels 9 - 12.  Within grade levels,
          males and females were represented approxi-
          mately equally.  Virtually all students had
          some I/T/V course exposure.  Industrial Tech-
          nological Knowledge test scores were col-
          lected from 826 subjects.
               Only those results most significant to
          interpretation and recommendations will be
          discussed.  For detailed hypotheses and re-
          sults of all statistical tests see the ori-
          ginal study (Hayden, 1989a).
               INSTRUMENT STATISTICS.  The internal
          consistency reliability of the total test was
          .83.  The internal consistency of the ITED
          subtests range from .86 to .96 (The Univer-
          sity of Iowa, 1987).  Application of the
          Spearman-Brown formula indicated that if the
          ITK instrument were the same length as the
          ITED composite instrument, the ITK's esti-
          mated reliability would be .975.
               The average interitem correlation of the
          ITK was rather low, being only .10.  Average
          item/total correlations were also small, be-
          ing .28.  It is possible that these corre-
          lations could be increased by revising the
          items and increasing sample size.  If the av-
          erage interitem correlation could be in-
          creased to .15 (the average interitem
          correlation of subtest #3), the reliability
          estimate for the 45 item instrument would be
          .89, uncorrected.
               There was no floor or ceiling effect on
          the ITK instrument.  The distribution of
          scores was nearly normal with an average item
          mean of .5.  This allowed for maximum vari-
          ance.  However, due to the potential for
          guessing, a slightly easier test would have
          given more information about test takers.
               The 2.94 standard error of measurement
          of the ITK instrument is proportionately
          large compared to the ITED composite.  The
          standard error of measurement for the 361
          item ITED composite is approximately 1.2.  It
          is possible that further item revision could
          decrease the ITK's standard error of measure-
          ment.  Table 1 summarizes the instrument's
          psychometric properties.
          TABLE 1
                                             Total test
          Number of items                        45
          Cronbach's alpha a                    .83
          Minimum score                          2
          Maximum score                          44
          Mean                                 22.23
          Median                                 23
          Mode                                   24
          Standard deviation                    7.12
          Standard error of measurement         2.94
          Average item/total score correlation  .28
          Skewness of distribution             -0.13
          Kurtosis of distribution             -0.55
          Average inter-item correlation        .10
          Average item mean                     .50
          a Calculated using uncorrected point-biserial
               The data collected from the main phase
          of this study is equivalent to the main field
          testing phase of a research and development
          project.  It is the view of this researcher
          that the ITK instrument exhibits satisfactory
          psychometric properties to be worthy of re-
          vision and advancement to the operational
          field testing stage of development.
               ITEM STATISTICS.  Among items and be-
          tween gender there was much fluctuation of
          item means.  Females do better on some items,
          while males do better on others.  These fluc-
          tuations are much more pronounced for items
          which are more difficult.  Females tended to
          perform better on items requiring reading
          comprehension.  There was no consistent pat-
          tern of item means between males and females
          except that males scored higher on most
               DIMENSIONALITY.  Three factors or sub-
          tests were extracted from the ITK instrument.
          Factor #1 primarily contained items pertain-
          ing to the human adaptive systems.  The two
          highest loading items on this factor and
          their loadings were:
          #18   Why can computers lead to greater pro-
                ductivity in industry?  REDUCTION OF
                SPEED, AND INCREASED ACCURACY.  Loading
                of .81.
          #29   The average worker has to be retrained
                because NEW METHODS OF DOING THINGS
                HAVE BEEN DEVELOPED.  Loading of .80.
               Factor #2 primarily contained items per-
          taining to the application of other disci-
          plines (most notably math and science) to the
          solution of technological problems.  The two
          highest loading items on this factor and
          their loadings were:
          #15   One of the most promising uses of
                microbes is for CLEANING THE ENVIRON-
                MENT.  Loading of .69.
          #31   Cooking in a microwave oven is done by
                EXCITING THE ATOMS IN THE FOOD.  Load-
                ing of .64.
               Factor #3 primarily contained items
          dealing with the interpretation of written
          and graphical material having a technological
          theme.  The two highest loading items on this
          factor and their loadings were:
          #37   Technology can best be defined as PEO-
                CAPABILITIES.  .76.
          #45   Consider this situation.  The design,
                manufacturing, marketing, and manage-
                ment of product XX was performed by
                hundreds of individuals.  After using
                product XX, many consumers were in-
                jured.  In a latter court case, injured
                person John Doe testified that he
                thought product XX was unsafe even be-
                fore he used it.  Several employees
                that helped make and sell the product
                also testified that they had always
                thought XX was dangerous.  The Judge
                asked the injured person why he went
                ahead and used the product.  He also
                asked the employees why they went ahead
                and made and sold the product even
                though they thought it was unsafe.  The
                situation presented above could be ac-
                counted for because INDIVIDUALS, WHEN
                ABOUT THE POSSIBLE OUTCOMES.  Loading
                of .66.
               It was found that the score on the ITK
          instrument could be predicted.  Table 2 shows
          the regression equations for males and fe-
          males.  Table 3 further summarizes the pre-
          diction findings.
          TABLE 2
          ***(Available as a graphic only)***
          TABLE 3
                                      Total    #1     #2     #3
          Grade                        M,F     F             F
          Gender                       x a     x      x      x
          Father                       M              M      M
          Father/mother combined       M                     M
          Indust./tech./vocat. course  F              F
          Q                            F              F
          SS                                          F
          NS                           M,F     M,F    M      M,F
          RT                           M       M
          Q, SS, NS and RT combined    M,F     M,F    M,F    M,F
          a Gender does not make a significant contrib-
          ution to ITK prediction when used in conjunc-
          tion with the other independent variables.
          However, gender affects the statistical re-
          sult of almost every hypothesis tested.
          Notes: M - indicates significant predictor for
                 F - indicates significant predictor for
               GRADE LEVEL EFFECTS.  This author theo-
          rized that grade level, being closely related
          to cognitive development, should correlate
          positively with the construct.  Grade level
          accounted for unique variance in ITK total
          score for males but not females.  However,
          for either gender there is no evidence that
          score on the ITK instrument has a positive
          relationship with grade level for high school
               A post hoc analysis was conducted to in-
          vestigate grade level effects.  By interview
          and observation, the amount of elapsed time
          since taking a course that fit the definition
          of Technology Education (TE) was ascertained.
          The following was discovered.  It is required
          of virtually every student (male and female)
          in Iowa to take 1 or 2 semesters of technol-
          ogy type education in grades 7 or 8. These
          courses most closely fit the definition of
          TE.  They are general in nature, do not focus
          on skill development and stress the impacts
          of technology.  It was found that the more
          recent the student had taken a TE course the
          higher their score.  This was after factoring
          out the effect of the other variables.
               GENDER EFFECTS.  Only for subscore #2
          did gender account for unique variance in
          score.  It was hypothesized that males would
          receive higher ITK total scores because of
          stereotypical male roles in society.  While
          mean scores for males and females were not
          significantly different on the ITK total
          test, gender did play a significant role in
          testing hypotheses.  Males and females per-
          formed very differently on most items.  The
          total variance accounted for in ITK total
          score by the other independent variables was
          roughly two-thirds for females of what it was
          for males.  For males the independent vari-
          ables explain 52% of the variance in ITK
          total score; for females the figure is 35%.
               For both sexes, but especially for fe-
          males, there is much unexplained variance at-
          tributable to sources other than the
          variables investigated.  The source of the
          unexplained variance could be attributable to
          various factors.  However, from talking to
          students and observing instruction, it seems
          tenable that the student's level of literacy
          about industrial technology is related to
          specific course content learned and what the
          student reads, watches on television, etc.
          Other possible sources of variance include
          hobbies, part-time jobs, and
          interest/aptitude areas.
               PARENTAL EFFECTS.  It was hypothesized
          that parental contact with tools or machines
          should correlate positively with the con-
          struct.  For males, father contact with tools
          or machines as perceived by the subject cor-
          relates positively with ITK total score.  The
          unique variance accounted for in ITK score by
          this variable was attributable to subtest #1.
          The two highest loading items in subtest #1
          concern working in industry.  For females,
          father contact with tools or machines played
          no part in ITK score.  Mother contact with
          tools or machines played no part in ITK score
          for males or females.
               COURSE EXPOSURE EFFECTS.  Central to
          this writer's theory of TL and especially the
          industrial strata is that exposure to I/T/V
          courses should increase a person's level of
          the attribute.  For males this was not found
          to be the case.  However, for females such a
          positive relationship was exhibited.  Two
          findings clarify this gender difference.
          First, the mean number of I/T/V courses taken
          by males was over three times that of fe-
          males.  Second, after one or two semesters of
          more general I/T/V courses (which are closest
          in content to the intended curriculum of TE)
          the additional courses taken are often more
          of a craft or vocational nature (which often
          deviate significantly from the intended con-
          tent of TE).  An examination of the plot be-
          tween the semesters of I/T/V courses taken by
          males and their ITK score reveals a
          curvilinear relationship.  When the ITK score
          is recoded into the categories of 0, 1, 2,
          and 3 or more semesters, the correlation is
          significantly positive.  Based on course ex-
          posure effects, it is concluded that there is
          substantial support for the existence of in-
          dustrial TL.
               ITED SUBSCORE EFFECTS.  The proposed
          theory of TL states that the construct is a
          partial subset of general achievement.  It
          was hypothesized that TL or more narrowly the
          industrial strata should intercorrelate and
          behave psychometrically in a similar manner
          to other subsets of general achievement.  In
          this research project the other measures of
          general achievement were ITED subscores.  The
          ITK instrument was found to intercorrelate in
          a similar fashion with ITED subtest intercor-
          relations.  However, the ITK should not cor-
          relate too highly with other subsets of
          achievement if it is to be worthy of being
          partitioned from the encompassing set of at-
          tributes.  Correcting for attenuation, the
          shared variance between ITK score and ITED
          subscores ranges from 36% to 56%.  These per-
          centages are large enough to infer a meaning-
          ful relationship between the latent trait
          being measured by the ITK and that being
          measured by the ITED.  However, the shared
          variance is not so large as to negate the ITK
          instrument as an additional means of measur-
          ing general achievement.  It is concluded
          that the ITK measures a part of general
          achievement, only part of which is also meas-
          ured by ITED subscores.
               The results of the research give support
          for the existence of the construct of indus-
          trial technological literacy; and by extrapo-
          lation, technological literacy.  Results also
          indicate that TL can be measured reliably.
          It is concluded that TL is a subset of the
          general achievement domain but is worthy of
          separate consideration.  Table 4 summarizes
          the author's interpretation of the construct
          validity evidence.
          TABLE 4
                                   Total  #1   #2   #3
          Grade                     0,2a   0    0    0
          Gender                      1b   0    0    0
          Father                      2    2    0    0
          Mother                      0    0    0    0
          Indus./tech./vocat. courses 2    1    2    1
          Q                           1    0    0    0
          SS                          1    0    0    0
          NS                          2    2    0    2
          RT                          1    0    0    0
          a Substantial evidence when courses are recoded
          as to their TE content.
          b Gender has less of an affect when courses are
          recoded as to their TE content.
          Notes: 0 = little or no construct validity sup-
                 1 = partial construct validity support.
                 2 = substantial construct validity sup-
               Based on responses to items in the in-
          strument it is this researcher's conclusion
          that there is widespread misunderstanding of
          technology on the part of many high school
          students.  Based on observation, interviews,
          and the post hoc analyses it is this re-
          searcher's conclusion that the most appropri-
          ate content and pedagogy to increase TL would
          be those of TE.
               Based on the results and the insights
          gained from conducting this research the fol-
          lowing additional research projects are re-
          1.  Revision and operational field testing of
              the ITK instrument.  The benefits of this
              would be improvement of psychometric
              properties and gathering additional va-
              lidity evidence.
          2.  Replication of this research for cross
              validation purposes.
          3.  Investigation of strata other than indus-
              trial, e.g., agricultural, biomedical,
          4.  Development of alternate forms of the ITK
              instrument so that alternate forms reli-
              ability can be calculated.
          5.  Development of other means of measuring
              TL and subsequent multi-method multi-
              trait studies.
          6.  Research aimed at investigating what type
              of content and/or methodology contributes
              to and/or increases TL most efficiently.
              There is much corroborating research
              (Baker, 1989; Croft, 1990; Hameed, 1988;
              Hatch, 1985) which defines and gives evi-
              dence for the validity and utility of
              measuring TL.  This author believes this
              recommendation to be most important.  It
              seems logical that before we try to in-
              crease a student's TL we first know the
              best way or ways to do so.
          Michael A. Hayden is Assistant Professor, De-
          partment of Technology and Education,
          Mississippi State University, Mississippi
          State, MS.
          Baker, D.  (1989).  Technological literacy:
             The essential criteria for a definition.
             (University Microfilms No. 89-00392)
          Croft, V. E.  (1990).  TECHNOLOGICAL
             SCHOOL GRADUATE.  Paper presented at the
             International Technology Education Confer-
             ence, Indianapolis, Indiana.
          Fleming, R.  (1989).  Literacy for a techno-
             logical age.  SCIENCE EDUCATION, 73(4),
          Hameed, A.  (1988).  Development of a test of
             technological literacy.  DISSERTATION AB-
             STRACTS INTERNATIONAL, (University Micro-
             films No. 88-24509)
          Hatch, L. O.  (1985).  TECHNOLOGICAL
             SCIENCE DATA.  Unpublished doctoral dis-
             sertation, University of Maryland, College
          Hayden, M. A.  (1989a).  The development and
             validation of a test of technological lit-
             TIONAL, (University Microfilms No.
          Hayden, M. A.  (1989b).  What is technolog-
             ical literacy?  BULLETIN OF SCIENCE, TECH-
             NOLOGY AND SOCIETY, 9(4), pp. 228-233.
          Ley, C.  (1987).  Technological Literacy and
             Investigative Imperatives for vocational
             education researchers.  JOURNAL OF VOCA-
             TIONAL EDUCATION RESEARCH, 12(1), 1-10.
          Maley, D.  (1985, October).  ISSUES AND
             presented at the Technology Education Sym-
             posium VII, California, Pennsylvania.
          The University of Iowa (1987).  MANUAL FOR
             FORMS X-8 AND Y-8.  Iowa City:  Author.
          Permission is given to copy any
          article or graphic provided credit is given and
          the copies are not intended for sale.
Journal of Technology Education   Volume 3, Number 1       Fall 1991