JTE v3n1 - The Development and Validation of a Test of Industrial Technological Literacy
Volume 3, Number 1
Fall 1991
The Development and Validation of a Test of Industrial Technological Literacy
Michael A. Hayden
It is often stated that technology af-
fects every person on this planet, is growing
exponentially, and is becoming increasingly
complicated. With technology being such a
force in our lives, it seems logical that we
should be knowledgeable about it. This know-
ledge of technology has been labeled--
technological literacy (TL). Several reports
on education have addressed the TL issue and
have all called for an increase in the level
of TL exhibited by our students. Maley
(1985) noted that, "in 1984 alone, ten major
studies of education were reported...each one
calling for changes in school to prepare our
students to live in a technological society"
(p. 16). Since 1984 an increasing amount of
research concerning TL has been conducted.
A large number of educators and special-
ists in several disciplines are concerned
about the level of TL displayed by the Ameri-
can public. They voice an opinion similar to
Ley (1987), namely, "the level of technolog-
ical literacy to which educators and others
are able to bring the general population will
determine the future world in which humankind
will exist" (p. 7).
A set of very pertinent questions has
recently been on the minds of many concerned
leaders. These questions include:
1. As a public, how technologically literate
are we?
2. How do we become technologically liter-
ate, or become more so?
To answer these questions we must first
answer such questions as:
1. What is technological literacy?
2. Does it really exist, i.e., is it worthy
of study by itself or is it part of some-
thing else?
3. Assuming TL exits, can we measure it with
utility?
If TL is a viable concept and can be
measured reliably, then studies can be con-
ducted to determine the most advantageous ed-
ucational environment for the attainment of
such a characteristic. A test of TL with
sufficient validity could be used to identify
and diagnose those students in need of
greater technological knowledge, as well as
evaluate the effectiveness of programs teach-
ing the construct.
PURPOSE OF THE STUDY
The purpose of the study was to investi-
gate the validity of the construct of TL.
The definition of TL will not be expounded
upon here since it has been exhaustively cov-
ered by this and many other authors (see
Hayden, 1989b). In this study the following
definition of technological literacy was
used: technological literacy is having the
knowledge and ability to select, properly ap-
ply, then monitor and evaluate appropriate
technology given the context.
Following are general questions the
study investigated.
1. Does the empirical model (an instrument
measuring TL) exhibit satisfactory
psychometric properties?
2. Does the empirical model exhibit multiple
dimensions?
3. How do the possible correlates (independ-
ent variables) of general achievement,
grade level, gender, parental contact
with technology, and prior industrial
course exposure affect TL?
THEORY OF TECHNOLOGICAL LITERACY
For the purpose of this study, technol-
ogy was defined as: the process by which hu-
mans utilize available resources to extend
their potential or alter their environment
(Hayden, 1989b). Based on this definition it
was theorized that technology is a learned
phenomenon. Hence, being literate about
technology should also be a learned phenome-
non. Technological literacy is not skill in
directly applying specific technology, i.e.,
technical literacy. Technological literacy
for the purpose of this research was viewed
as general knowledge, abilities and behaviors
concerning technology. Furthermore, techno-
logical literacy was not viewed as being re-
stricted to general skills in applying
technology, i.e. technological operacy.
Therefore, TL was assumed to be more closely
related to achievement than aptitude.
Being related to general achievement, a
TL instrument should perform psychometrically
similar to other measures of general achieve-
ment. If TL is in part related to achieve-
ment, a person's standing on the attribute
should increase with age due to developmental
factors. Also, a person's score on a TL in-
strument should increase with additional
learning and exposure related to the attri-
bute. Additional exposure could come from
several sources, e.g., parental/student
interaction, school curriculum and hobbies.
RESEARCH QUESTIONS
To investigate the questions of the
study, 29 hypotheses were tested, the major-
ity of which related to eight dependent vari-
ables (total test and three subtests for male
and female). A synopsis of the research
questions follow.
1. Is the instrument unidimensional?
2. To what extent do the independent vari-
ables account for unique variance in in-
strument score.
3. Can the independent variables be used to
predict scores on the instrument?
4. Does the instrument behave
psychometrically in a similar fashion to
other measures of general achievement?
5. Does the score on the instrument have a
linear relationship with grade level?
6. Do males and females score differently?
7. Is there any interaction between gender
and grade level on the instrument score?
METHODOLOGY
INSTRUMENTATION
The instrument was developed from a rig-
orous test plan which included a detailed ra-
tionale and definition of the domain
including limitations, delimitations, the
target population, uses and constraints of
the test. A blueprint was created that de-
tailed the format and scoring procedures.
The test plan also included generation of
items and revising algorithms and adminis-
tration instructions.
Clearly, technological literacy is a
broad term. Many authorities maintain that
it encompasses operacy, i.e, what we can do
with the technology, in addition to typical
literacy concepts. Many (see Fleming, 1989)
see technological literacy as a type of
empowerment enabling us to do a very broad
range of things. Additionally, general lit-
eracy concerning technology would cover all
technology, e. g., agricultural, medical mil-
itary, etc. Lastly, literacy about technol-
ogy would encompass affective, cognitive and
psychomotor domains.
The time and monetary limitations placed
on this project necessitated a tightly fo-
cused test blueprint. The following delimi-
tations and their accompanying rationale were
used to focus the instrument.
1. Focus on the industrial strata because it
was assumed that if industrial technolog-
ical literacy was a valid concept then
the more encompassing concept of TL was
also valid; this strata was the research-
er's area of expertise; prior research
had most clearly explained this strata;
and it was perceived that an instrument
focused on industrial technology would
have the most future utility for use in
industrial programs.
2. Maintain a primary focus on the cognitive
domain because it readily lends itself to
item writing, test administration, scor-
ing and data analysis techniques which
are relatively unambiguous and can be
used with a large number of items and
subjects in a timely fashion.
3. Delimit the content of the items to gen-
eral principles of technology or recent
technological innovations/impacts because
it was assumed that if an individual un-
derstood and was literate about these
then they would know or could easily
learn about other time periods.
The instrument was titled "Industrial Techno-
logical Knowledge" (ITK). Trained item writ-
ers developed and refined the items in the
instrument. The writers were all faculty or
doctoral students of the Industrial Education
and Technology or Research and Evaluation De-
partments at Iowa State University. The mean
number of years of industrial/consulting and
teaching experience of the writers was nine
and four years respectively. The items
underwent several iterations of qualitative
and statistical selection and revision.
There were several item tryouts in addition
to a pilot study.
Parental contact with technology was
measured dichotomously for father and mother.
Students responded to the question, "Does
your parent through work, major home respon-
sibilities or hobbies work with tools and/or
machines?"
Industrial course exposure was measured
by recording the number of semesters of
industrial/technical/vocational (I/T/V)
courses taken. This included the current se-
mester.
General achievement was measured by the
students' Iowa Test of Educational Develop-
ment (ITED) subscores of Natural Science
(NS), Quantitative (Q), Reading Total (RT)
and Social Science (SS).
DATA ANALYSIS
Exhaustive analyses were conducted.
Only the results will be summarized in this
paper. For most analyses, cases with missing
data were deleted case-wise, i.e., only sub-
jects for whom complete information on the
independent variables was obtained were used
in the analyses. All correlational and t-
tests were two-tailed and performed at the
.05 level. Several post hoc analyses were
conducted. Many of these were outside of the
realm of the study and were of small sample
size. However, they play a significant role
in interpretation of the results and recomm-
endations for future inquiry.
Primary analysis procedures included
factor analysis to determine the dimensional-
ity of the instrument and correlational,
analysis of variance and regression proce-
dures to investigate variance and predict-
ability questions. The instrument's
characteristics were compared with other in-
struments measuring general achievement to
assess its psychometric properties. Visual
inspection of plotted cell means and variable
correlations were also used to assess statis-
tical procedure assumptions such as linear
relationships.
Descriptive statistics of the sample,
the ITK scores, and the individual item char-
acteristics were generated by a variety of
techniques including one parameter logistic
scaling, analysis of variance, and
correlational methods.
POPULATION AND SAMPLE
The population of this study was high
school students in the state of Iowa. The
sample was comprised of volunteering school
buildings. Within the schools, intact
classes were targeted for participation in
the study. The classes selected were as het-
erogeneous as possible. Most classes se-
lected were required courses. Approximately
25% of the subjects were from each of the
grade levels 9 - 12. Within grade levels,
males and females were represented approxi-
mately equally. Virtually all students had
some I/T/V course exposure. Industrial Tech-
nological Knowledge test scores were col-
lected from 826 subjects.
INTERPRETATION OF RESULTS
Only those results most significant to
interpretation and recommendations will be
discussed. For detailed hypotheses and re-
sults of all statistical tests see the ori-
ginal study (Hayden, 1989a).
PSYCHOMETRIC PROPERTIES
INSTRUMENT STATISTICS. The internal
consistency reliability of the total test was
.83. The internal consistency of the ITED
subtests range from .86 to .96 (The Univer-
sity of Iowa, 1987). Application of the
Spearman-Brown formula indicated that if the
ITK instrument were the same length as the
ITED composite instrument, the ITK's esti-
mated reliability would be .975.
The average interitem correlation of the
ITK was rather low, being only .10. Average
item/total correlations were also small, be-
ing .28. It is possible that these corre-
lations could be increased by revising the
items and increasing sample size. If the av-
erage interitem correlation could be in-
creased to .15 (the average interitem
correlation of subtest #3), the reliability
estimate for the 45 item instrument would be
.89, uncorrected.
There was no floor or ceiling effect on
the ITK instrument. The distribution of
scores was nearly normal with an average item
mean of .5. This allowed for maximum vari-
ance. However, due to the potential for
guessing, a slightly easier test would have
given more information about test takers.
The 2.94 standard error of measurement
of the ITK instrument is proportionately
large compared to the ITED composite. The
standard error of measurement for the 361
item ITED composite is approximately 1.2. It
is possible that further item revision could
decrease the ITK's standard error of measure-
ment. Table 1 summarizes the instrument's
psychometric properties.
TABLE 1
INSTRUMENT STATISTICS - MALE AND FEMALE COM-
BINED
---------------------------------------------
Total test
H.S.
Number of items 45
Cronbach's alpha a .83
Minimum score 2
Maximum score 44
Mean 22.23
Median 23
Mode 24
Standard deviation 7.12
Standard error of measurement 2.94
Average item/total score correlation .28
Skewness of distribution -0.13
Kurtosis of distribution -0.55
Average inter-item correlation .10
Average item mean .50
---------------------------------------------
a Calculated using uncorrected point-biserial
correlation.
The data collected from the main phase
of this study is equivalent to the main field
testing phase of a research and development
project. It is the view of this researcher
that the ITK instrument exhibits satisfactory
psychometric properties to be worthy of re-
vision and advancement to the operational
field testing stage of development.
ITEM STATISTICS. Among items and be-
tween gender there was much fluctuation of
item means. Females do better on some items,
while males do better on others. These fluc-
tuations are much more pronounced for items
which are more difficult. Females tended to
perform better on items requiring reading
comprehension. There was no consistent pat-
tern of item means between males and females
except that males scored higher on most
items.
DIMENSIONALITY. Three factors or sub-
tests were extracted from the ITK instrument.
Factor #1 primarily contained items pertain-
ing to the human adaptive systems. The two
highest loading items on this factor and
their loadings were:
#18 Why can computers lead to greater pro-
ductivity in industry? REDUCTION OF
HUMAN ERROR, REDUCTION OF COST, GREATER
SPEED, AND INCREASED ACCURACY. Loading
of .81.
#29 The average worker has to be retrained
because NEW METHODS OF DOING THINGS
HAVE BEEN DEVELOPED. Loading of .80.
Factor #2 primarily contained items per-
taining to the application of other disci-
plines (most notably math and science) to the
solution of technological problems. The two
highest loading items on this factor and
their loadings were:
#15 One of the most promising uses of
microbes is for CLEANING THE ENVIRON-
MENT. Loading of .69.
#31 Cooking in a microwave oven is done by
EXCITING THE ATOMS IN THE FOOD. Load-
ing of .64.
Factor #3 primarily contained items
dealing with the interpretation of written
and graphical material having a technological
theme. The two highest loading items on this
factor and their loadings were:
#37 Technology can best be defined as PEO-
PLE USING TOOLS, RESOURCES, AND PROC-
ESSES TO SOLVE PROBLEMS OR EXTEND THEIR
CAPABILITIES. .76.
#45 Consider this situation. The design,
manufacturing, marketing, and manage-
ment of product XX was performed by
hundreds of individuals. After using
product XX, many consumers were in-
jured. In a latter court case, injured
person John Doe testified that he
thought product XX was unsafe even be-
fore he used it. Several employees
that helped make and sell the product
also testified that they had always
thought XX was dangerous. The Judge
asked the injured person why he went
ahead and used the product. He also
asked the employees why they went ahead
and made and sold the product even
though they thought it was unsafe. The
situation presented above could be ac-
counted for because INDIVIDUALS, WHEN
GIVEN DIRECTIONS BY PEOPLE IN AUTHOR-
ITY, OFTEN DO THINGS WITHOUT THINKING
ABOUT THE POSSIBLE OUTCOMES. Loading
of .66.
PREDICTING SCORES
It was found that the score on the ITK
instrument could be predicted. Table 2 shows
the regression equations for males and fe-
males. Table 3 further summarizes the pre-
diction findings.
TABLE 2
REGRESSION EQUATION AND SUMMARY STATISTICS
FOR PREDICTION OF ITK TOTAL SCORE
***(Available as a graphic only)***
TABLE 3
SIGNIFICANTLY PREDICTING VARIABLES
------------------------------------------------------
Total #1 #2 #3
Grade M,F F F
Gender x a x x x
Father M M M
Mother
Father/mother combined M M
Indust./tech./vocat. course F F
Q F F
SS F
NS M,F M,F M M,F
RT M M
Q, SS, NS and RT combined M,F M,F M,F M,F
------------------------------------------------------
a Gender does not make a significant contrib-
ution to ITK prediction when used in conjunc-
tion with the other independent variables.
However, gender affects the statistical re-
sult of almost every hypothesis tested.
Notes: M - indicates significant predictor for
males.
F - indicates significant predictor for
females.
CONSTRUCT VALIDITY EVIDENCE
GRADE LEVEL EFFECTS. This author theo-
rized that grade level, being closely related
to cognitive development, should correlate
positively with the construct. Grade level
accounted for unique variance in ITK total
score for males but not females. However,
for either gender there is no evidence that
score on the ITK instrument has a positive
relationship with grade level for high school
students.
A post hoc analysis was conducted to in-
vestigate grade level effects. By interview
and observation, the amount of elapsed time
since taking a course that fit the definition
of Technology Education (TE) was ascertained.
The following was discovered. It is required
of virtually every student (male and female)
in Iowa to take 1 or 2 semesters of technol-
ogy type education in grades 7 or 8. These
courses most closely fit the definition of
TE. They are general in nature, do not focus
on skill development and stress the impacts
of technology. It was found that the more
recent the student had taken a TE course the
higher their score. This was after factoring
out the effect of the other variables.
GENDER EFFECTS. Only for subscore #2
did gender account for unique variance in
score. It was hypothesized that males would
receive higher ITK total scores because of
stereotypical male roles in society. While
mean scores for males and females were not
significantly different on the ITK total
test, gender did play a significant role in
testing hypotheses. Males and females per-
formed very differently on most items. The
total variance accounted for in ITK total
score by the other independent variables was
roughly two-thirds for females of what it was
for males. For males the independent vari-
ables explain 52% of the variance in ITK
total score; for females the figure is 35%.
For both sexes, but especially for fe-
males, there is much unexplained variance at-
tributable to sources other than the
variables investigated. The source of the
unexplained variance could be attributable to
various factors. However, from talking to
students and observing instruction, it seems
tenable that the student's level of literacy
about industrial technology is related to
specific course content learned and what the
student reads, watches on television, etc.
Other possible sources of variance include
hobbies, part-time jobs, and
interest/aptitude areas.
PARENTAL EFFECTS. It was hypothesized
that parental contact with tools or machines
should correlate positively with the con-
struct. For males, father contact with tools
or machines as perceived by the subject cor-
relates positively with ITK total score. The
unique variance accounted for in ITK score by
this variable was attributable to subtest #1.
The two highest loading items in subtest #1
concern working in industry. For females,
father contact with tools or machines played
no part in ITK score. Mother contact with
tools or machines played no part in ITK score
for males or females.
COURSE EXPOSURE EFFECTS. Central to
this writer's theory of TL and especially the
industrial strata is that exposure to I/T/V
courses should increase a person's level of
the attribute. For males this was not found
to be the case. However, for females such a
positive relationship was exhibited. Two
findings clarify this gender difference.
First, the mean number of I/T/V courses taken
by males was over three times that of fe-
males. Second, after one or two semesters of
more general I/T/V courses (which are closest
in content to the intended curriculum of TE)
the additional courses taken are often more
of a craft or vocational nature (which often
deviate significantly from the intended con-
tent of TE). An examination of the plot be-
tween the semesters of I/T/V courses taken by
males and their ITK score reveals a
curvilinear relationship. When the ITK score
is recoded into the categories of 0, 1, 2,
and 3 or more semesters, the correlation is
significantly positive. Based on course ex-
posure effects, it is concluded that there is
substantial support for the existence of in-
dustrial TL.
ITED SUBSCORE EFFECTS. The proposed
theory of TL states that the construct is a
partial subset of general achievement. It
was hypothesized that TL or more narrowly the
industrial strata should intercorrelate and
behave psychometrically in a similar manner
to other subsets of general achievement. In
this research project the other measures of
general achievement were ITED subscores. The
ITK instrument was found to intercorrelate in
a similar fashion with ITED subtest intercor-
relations. However, the ITK should not cor-
relate too highly with other subsets of
achievement if it is to be worthy of being
partitioned from the encompassing set of at-
tributes. Correcting for attenuation, the
shared variance between ITK score and ITED
subscores ranges from 36% to 56%. These per-
centages are large enough to infer a meaning-
ful relationship between the latent trait
being measured by the ITK and that being
measured by the ITED. However, the shared
variance is not so large as to negate the ITK
instrument as an additional means of measur-
ing general achievement. It is concluded
that the ITK measures a part of general
achievement, only part of which is also meas-
ured by ITED subscores.
SUMMARY
The results of the research give support
for the existence of the construct of indus-
trial technological literacy; and by extrapo-
lation, technological literacy. Results also
indicate that TL can be measured reliably.
It is concluded that TL is a subset of the
general achievement domain but is worthy of
separate consideration. Table 4 summarizes
the author's interpretation of the construct
validity evidence.
TABLE 4
STRENGTH OF CONSTRUCT VALIDITY EVIDENCE
---------------------------------------------
Total #1 #2 #3
Grade 0,2a 0 0 0
Gender 1b 0 0 0
Father 2 2 0 0
Mother 0 0 0 0
Indus./tech./vocat. courses 2 1 2 1
Q 1 0 0 0
SS 1 0 0 0
NS 2 2 0 2
RT 1 0 0 0
---------------------------------------------
a Substantial evidence when courses are recoded
as to their TE content.
b Gender has less of an affect when courses are
recoded as to their TE content.
Notes: 0 = little or no construct validity sup-
port.
1 = partial construct validity support.
2 = substantial construct validity sup-
port.
Based on responses to items in the in-
strument it is this researcher's conclusion
that there is widespread misunderstanding of
technology on the part of many high school
students. Based on observation, interviews,
and the post hoc analyses it is this re-
searcher's conclusion that the most appropri-
ate content and pedagogy to increase TL would
be those of TE.
RECOMMENDATIONS
Based on the results and the insights
gained from conducting this research the fol-
lowing additional research projects are re-
commended.
1. Revision and operational field testing of
the ITK instrument. The benefits of this
would be improvement of psychometric
properties and gathering additional va-
lidity evidence.
2. Replication of this research for cross
validation purposes.
3. Investigation of strata other than indus-
trial, e.g., agricultural, biomedical,
etc.
4. Development of alternate forms of the ITK
instrument so that alternate forms reli-
ability can be calculated.
5. Development of other means of measuring
TL and subsequent multi-method multi-
trait studies.
6. Research aimed at investigating what type
of content and/or methodology contributes
to and/or increases TL most efficiently.
There is much corroborating research
(Baker, 1989; Croft, 1990; Hameed, 1988;
Hatch, 1985) which defines and gives evi-
dence for the validity and utility of
measuring TL. This author believes this
recommendation to be most important. It
seems logical that before we try to in-
crease a student's TL we first know the
best way or ways to do so.
----------------
Michael A. Hayden is Assistant Professor, De-
partment of Technology and Education,
Mississippi State University, Mississippi
State, MS.
REFERENCES
Baker, D. (1989). Technological literacy:
The essential criteria for a definition.
DISSERTATION ABSTRACTS INTERNATIONAL,
(University Microfilms No. 89-00392)
Croft, V. E. (1990). TECHNOLOGICAL
LITERACY: CHARACTERISTICS OF A HIGH
SCHOOL GRADUATE. Paper presented at the
International Technology Education Confer-
ence, Indianapolis, Indiana.
Fleming, R. (1989). Literacy for a techno-
logical age. SCIENCE EDUCATION, 73(4),
391-404.
Hameed, A. (1988). Development of a test of
technological literacy. DISSERTATION AB-
STRACTS INTERNATIONAL, (University Micro-
films No. 88-24509)
Hatch, L. O. (1985). TECHNOLOGICAL
LITERACY: A SECONDARY ANALYSIS OF NA-
TIONAL ASSESSMENT OF EDUCATION PROGRESS
SCIENCE DATA. Unpublished doctoral dis-
sertation, University of Maryland, College
Park.
Hayden, M. A. (1989a). The development and
validation of a test of technological lit-
eracy. DISSERTATION ABSTRACTS INTERNA-
TIONAL, (University Microfilms No.
89-20136)
Hayden, M. A. (1989b). What is technolog-
ical literacy? BULLETIN OF SCIENCE, TECH-
NOLOGY AND SOCIETY, 9(4), pp. 228-233.
Ley, C. (1987). Technological Literacy and
Investigative Imperatives for vocational
education researchers. JOURNAL OF VOCA-
TIONAL EDUCATION RESEARCH, 12(1), 1-10.
Maley, D. (1985, October). ISSUES AND
TRENDS IN TECHNOLOGY EDUCATION. Paper
presented at the Technology Education Sym-
posium VII, California, Pennsylvania.
The University of Iowa (1987). MANUAL FOR
TEACHERS, ADMINISTRATORS AND COUNSELORS:
IOWA TESTS OF EDUCATIONAL DEVELOPMENT:
FORMS X-8 AND Y-8. Iowa City: Author.
Permission is given to copy any
article or graphic provided credit is given and
the copies are not intended for sale.
Journal of Technology Education Volume 3, Number 1 Fall 1991