JTE v9n1 - Choosing Qualitative Research: A Primer for Technology Education Researchers
Choosing Qualitative Research: A Primer for Technology Education Researchers
Marie C. HoepflA number of writers have commented on the dearth of substantive research within the field of technology education, and point to the expansion of its research agenda as a means of strengthening the discipline. Waetjen, in his call for good research in technology education, states that "the plea is to use experimental type research as much as possible" ( 1992, p. 30 ). Interestingly, the three areas of research need outlined in his essay would all lend themselves to alternative methodologies, including qualitative methodologies.
More recently, others have called for an expansion in the types of research methods used. Of the 220 reports included in Zuga's review of technology education-related research ( 1994 ), only 16 are identified as having used qualitative methods, and Zuga notes that many of those studies were conducted outside the United States. Johnson ( 1995 ) suggests that technology educators "engage in research that probes for deeper understanding rather than examining surface features." He notes that qualitative methodologies are powerful tools for enhancing our understanding of teaching and learning, and that they have "gained increasing acceptance in recent years" (p. 4).
There are compelling reasons for the selection of qualitative methodologies within the educational research arena, yet many people remain unfamiliar with these methods. Researchers trained in the use of quantitative designs face real challenges when called upon to use or teach qualitative research ( Stallings, 1995 ). There is, however, a growing body of literature devoted to qualitative research in education, some of which is synthesized here. The goals of this article are to elaborate on the reasons for choosing qualitative methodologies, and to provide a basic introduction to the features of this type of research.
Qualitative Versus Quantitative Research Paradigms
Researchers have long debated the relative value of qualitative and quantitative inquiry ( Patton, 1990 ). Phenomenological inquiry, or qualitative research, uses a naturalistic approach that seeks to understand phenomena in context-specific settings. Logical positivism, or quantitative research, uses experimental methods and quantitative measures to test hypothetical generalizations. Each represents a fundamentally different inquiry paradigm, and researcher actions are based on the underlying assumptions of each paradigm.
Qualitative research, broadly defined, means "any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification" ( Strauss and Corbin, 1990, p. 17 ). Where quantitative researchers seek causal determination, prediction, and generalization of findings, qualitative researchers seek instead illumination, understanding, and extrapolation to similar situations. Qualitative analysis results in a different type of knowledge than does quantitative inquiry.
Eisner points out that all knowledge, including that gained through quantitative research, is referenced in qualities, and that there are many ways to represent our understanding of the world:
There is a kind of continuum that moves from the fictional that is "true"-the novel for example-to th e highly controlled and quantitatively described scientific experiment. Work at either end of this continuum has the capacity to inform significantly. Qualitative research and evaluation are located toward the fictive end of the continuum without being fictional in the narrow sense of the term Eisner, 1991, pp. 30-31 ).
This sentiment echoes that of an earlier writer. Cronbach ( 1975 ) states that "the special task of the social scientist in each generation is to pin down the contemporary facts. Beyond that, he shares with the humanistic scholar and the artist in the effort to gain insight into contemporary relationships" (p. 126).
Cronbach claims that statistical research is not able to take full account of the many interaction effects that take place in social settings. He gives examples of several empirical "laws" that do not hold true in actual settings to illustrate this point. Cronbach states that "the time has come to exorcise the null hypothesis," because it ignores effects that may be important, but that are not statistically significant (1975, p. 124). Qualitative inquiry accepts the complex and dynamic quality of the social world.
However, it is not necessary to pit these two paradigms against one another in a competing stance. Patton ( 1990 ) advocates a "paradigm of choices" that seeks " methodological appropriateness as the primary criterion for judging methodological quality." This will allow for a "situational responsiveness" that strict adherence to one paradigm or another will not (p. 39). Furthermore, some researchers believe that qualitative and quantitative research can be effectively combined in the same research project ( Strauss and Corbin, 1990; Patton, 1990 ). For example, Russek and Weinberg ( 1993 ) claim that by using both quantitative and qualitative data, their study of technology-based materials for the elementary classroom gave insights that neither type of analysis could provide alone.
Basis for the Use of a Qualitative Methodology
There are several considerations when deciding to adopt a qualitative research methodology. Strauss and Corbin ( 1990 ) claim that qualitative methods can be used to better understand any phenomenon about which little is yet known. They can also be used to gain new perspectives on things about which much is already known, or to gain more in-depth information that may be difficult to convey quantitatively. Thus, qualitative methods are appropriate in situations where one needs to first identify the variables that might later be tested quantitatively, or where the researcher has determined that quantitative measures cannot adequately describe or interpret a situation. Research problems tend to be framed as open-ended questions that will support discovery of new information. Greene's 1994 study of women in the trades, for example, asked " What personal characteristics do tradeswomen have in common? In what way, if any, did role models contribute to women's choices to work in the trades?"
(p. 524a).
The ability of qualitative data to more fully describe a phenomenon is an important consideration not only from the researcher's perspective, but from the reader's perspective as well. "If you want people to understand better than they otherwise might, provide them information in the form in which they usually experience it" ( Lincoln and Guba, 1985, p. 120 ). Qualitative research reports, typically rich with detail and insights into participants' experiences of the world, "may be epistemologically in harmony with the reader's experience" ( Stake, 1978, p. 5 ) and thus more meaningful.
Features of Qualitative Research
Several writers have identified what they consider to be the prominent characteristics of qualitative, or naturalistic, research (see, for example: Bogdan and Biklen, 1982; Lincoln and Guba, 1985; Patton, 1990; Eisner, 1991 ). The list that follows represents a synthesis of these authors' descriptions of qualitative research:
1. Qualitative research uses the natural setting as the source of data. The researcher attempts to observe, describe and interpret settings as they are, maintaining what Patton calls an "empathic neutrality" (1990, p. 55).
2. The researcher acts as the "human instrument" of data collection.
3. Qualitative researchers predominantly use inductive data analysis.
4. Qualitative research reports are descriptive, incorporating expressive language and the "presence of voice in the text" (Eisner, 1991, p. 36).
5. Qualitative research has an interpretive character, aimed at discovering the meaning events have for the individuals who experience them, and the interpretations of those meanings by the researcher.
6. Qualitative researchers pay attention to the idiosyncratic as well as the pervasive, seeking the uniqueness of each case.
7. Qualitative research has an emergent (as opposed to predetermined) design, and researchers focus on this emerging process as well as the outcomes or product of the research.
8. Qualitative research is judged using special criteria for trustworthiness (these will be discussed in some detail in a later section).
Patton (1990) points out that these are not "absolute characteristics of qualitative inquiry, but rather strategic ideals that provide a direction and a framework for developing specific designs and concrete data collection tactics" (p. 59). These characteristics are considered to be "interconnected" (Patton, 1990, p. 40) and "mutually reinforcing" (Lincoln and Guba, 1985, p. 39).
It is important to emphasize the emergent nature of qualitative research design. Because the researcher seeks to observe and interpret meanings in context, it is neither possible nor appropriate to finalize research strategies before data collection has begun (Patton, 1990). Qualitative research proposals should, however, specify primary questions to be explored and plans for data collection strategies.
The particular design of a qualitative study depends on the purpose of the inquiry, what information will be most useful, and what information will have the most credibility. There are no strict criteria for sample size (Patton, 1990). "Qualitative studies typically employ multiple forms of evidence....[and] there is no statistical test of significance to determine if results 'count'" (Eisner, 1991, p. 39). Judgments about usefulness and credibility are left to the researcher and the reader.
The Role of the Researcher in Qualitative Inquiry
Before conducting a qualtitative study, a researcher must do three things. First, (s)he must adopt the stance suggested by the characteristics of the naturalist paradigm. Second, the researcher must develop the level of skill appropriate for a human instrument, or the vehicle through which data will be collected and interpreted. Finally, the researcher must prepare a research design that utilizes accepted strategies for naturalistic inquiry (Lincoln and Guba, 1985).
Glaser and Strauss ( 1967 ) and Strauss and Corbin ( 1990 ) refer to what they call the "theoretical sensitivity" of the researcher. This is a useful concept with which to evaluate a researcher's skill and readiness to attempt a qualitative inquiry.
Theoretical sensitivity refers to a personal quality of the researcher. It indicates an awareness of the subtleties of meaning of data. ...[It] refers to the attribute of having insight, the ability to give meaning to data, the capacity to understand, and capability to separate the pertinent from that which isn't ( Strauss and Corbin, 1990, p. 42 ).
Strauss and Corbin believe that theoretical sensitivity comes from a number of sources, including professional literature, professional experiences, and personal experiences. The credibility of a qualitative research report relies heavily on the confidence readers have in the researcher's ability to be sensitive to the data and to make appropriate decisions in the field ( Eisner, 1991; Patton, 1990 ).
Lincoln and Guba ( 1985 ) identify the characteristics that make humans the "instrument of choice" for naturalistic inquiry. Humans are responsive to environmental cues, and able to interact with the situation; they have the ability to collect information at multiple levels simultaneously; they are able to perceive situations holistically; they are able to process data as soon as they become available; they can provide immediate feedback and request verification of data; and they can explore atypical or unexpected responses.
Research Design and Data Collection Strategies
Eisner ( 1991 ) claims there is a "paucity of methodological prescriptions" for qualitative research, because such inquiry places a premium on the strengths of the researcher rather than on standardization (p. 169). Lincoln and Guba ( 1985 ) provide a fairly detailed outline for the design of naturalistic inquiry, which includes these general steps:
1. Determine a focus for the inquiry. This should establish a boundary for the study, and provide inclusion/exclusion criteria for new information. Boundaries, however, can be altered, and typically are.
2. Determine the fit of the research paradigm to the research focus. The researcher must compare the characteristics of the qualitative paradigm with the goals of the research.
3. Determine where and from whom data will be collected.
4. Determine what the successive phases of the inquiry will be. Phase one, for example, might feature open-ended data collection, while successive phases will be more focused.
5. Determine what additional instrumentation may be used, beyond the researcher as the human instrument.
6. Plan data collection and recording modes. This must include how detailed and specific research questions will be, and how faithfully data will be reproduced.
7. Plan which data analysis procedures will be used.
8. Plan the logistics of data collection, including scheduling and budgeting.
9. Plan the techniques that will be used to determine trustworthiness.
Steps one and two have been addressed in previous sections; the remaining steps will be addressed below.
Sampling Strategies for Qualitative Researchers
In quantitative inquiry, the dominant sampling strategy is probability sampling, which depends on the selection of a random and representative sample from the larger population. The purpose of probability sampling is subsequent generalization of the research findings to the population. By contrast, purposeful sampling is the dominant strategy in qualitative research. Purposeful sampling seeks information-rich cases which can be studied in depth ( Patton, 1990 ).
Patton identifies and describes 16 types of purposeful sampling. These include: extreme or deviant case sampling; typical case sampling; maximum variation sampling; snowball or chain sampling; confirming or disconfirming case sampling; politically important case sampling; convenience sampling; and others ( 1990, pp. 169-183 ). According to Lincoln and Guba ( 1985 ), the most useful strategy for the naturalistic approach is maximum variation sampling. This strategy
aims at capturing and describing the central themes or principal outcomes that cut across a great deal of participant or program variation. For small samples a great deal of heterogeneity can be a problem because individual cases are so different from each other. The maximum variation sampling strategy turns that apparent weakness into a strength by applying the following logic: Any common patterns that emerge from great variation are of particular interest and value in capturing the core experiences and central, shared aspects or impacts of a program ( Patton, 1990, p. 172 ).
Maximum variation sampling can yield detailed descriptions of each case, in addition to identifying shared patterns that cut across cases. See Hoepfl ( 1994 ) for an illustration of this strategy applied to technology education research. Several examples of studies employing case sampling can also be found in the technology education literature (see Brown, 1995; Hansen, 1995; and Lewis, 1995 and 1997 )
In spite of the apparent flexibility in purposeful sampling, researchers must be aware of three types of sampling error that can arise in qualitative research. The first relates to distortions caused by insufficient breadth in sampling; the second from distortions introduced by changes over time; and the third from distortions caused by lack of depth in data collection at each site ( Patton, 1990 ).
Data Collection Techniques
The two prevailing forms of data collection associated with qualitative inquiry are interviews and observation.
Interviews
Qualitative interviews may be used either as the primary strategy for data collection, or in conjunction with observation, document analysis, or other techniques ( Bogdan and Biklen, 1982 ). Qualitative interviewing utilizes open-ended questions that allow for individual variations. Patton ( 1990 ) writes about three types of qualitative interviewing: 1) informal, conversational interviews; 2) semi-structured interviews; and 3) standardized, open-ended interviews.
An interview guide or "schedule" is a list of questions or general topics that the interviewer wants to explore during each interview. Although it is prepared to insure that basically the same information is obtained from each person, there are no predetermined responses, and in semi-structured interviews the interviewer is free to probe and explore within these predetermined inquiry areas. Interview guides ensure good use of limited interview time; they make interviewing multiple subjects more systematic and comprehensive; and they help to keep interactions focused. In keeping with the flexible nature of qualitative research designs, interview guides can be modified over time to focus attention on areas of particular importance, or to exclude questions the researcher has found to be unproductive for the goals of the research ( Lofland and Lofland, 1984 ).
Recording Data . A basic decision going into the interview process is how to record interview data. Whether one relies on written notes or a tape recorder appears to be largely a matter of personal preference. For instance, Patton says that a tape recorder is "indispensable" (1990, p. 348), while Lincoln and Guba "do not recommend recording except for unusual reasons" (1985, p. 241). Lincoln and Guba base their recommendation on the intrusiveness of recording devices and the possibility of technical failure. Recordings have the advantage of capturing data more faithfully than hurriedly written notes might, and can make it easier for the researcher to focus on the interview.
Observations
The classic form of data collection in naturalistic or field research is observation of participants in the context of a natural scene. Observational data are used for the purpose of description-of settings, activities, people, and the meanings of what is observed from the perspective of the participants. Observation can lead to deeper understandings than interviews alone, because it provides a knowledge of the context in which events occur, and may enable the researcher to see things that participants themselves are not aware of, or that they are unwilling to discuss ( Patton, 1990 ). A skilled observer is one who is trained in the process of monitoring both verbal and nonverbal cues, and in the use of concrete, unambiguous, descriptive language. Sours' ( 1997 ) study of teaching and learning styles provides a good example of descriptive language applied to the technology classroom.
There are several observation strategies available. In some cases it may be possible and desirable for the researcher to watch from outside, without being observed. Another option is to maintain a passive presence, being as unobtrusive as possible and not interacting with participants. A third strategy is to engage in limited interaction, intervening only when further clarification of actions is needed. Or the researcher may exercise more active control over the observation, as in the case of a formal interview, to elicit specific types of information. Finally, the researcher may act as a full participant in the situation, with either a hidden or known identity. Each of these strategies has specific advantages, disadvantages and concerns which must be carefully examined by the researcher ( Schatzman and Strauss, 1973 ).
The presence of an observer is likely to introduce a distortion of the natural scene which the researcher must be aware of, and work to minimize. Critical decisions, including the degree to which researcher identity and purposes will be revealed to participants, the length of time spent in the field, and specific observation techniques used, are wholly dependent on the unique set of questions and resources brought to each study. In any case, the researcher must consider the legal and ethical responsibilities associated with naturalistic observation.
Recording Data . Field researchers rely most heavily on the use of field notes, which are running descriptions of settings, people, activities, and sounds. Field notes may include drawings or maps. Acknowledging the difficulty of writing extensive field notes during an observation, Lofland and Lofland ( 1984 ) recommend jotting down notes that will serve as a memory aid when full field notes are constructed. This should happen as soon after observation as possible, preferably the same day. In addition to field notes, researchers may use photographs, videotapes, and audio tapes as means of accurately capturing a setting.
Gaining Access and Researcher Obligations
Based on their experience with naturalistic research, Lofland and Lofland ( 1984 ) believe that researchers are more likely to gain successful access to situations if they make use of contacts that can help remove barriers to entrance; if they avoid wasting respondents' time by doing advance research for information that is already part of the public record; and if they treat respondents with courtesy. Because naturalistic researchers are asking participants to "grant access to their lives, their minds, [and] their emotions," it is also important to provide respondents with a straightforward description of the goals of the research (p. 25).
Other Sources of Data
Another source of information that can be invaluable to qualitative researchers is analysis of documents. Such documents might include official records, letters, newspaper accounts, diaries, and reports, as well as the published data used in a review of literature. In his study of technology teachers in training, Hansen ( 1995 ) analyzed journal entries and memos written by participants, in addition to interviews. Hoepfl ( 1994 ), in her study of closure of technology teacher education programs, used newspaper reports, university policy documents, and department self-evaluation data, where available, to supplement data gained through interviews.
There are some specialized forms of qualitative research which rely solely on analysis of documents. For example, Gagel ( 1997 ) used a process known as hermeneutic inquiry to investigate the literature on both literacy and technology. Patton ( 1990 ) provides a good overview of the various theoretical orientations that inform the "rich menu of alternative possibilities within qualitative research" (p. 65).
Deciding When to Stop Sampling
Qualitative researchers have few strict guidelines for when to stop the data collection process. Criteria include: 1) exhaustion of resources; 2) emergence of regularities; and 3) overextension, or going too far beyond the boundaries of the research ( Guba, 1978 ). The decision to stop sampling must take into account the research goals, the need to achieve depth through triangulation of data sources, and the possibility of greater breadth through examination of a variety of sampling sites.
Analysis of Data
Bogdan and Biklen define qualitative data analysis as "working with data, organizing it, breaking it into manageable units, synthesizing it, searching for patterns, discovering what is important and what is to be learned, and deciding what you will tell others" ( 1982, p. 145 ). Qualitative researchers tend to use inductive analysis of data, meaning that the critical themes emerge out of the data ( Patton, 1990 ). Qualitative analysis requires some creativity, for the challenge is to place the raw data into logical, meaningful categories; to examine them in a holistic fashion; and to find a way to communicate this interpretation to others.
Sitting down to organize a pile of raw data can be a daunting task. It can involve literally hundreds of pages of interview transcripts, field notes and documents. The mechanics of handling large quantities of qualitative data can range from physically sorting and storing slips of paper to using one of the several computer software programs that have been designed to aid in this task (see Brown, 1996, for a description of one of these programs ).
Analysis begins with identification of the themes emerging from the raw data, a process sometimes referred to as "open coding" ( Strauss and Corbin, 1990 ). During open coding, the researcher must identify and tentatively name the conceptual categories into which the phenomena observed will be grouped. The goal is to create descriptive, multi-dimensional categories which form a preliminary framework for analysis. Words, phrases or events that appear to be similar can be grouped into the same category. These categories may be gradually modified or replaced during the subsequent stages of analysis that follow.
As the raw data are broken down into manageable chunks, the researcher must also devise an "audit trail"-that is, a scheme for identifying these data chunks according to their speaker and the context. The particular identifiers developed may or may not be used in the research report, but speakers are typically referred to in a manner that provides a sense of context (see, for example, Brown, 1996; Duffee and Aikenhead, 1992; and Sours, 1997 ). Qualititative research reports are characterized by the use of "voice" in the text; that is, participant quotes that illustrate the themes being described.
The next stage of analysis involves re-examination of the categories identified to determine how they are linked, a complex process sometimes called "axial coding" ( Strauss and Corbin, 1990 ). The discrete categories identified in open coding are compared and combined in new ways as the researcher begins to assemble the "big picture." The purpose of coding is to not only describe but, more importantly, to acquire new understanding of a phenomenon of interest. Therefore, causal events contributing to the phenomenon; descriptive details of the phenomenon itself; and the ramifications of the phenomenon under study must all be identified and explored. During axial coding the researcher is responsible for building a conceptual model and for determining whether sufficient data exists to support that interpretation.
Finally, the researcher must translate the conceptual model into the story line that will be read by others. Ideally, the research report will be a rich, tightly woven account that "closely approximates the reality it represents" ( Strauss and Corbin, 1990, p. 57 ). Many of the concerns surrounding the presentation of qualitative research reports are discussed in the section "Judging Qualitative Research" which follows.
Although the stages of analysis are described here in a linear fashion, in practice they may occur simultaneously and repeatedly. During axial coding the researcher may determine that the initial categories identified must be revised, leading to re-examination of the raw data. Additional data collection may occur at any point if the researcher uncovers gaps in the data. In fact, informal analysis begins with data collection, and can and should guide subsequent data collection. For a more detailed yet very understandable description of the analysis process, see Simpson and Tuson ( 1995) .
The Product of Qualitative Data Analysis
In their classic text Discovery of Grounded Theory , Glaser and Strauss ( 1967,/a>) describe what they believe to be the primary goal of qualitative research: the generation of theory, rather than theory testing or mere description. According to this view, theory is not a "perfected product" but an "ever-developing entity" or process (p. 32). Glaser and Strauss claim that one of the requisite properties of grounded theory is that it be "sufficiently general to be applicable to a multitude of diverse situations within the substantive area" (p. 237).
The grounded theory approach described by Glaser and Strauss represents a somewhat extreme form of naturalistic inquiry. It is not necessary to insist that the product of qualitative inquiry be a theory that will apply to a "multitude of diverse situations." Examples of a more flexible approach to qualitative inquiry can be gained from a number of sources. For example, both Patton ( 1990 ) and Guba ( 1978 ) state, in the same words, that "naturalistic inquiry is always a matter of degree" of the extent to which the researcher influences responses and imposes categories on the data. The more "pure" the naturalistic inquiry, the less reduction of data into categories.
Figure 1 illustrates one interpretation of the relationship between description, verification, and generation of theory-or, in this case, the development of what Cronbach ( 1975 ) calls "working hypotheses, " which suggests a more tractable form of analysis than the word "theory." According to this interpretation, a researcher may move between points on the description/ verification continuum during analysis, but the final product will fall on one particular point, depending on the degree to which it is naturalistic.
Figure 1. Description, verification and generation of working hypotheses in qualitative research.
In keeping with a naturalistic stance, the researcher might conclude that, to the extent that findings are based on information from a variety of diverse situations, they may be applicable to a larger substantive area. However, their applicability to a particular situation is wholly dependent upon the conditions of the situation and the usefulness of the research findings to individual readers.
Judging Qualitative Research
The Role of the Reader
Those who are in a position to judge or use the findings of a qualitative inquiry must play a different type of role than people who review quantitative research. This is because "there are no operationally defined truth tests to apply to qualitative research" ( Eisner, 1991, p. 53 ). Instead, researcher and readers "share a joint responsibility" for establishing the value of the qualitative research product ( Glaser and Strauss, 1967, p. 232 ). "Pragmatic validation [of qualitative research] means that the perspective presented is judged by its relevance to and use by those to whom it is presented: their perspective and actions joined to the [researcher's] perspective and actions" ( Patton, 1990, p. 485 ).
Eisner ( 1991 ) believes that the following three features of qualitative research should be considered by reviewers:
Coherence : Does the story make sense? How have the conclusions been supported? To what extent have multiple data sources been used to give credence to the interpretation that has been made? (p. 53).
Related to coherence is the notion of "structural corroboration," also known as triangulation (p. 55).
Consensus : The condition in which the readers of a work concur that the findings and/or interpretations reported by the investigator are consistent with their own experience or with the evidence presented (p. 56).
Finally, reviewers must assess the report's:
Instrumental Utility : The most important test of any qualitative study is its usefulness. A good qualitative study can help us understand a situation that would otherwise be enigmatic or confusing (p. 58).
A good study can help us anticipate the future, not in the predictive sense of the word, but as a kind of road map or guide. "Guides call our attention to aspects of the situation or place we might otherwise miss" ( Eisner, 1991, p. 59 ).
Addressing Trustworthiness in Qualitative Research
The basic question addressed by the notion of trustworthiness, according to Lincoln and Guba, is simple: "How can an inquirer persuade his or her audiences that the research findings of an inquiry are worth paying attention to?" (1985, p. 290). When judging qualitative work, Strauss and Corbin ( 1990 ) believe that the "usual canons of 'good science'...require redefinition in order to fit the realities of qualitative research" (p. 250). Lincoln and Guba ( 1985, p. 300 ) have identified one alternative set of criteria that correspond to those typically employed to judge quantitative work (see Table 1).
Table 1
Comparison of criteria for judging the quality of quantitative versus qualitative research
Conventional terms
Naturalistic terms
internal validity
credibility
external validity
transferability
reliability
dependability
objectivity
confirmability
Smith and Heshusius ( 1986 ) sharply criticize those writers, like Lincoln and Guba, who they believe have adopted a stance of "detente" with rationalists. They are particularly incensed by Lincoln and Guba's use of "comparable criteria," which to their eyes look little different than the conventional criteria they supposedly replace. In either case, there must be a "belief in the assumption that what is known-be it an existent reality or an interpreted reality-stands independent of the inquirer and can be described without distortion by the inquirer" (p. 6). Smith and Heshusius claim that naturalistic research can offer only an "interpretation of the interpretations of others, " and that to assume an independent reality is "unacceptable" for the qualitative researcher (p. 9).
Their stance is a strong one, because the only reality it accepts is a completely mind-dependent one, which will vary from individual to individual; in other words, for Smith and Heshusius, there is no "out there" out there. For these researchers, it would not be possible to choose a best interpretation from among the many available, because no technique or interpretation can be "epistemologically privileged" (p. 9). To maintain this stance would seem to negate the value of doing research at all, because it prohibits the possibility of reconciling alternative interpretations.
Therefore, it is important to determine which criteria are consistent with the naturalistic paradigm, yet which allow for a declaration that "good science" has been carried out. In the following sections, conventional and naturalistic criteria will be discussed, with the goal of selecting criteria which are appropriate for judging the overall trustworthiness of a qualitative study.
Internal Validity versus Credibility
In conventional inquiry, internal validity refers to the extent to which the findings accurately describe reality. Lincoln and Guba ( 1985 ) state that "the determination of such isomorphism is in principle impossible" ( p. 294 ), because one would have to know the "precise nature of that reality" and, if one knew this already, there would be no need to test it ( p. 295 ). The conventional researcher must postulate relationships and then test them; the postulate cannot be proved, but only falsified. The naturalistic researcher, on the other hand, assumes the presence of multiple realities and attempts to represent these multiple realities adequately. Credibility becomes the test for this.
Credibility depends less on sample size than on the richness of the information gathered and on the analytical abilities of the researcher ( Patton, 1990 ). It can be enhanced through triangulation of data. Patton identifies four types of triangulation: 1) methods triangulation; 2) data triangulation; 3) triangulation through multiple analysts; and 4) theory triangulation. Other techniques for addressing credibility include making segments of the raw data available for others to analyze, and the use of "member checks," in which respondents are asked to corroborate findings ( Lincoln and Guba, 1985, pp. 313-316 ).
External Validity / Generalizability versus Transferability
In conventional research, external validity refers to the ability to generalize findings across different settings. Making generalizations involves a trade-off between internal and external validity ( Lincoln and Guba, 1985 ). That is, in order to make generalizable statements that apply to many contexts, one can include only limited aspects of each local context.
Lincoln and Guba ( 1985 ) admit that generalizability is "an appealing concept," because it allows a semblance of prediction and control over situations ( pp. 110-111 ). Yet they suggest that the existence of local conditions "makes it impossible to generalize" (p. 124). Cronbach ( 1975 ) discusses the problem by saying:
The trouble, as I see it, is that we cannot store up generalizations and constructs for ultimate assembly into a network. It is as if we needed a gross of dry cells to power an engine and could only make one a month. The energy would leak out of the first cells before we had half the battery completed ( p. 123 ).
According to Cronbach, "when we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion" ( p. 125 ).
In the naturalistic paradigm, the transferability of a working hypothesis to other situations depends on the degree of similarity between the original situation and the situation to which it is transferred. The researcher cannot specify the transferability of findings; he or she can only provide sufficient information that can then be used by the reader to determine whether the findings are applicable to the new situation ( Lincoln and Guba, 1985 ). Other writers use similar language to describe transferability, if not the word itself. For example, Stake ( 1978 ) refers to what he calls "naturalistic generalization" (p. 6). Patton suggests that "extrapolation" is an appropriate term for this process (1990, p. 489). Eisner says it is a form of "retrospective generalization" that can allow us to understand our past (and future) experiences in a new way (1991, p. 205).
Reliability versus Dependability
Kirk and Miller ( 1986 ) identify three types of reliability referred to in conventional research, which relate to: 1) the degree to which a measurement, given repeatedly, remains the same; 2) the stability of a measurement over time; and 3) the similarity of measurements within a given time period ( pp. 41-42 ). They note that "issues of reliability have received little attention" from qualitative researchers, who have instead focused on achieving greater validity in their work (p. 42). Although they give several examples of how reliability might be viewed in qualitative work, the essence of these examples can be summed up in the following statement by Lincoln and Guba ( 1985 ): "Since there can be no validity without reliability (and thus no credibility without dependability), a demonstration of the former is sufficient to establish the latter" (p. 316).
Nevertheless, Lincoln and Guba do propose one measure which might enhance the dependability of qualitative research. That is the use of an "inquiry audit," in which reviewers examine both the process and the product of the research for consistency ( 1985, p. 317 ).
Objectivity versus Confirmability
Conventional wisdom says that research which relies on quantitative measures to define a situation is relatively value-free, and therefore objective. Qualitative research, which relies on interpretations and is admittedly value-bound, is considered to be subjective. In the world of conventional research, subjectivity leads to results that are both unreliable and invalid. There are many researchers, however, who call into question the true objectivity of statistical measures and, indeed, the possibility of ever attaining pure objectivity at all ( Lincoln and Guba, 1985; Eisner, 1991 ).
Patton ( 1990 ) believes that the terms objectivity and subjectivity have become "ideological ammunition in the paradigms debate." He prefers to "avoid using either word and to stay out of futile debates about subjectivity versus objectivity." Instead, he strives for "empathic neutrality" (p. 55). While admitting that these two words appear to be contradictory, Patton points out that empathy "is a stance toward the people one encounters, while neutrality is a stance toward the findings" (p. 58). A researcher who is neutral tries to be non-judgmental, and strives to report what is found in a balanced way.
Lincoln and Guba ( 1985 ) choose to speak of the "confirmability" of the research. In a sense, they refer to the degree to which the researcher can demonstrate the neutrality of the research interpretations, through a "confirmability audit." This means providing an audit trail consisting of 1) raw data; 2) analysis notes; 3) reconstruction and synthesis products; 4) process notes; 5) personal notes; and 6) preliminary developmental information ( pp. 320 -321 ).
With regard to objectivity in qualitative research, it may be useful to turn to Phillips ( 1990 ), who questions whether there is really much difference between quantitative and qualitative research:
Bad work of either kind is equally to be deplored; and good work of either kind is still-at best-only tentative. But the good work in both cases will be objective, in the sense that it has been opened up to criticism, and the reasons and evidence offered in both cases will have withstood serious scrutiny. The works will have faced potential refutation, and insofar as they have survived, they will be regarded as worthy of further investigation (p. 35).
Discussion and Conclusion
The increased interest in qualitative research in recent years warrants a basic understanding of this paradigm on the part of all technology education researchers. This overview of qualitative research methods and issues represents a starting point only for those who are interested in using and/or reviewing qualitative research. Readers can choose from a growing body of literature on the topic for further guidance.
The decision to use qualitative methodologies should be considered carefully; by its very nature, qualitative research can be emotionally taxing and extraordinarily time consuming. At the same time, it can yield rich information not obtainable through statistical sampling techniques.
In the past, graduate students contemplating the use of qualitative inquiry were told that they would have to "sell" the idea to members of their research committees, who would probably view qualitative research as inferior to quantitative research. Fortunately, in most universities that belief has changed, to the point where qualitative research is the paradigm of choice in some schools. In spite of this growing acceptance, new researchers may still encounter difficulties in finding faculty advisors who are skilled in this type of research.
Qualitative researchers have a special responsibility to their subjects and their readers. Since there are no statistical tests for significance in qualitative studies, the researcher bears the burden of discovering and interpreting the importance of what is observed, and of establishing a plausible connection between what is observed and the conclusions drawn in the research report. To do all of this skillfully requires a solid understanding of the research paradigm and, ideally, guided practice in the use of qualitative observation and analysis techniques.
There are many useful research designs, the selection of which depends on the research questions being asked. Most importantly, technology educators must rise to the challenge to find and use rigorous, appropriate research techniques that address the significant questions facing the field.
References
Brown, D. C. (1996). Why ask why: patterns and themes of causal attribution in the workplace. Journal of Industrial Teacher Education , 33 (4), 47-65.
Gagel, C. (1997). Literacy and technology: reflections and insights for technological literacy. Journal of Industrial Teacher Education , 34 (3), 6-34.
Hansen, R. E. (1995). Teacher socialization in technological education. Journal of Technology Education , 6 (2), 34-45.
Johnson, S. D. (1995, Spring). Will our research hold up under scrutiny? Journal of Industrial Teacher Education , 32 (3), 3-6.
Lewis, T. (1997). Impact of technology on work and jobs in the printing industry - implications for vocational curriculum. Journal of Industrial Teacher Education , 34 (2), 7-28.
Lewis, T. (1995). Inside three workplace literacy initiatives: possibilities and limits of vocational institutions. Journal of Industrial Teacher Education , 33 (1), 60-82.
Schatzman, L., & Strauss, A. L. (1973). Field research . Englewood Cliffs, N.J.: Prentice-Hall, Inc.
Marie Hoepfl is Assistant Professor in the Department of Technology at Appalachian State University, Boone, NC.