JCTE Logo

Journal of Career and Technical Education

Current Co-Editors:
Edward C. Fletcher Jr.   ecfletcher@usf.edu
Victor M. Hernandez-Gantes   victorh@usf.edu

Volume 20, Number 2
Spring 2004

DLA Ejournal Home | JCTE Home | Table of Contents for this issue | Search JCTE and other ejournals

THE ART AND POLITICS OF PEER REVIEW

Jay W. Rojewski
Desirae M. Domenico
University of Georgia



ABSTRACT

Peer reviewed manuscripts contain a certain level of merit, as they have survived the scrutiny of reviewers who possess some expertise in the relevant area. This article discusses the purpose of reviewing manuscripts for publication in scholarly journals. Various aspects and issues of the peer review process are described, including reviewer responsibilities. Criticisms and concerns associated with the current peer review system are addressed, as well as guidelines for reviewing manuscripts. Finally, recommendations for improving the peer review process are provided.

INTRODUCTION

The publication of research and scholarly papers plays an integral role in advancing our understanding of and ability to act on issues related to workforce preparation and career development. At its very foundation, scholarly publication, regardless of field, rests on the editorial and peer review system. It is ". . . an inevitable part of the process of getting published" (Hawkey, 2001, p. 65).

The process of soliciting peers (experts) to evaluate scholarly work prior to publication was initiated by Henry Oldenburg, editor of the first scientific journal Philosophical Transactions, as a means of providing a method for reporting experimental results that encouraged submissions while lessening the chances of one's work being stolen (Berkenkotter, 1995). Despite its tenuous beginnings, and while not without critics or limitations, the review process today is intended to nurture ". . . a communal trust in the publication decision by creating a unique formal consultation among authors, editors, and reviewers, or 'referees', about the merits, scope, style, methods, substance, and knowledge claims of a potential article" (Chubin & Hackett, 1990, p. 84). The idea of nurturing trust and collegiality through peer review was reinforced by Young (2003) who suggests that the primary aims of peer review are to select/reject manuscripts for publication and make the entire process transparent, accurate, and practical. Benos, Kirk, and Hall (2003) offer even more specific direction.

The purpose of peerreview is to ensure 1) quality, checking that no mistakes in procedure or logic have been made; 2) that the results presented support the conclusion drawn; 3) that no errors in citations to previous work have been made; 4) that all human and animal protocols conducted follow proper review and approval by appropriate institutional review committees; and, very importantly, 5) that the work is original and significant. (p. 48)

Typically, the peer review process used by scholarly journals is fairly uniform. On receipt, editors first determine the overall quality and general suitability of the manuscript. If judged to be a poor fit with the journal or representing "sloppy, shoddy work," the manuscript is rejected without review and returned to the author (Uchiyamam, Simnone, & Borko, 1999). While some argue that this action injects an unwanted element of bias and unbalanced control, the review process necessarily represents both objective and subjective elements (Coelho & LaForge, 2000). Murray and Raths (1996) supported the need for fairness and balance (what they called democratic principles) in the review process, and acknowledged that ". . . scholarship is undoubtedly enhanced by democratic principles." Even so, they were quick to point out ". . . scholarly work is not determined by majority vote, nor are editorial judgments simply a matter of tallying votes" (p. 419).

Another subjective aspect of the peer review process is the selection of reviewers. Typically, the editor selects three (or more) professionals based on some combination of the following qualities: familiarity with the topic, diversity, skill with the review process, sensitivity, honesty, and timeliness While reviewing a manuscript is work, Benos et al. (2003) consider the task ". . . an honor, not only because you are being recognized for your eminence in a particular area of research but also because of the responsibility and service you provide to the journal and the scientific community" (p. 47).

Obviously, reviewers should be knowledgeable about the topic and have a clear understanding of the historical context in which the work was done (Benos et al., 2003). Sometimes a diverse group of reviewers is purposefully sought to gather opinions from various, perhaps even opposing, perspectives. This approach has advantages, particularly when critiquing qualitative or mixed methods designs. However, "it can wreak havoc for the writer seeking to publish who happens to get reviewers who have little regard for (or outright hostility to) the writer's approach, methods, or theoretical/disciplinary framework" (Berkenkotter, 1995, p. 246). For the most part, however, a diverse mix of opinions and perspectives is healthy and provides for rigorous and constructive critiques. Further, assembling a like-minded group of scholars does not guarantee comparable conclusions. In fact, it is rare to have a manuscript unanimously recommended for publication without revision (Murray & Raths, 1996).

The importance of peer review should not be underestimated. In fact, the results of this process, i.e., published manuscripts, serve as a public record of our field in terms of interests, emphasis and contributions to workforce preparation, and the formation and evolution of discipline and professionalization (Benos et al., 2003; Uchiyamam et al., 1999; Young, 2003). Further, peer reviewers and editors are often described as gatekeepers who monitor and construct the type and quality of new knowledge entering the field and, perhaps, in the process preventing competing ideas from being published (Berkenkotter, 1995; Taylor, Beck, & Ainsworth, 2001). Yet, despite its importance, a relatively limited body of literature exists on peer review, including the specific criteria used to determine which manuscripts are selected for publication (Coelho & LaForge, 2000; Rowney & Zenisek, 1980). What we do know is that many professionals lack an adequate working knowledge of how to effectively critique research, particularly novice scholars (Seals & Tanaka, 2000). Unfortunately, formal training in effective analysis of research manuscripts is rarely, if ever, available (Benos et al.). To address this concern, the remainder of this article focuses on select aspects of the editorial review process. We will, in turn, identify primary criticisms and concerns about the current system, offer guidelines for reviewing scholarly manuscripts, and give recommendations for improving the peer review system in use in the vast majority of scholarly journals, including the Journal of Career and Technical Education.

CRITICISMS

Criticisms of the peer review process are fairly easy to find in the literature and are often quite harsh. Take this assessment of peer review made by Ponsi (2003) as an example.
Despite its wide acceptance, peer review has been subjected to a variety of criticisms: the evaluation procedures are often inadequately performed, and in general it can be said that research on the peer review process does not provide unquestionable evidence of its value. This time-consuming and resource-intensive process is slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud. (p. 444)

Ouch! We wonder what Ponsi really thinks about the process. But wait, there's more. A decade earlier, Bornstein (1990) decried the review process as ". . . unreliable, unconstructive, and biased in a number of ways (e.g., biased against nonsignificant findings, against replications of previous work, against unknown authors and less prestigious institutions, and against unpopular or counterintuitive findings)" (p. 672).

Okay, if it's really so bad, then why continue using it? Frankly, the answer comes down to this: Despite the flaws, we really don't have anything better (Young, 2003). Berkenkotter (1995) agrees, but sees the review process more in terms of a social mechanism that allows a discipline's experts to maintain quality control over new knowledge entering a field. "While the peer review is not infallible, it remains the primary means through which authority and authenticity are conferred upon scientific and scholarly papers by journal editors and the expert judges they have consulted" (p. 245). Arrington (1995) took a different perspective.

If given all of this, a submission is recommended for publication by two or more blind reviewers, chances are it's because it succeeds in spite of these complications and may stand a better chance of being considered important and valuable by other interested and knowledgeable readers. (p. 252)

In many ways, we concur with the observations of these authors. There is little doubt that the review process contains flaws. Surely, most everyone reading this article can attest to some unpleasant or frustrating experience incurred while trying to publish in a scholarly journal. However, we believe that the review process can work effectively and provides a fairly reliable mechanism for determining the quality of manuscripts for publication. One key to the success of the review process is the journal editor who must be vigilant about the subjective elements of the review process. Contrary to the belief of some, we don't believe it is possible to eliminate subjectivity from the decision-making process. However, the editor can ensure that the overall process is fair. A first step in ensuring fairness is to identify criticisms of the process and then minimize the likelihood of their occurrence.

Given the partially subjective nature of the review process, a number of criticisms have surfaced over the years. We address several of the more common ones are addressed in this section including a lack of understanding of reviewers' (and editors') proper role, the role of politics and conflict of interest, allowing emotion to enter into the decisionmaking process, and unequal treatment of various manuscript sections by reviewers. Although we focus on criticisms levied against the review process, we are encouraged by the optimism of Benos et al. (2003) who asserted that specific review techniques can be nurtured and learned. Perhaps understanding the problems is one of the first steps in that direction.

Who's Advocate?

What is the role of manuscript reviewers? Should they be advocates for journals or for authors? The basic argument goes something like this: Reviewers are in a unique position to provide manuscript authors with guidance about unexplored aspects of their data analysis or recommendations for deeper interpretations of results. Acting as an advocate for the author, reviewers would find reasons for supporting the publication of the manuscript by providing "a critique that is positive, critical yet objective and balanced, contains no personally offensive comments, and is returned promptly. When specific criticisms are made, the reviewer should indicate precisely what the problems are and how they may be overcome" (Benos et al., 2003, p. 48). Sternberg (2003) asserts that any criticism, despite its severity, can be communicated in a tactful manner.

Unfortunately, the experience of many authors is anything but collegial. In fact, authors often perceive an antagonistic relationship between themselves and reviewers, created by the general tone and content of reviewers' comments. Fontaine (1995) suggests that the potential for a hostile, albeit undeclared, rivalry between authors and reviewers may be endemic to a system where both groups are basically vying for the same goal, recognition and publication of their work. As a result, many reviewers gradually adopt a critical, rather than collaborative, approach to the review process. This type of interaction is often the result of what Benos et al. (2003) refers to as adopting a journal advocacy role. In an advocacy role for the journal, a reviewer's job is to ensure that the best possible manuscripts appear in print. While acting as a journal advocate does not automatically result in harsh or disparaging treatment, the nature of the task (to serve in a gatekeeping role by providing a rigorous judgment of the quality, scholarship, and originality of a manuscript) and the system itself (double-blind review conducted by potential rivals) are potent catalysts for negative experiences to occur.

Unequal Treatment of Manuscript

When asked about the importance of various aspects of the review process, Coelho and LaForge's (2000) journal reviewers considered 5 critical areas, including ". . . (a) accuracy of information; (b) reasonableness of conclusions based upon appropriate data analysis; (c) contribution the manuscript offers to the field; (d) readability, grammar, and style; and (e) organization in terms of logic (cogency) and standard format" (p. 6). Unfortunately, not all reviewers consider each of these areas in equal measure. In fact, it is common for reviewers, particularly novice ones, to ". . . overemphasize certain limitations (usually methodological), while missing other key points related to the scientific method that should be weighted more heavily" (Seals & Tanaka, 2000, p. 52). Even so, we don't necessarily see this as a big problem in obtaining quality reviews. In fact, a savvy and knowledgeable editor can actually use this situation to her or his advantage by selecting reviewers known to have certain predispositions. A problem does exist, however, if reviewers recommend rejecting manuscripts because of an inability to separate major conceptual or methodological issues from routine editing concerns. To avoid this type of situation, Seals and Tanaka (2000) offer this advice,

It is important forreviewers to organize their comments in a way that distinguishes between the major concerns on which the acceptability of the manuscript depends and the necessary 'housekeeping' chores associated with the revision process (correcting typographical errors, minor wording changes, adding informational details, etc.). The latter are indeed essential (they should not be considered 'optional'), but author and editor both benefit from a clear delineation of a major and minor concerns of the reviewer. (p. 57)

Sometimes reviewers will focus on minutiae while paying relatively little attention to necessary components of a manuscript. Hyman (1995) describes a situation where he engaged a colleague in a lengthy rebuttal about the appropriateness of using one particular formula versus another. In actuality, either formula was acceptable; the argument was about preference. By focusing on the detail, the potential contribution of the manuscript as a whole was lost. By bringing a sense of balance to the review process, a more equal treatment of the entire manuscript can be accomplished.

Competing Recommendations (Feedback)

While uneven treatment of a manuscript can be a bit frustrating, it is downright infuriating when three reviewers offer three different recommendations for a single manuscript that range from acceptance to rejection. What gives? How can three supposedly intelligent people draw such diverse conclusions about the same manuscript? Actually, it shouldn't be all that "surprising that different readers expect, notice, and value different qualities in an academic submission, a fact that explains why reviewers may offer conflicting reviews and verdicts" (Arrington, 1995, p. 251). When this occurs, and it occurs with relative frequency, the ultimate burden of deciding falls squarely on the shoulders of the journal editor who must determine which review(s) are more convincing (Berkenkotter, 1995). When decisions are split, reviews need to provide substantive feedback both in terms of problems and potential solutions.

Emotion, Casting Aspersions, Politics, and Conflicts of Interest

The literature is replete with stories of reviews containing ad hominem attacks and emotional language, accusations of political agendas, and conflicts of interest affecting the outcome of manuscript reviews. For example, over 70% of Fontaine's (1995) respondents reported an experience where reviewers' personal or professional agendas blatantly interfered with receiving an unbiased manuscript review. In our view, and that of Young (2003), unjustified reviewer biases have no place in the peer review process.

Sternberg (2003) stated, "Many professionals write hostile reviews that at times have more of the characteristic of a personal attack than of a constructive critique of the work they are reviewing" (p. 159). It is troubling to hear of personal attacks at authors through written reviews. This concern is further heightened when considering the anonymous nature of the blind review process. True, it may be a bit difficult to keep emotions in check when reacting to the work of a colleague who shares equal passion for your topic, but it is our strong opinion that this form of review is never justified. In fact, Hyman (1995) indicates that not only should personal attacks be avoided, but great pains should be taken ". . . to avoid even the possibility that comments might be wrongly construed as a personal attack" (p. 181). He further advises us to (a) assume that authors are rational and capable people, (b) use leniency in judging others' work, and (c) look for apparent weaknesses in how arguments are expressed rather than in the argument itself. "Ideally, a good [review] should allow the facts and arguments to speak for themselves. If you have a strong case, then you should not have to buttress it with valuative judgment" (p. 181). Additionally, authors are more likely to consider reviewers' comments if criticism is constructive and offered in a neutral or supportive tone.

Conflicts of interest can occur when reviewers have a vested stake in the outcome of the review process either as an author or as a potential rival. Houlihan, Hofschulte, Sachau, and Patten (1992) examined the potential conflict of interest when journal editors, associate editors, and editorial board members submit manuscripts to their own journals. They reported that approximately two-thirds of their sample had published in their own journals while serving in some editorial capacity, and over 80% felt that it was okay to do so provided that manuscripts went through blind review and, in the case of editors and associate editors, that they were handled by a guest editor. The good news is that it appears most editors and editorial boards are responsible in their handling of manuscripts submitted by scholars serving the journal.

Finally, we would be remiss if we did not say a few words about politics. Berkenkotter (1995) notes that it could be fairly easy for unscrupulous reviewers to use their position to act as gatekeepers, advancing the work of allies while preventing their competitors from getting published. In addition, instances have occurred where political pressure to shape educational publication has been wielded from forces outside the academy. Murray and Raths (1996) cited several concerns about the editorial process including the willingness of reviewers to do the work in a timely fashion, uninformed reviews, ideological or political bias, ethical conflicts of interest, and mistaken views of fairness held by the field. While naive to think we can eliminate all influence of politics, unethical practice, or conflicts of interest, it is possible to maintain an open and transparent review process.

MANUSCRIPT REVIEW

Manuscript review is an intellectual process with both objective and subjective elements (Coelho & La Forge, 2000). Examination of the criticisms of the process provides some direction when considering how to review a manuscript. This section suggests both general reviewer responsibilities, as well as specific criteria they can use to review qualitative or quantitative research.

General Reviewer Responsibilitie

Peer reviewers are generally charged with providing feedback to a journal editor about the merits of a particular scholarly work in terms of quality and contribution to the field. Ultimately, this feedback is used to decide about whether or not to publish the manuscript. So, what type of feedback is useful to editors? What constitutes the general responsibilities of peer reviewers?

While the specific charge to reviewers may vary, some general responsibilities for peer reviewers have been identified. These include to (a) evaluate a manuscript's strengths and weaknesses, as well as the subject matter and overall quality of the work, (b) report suspected duplicate publication fraud, plagiarism, or ethical concerns, (c) assist the editor in making a publication decision, (d) provide suggestions for improving the manuscript prior to publication, and (e) maintain the standards of the journal and discipline (Patterson & Bailar, 1985; Taylor et al., 2001). In performing their duties, reviewers are ethically bound to (a) review only those manuscripts for which they have expertise, (b) maintain confidentiality, and (c) write reviews in a timely, collegial, and constructive manner (Benos et al., 2003; Uchiyamam et al., 1999; Young, 2003).

ISSUES IN MANUSCRIPT REVIEW

General Issues

Murray and Raths (1996) provide seven general criteria used to assess scholarly manuscripts including "(a) quality of the literature reviewed, (b) significance of the topic, (c) potential impact of the [research] on research and practice, (d) contribution to the field, (e) appropriate length, (f) clarity of expression, and (g) balance and fairness" (p. 417). Others have also voiced their opinions about the essential criteria needed for a quality manuscript review mostly indicating a need to examine facts, interpretations, conclusions, scholarly and professional significance, and readability (Arrington, 2003). The essential question is, of course, the quality of the manuscript and its particular strengths and weaknesses.

Section-Specific Issues in Manuscript Review

Uchiyamam et al. (1999) acknowledge that editors and reviewers are interested in many aspects of a manuscript, although not always the same ones. While varied, common elements of manuscript reviews include the introduction (including rationale or context and purpose), method, results, and discussion (implications and significance of manuscript contents); what Seals and Tanaka (2000) refer to as the IMRAD format. Most importantly, reviewers must determine the overall quality and unique contribution of the work to the field at large. These general themes apply to both quantitative and qualitative research manuscripts (Taylor et al., 2001). The Appendix provides a list of general guidelines to consider when reviewing manuscripts.

Introduction. The introduction provides a general context for the paper and a rationale for engaging in the work. This section provides the foundation for understanding the remainder of the article and establishes the importance or significance of the topic. The particular niche for this work is identified. Variables, concepts, ideas, and terms are introduced and defined. Specific connections between variables, people, or concepts are also described. At the conclusion of this initial section, the purpose of the study or paper, and specific questions or objectives are stated clearly.

Eichorn and VandenBos (1985) observed that ". . . the literature research in a journal article cannot be exhaustive; rather it should selectively but systematically inform the reader about key theoretical and research issues and about previous and current research" (p. 1315). So, while relevant literature is cited in the introduction it should not constitute an exhaustive historical review nor be placed in a separate Review of Literature section. Rather, only literature providing an appropriate history and recognizing prior work leading to the present effort are included (American Psychological Association, 2001; Moore Burnett, & Moore, 1986; Seals & Tanaka, 2000; Uchiyamam et al., 1999). The introduction should lead readers to a logical end, i.e., the purpose of the paper, and clearly tie the work to a larger body of literature and an established theoretical base.

Method. In research papers, the method section provides a detailed account of activities and technical issues addressed in the completion of the study. The method section provides evidence that the research is valid, reliable, and objective (Moore et al., 1986). Major subsections usually include some combination of the following: research design, population/sample/participants, means of data collection, researcher bias, experimental design and protocols (when applicable), procedure, and data analysis. For each of the sections, authors should assess and discuss relative strengths and weaknesses.

In the case of quantitative research, a detailed description of the sample is necessary to understand the population being represented and the extent of generalizing results (Seals & Tanaka, 2000). We recommend that evidence be provided to demonstrate the representativeness of the sample to the larger population, regardless of whether it is real or abstract. This can be fairly easy to accomplish if a thorough description of participants is given and compared to known information about the population. The need for this extra step seems especially important when nonrandom samples are used. In contrast, qualitative researchers are generally expected to provide detailed descriptions of participants, the specific sampling strategies employed, and the context from which they were identified.

The description of quantitative instruments (and experimental treatments, when applicable) are crucial to readers' understanding and should include (a) physical and conceptual descriptions of instruments and the constructs they are designed to measure, and (b) past evidence of validity and reliability, as well as a discussion about how validity was (re-)established for the present sample. When questionnaires are specifically designed for the research, the method used to develop and pilot test instrument items should also be addressed. In qualitative studies, researchers must describe the data collection protocols selected, depending on the nature of the strategy(ies) they used (e.g., semi-structured interviews, document retrieval, and analysis). Typically, the method section also contains a brief account of major procedures used to complete the study.

Results. The results section presents a summary of the research findings (Rudner & Schafer, 1999). Generally, this section is the longest one of a research manuscript, and contains both narrative and statistical information. It is important to note that while text and statistics complement each other, both should be able to stand alone. "The purpose of presenting both types of information is to make it easier for the audience to understand" (p. 12). In addition, findings presented in the results section should directly relate to the general problem and specific research objectives or questions under investigation (Moore et al., 1986).

Data results must be complete and presented in a clear and well-organized manner. In quantitative studies, all descriptive and, if appropriate, inferential statistics must be accurate. Tables and figures should be appropriately labeled with correct units and limited to only those that help to further the readers understanding of the data (Rudner & Schafer, 1999; Seals & Tanaka, 2000). Moore et al. (1986) stressed the importance of carefully interpreting results of a study, as incorrect or misinterpreted results could jeopardize the entire research effort. Complications encountered during research such as missing data, outliers, or miscoded data are also addressed in the results section. Finally, authors of quantitative research studies should address emerging concerns about the practical importance of their findings through analyses such as effect size and confidence intervals.

Discussion and conclusions. In the final section of a research manuscript, results are discussed and interpreted. Conclusions are then drawn to establish the best possible answer to the research question(s) given the findings. Two errors commonly committed when writing conclusions are confusing results and conclusions, and overgeneralizing results (Moore et al., 1986; Rudner and Schafer, 1999). It is also crucial that the conclusions of a study are consistent with the results and firmly supported by the empirical evidence presented (Moore et al.; Seals & Tanaka, 2000).

Many researchers begin their discussion by highlighting key results and reminding readers of the specific problem under investigation. Findings should then be related to previous research (Rudner & Schafer, 1999). Final conclusions and interpretations of data should be reasonable. In addition, the unique contribution of the study to our understanding of the selected topic and the implications of findings to both research and practice should be clearly delineated (Moore et al., 1986; Seals & Tanaka, 2000).

SUMMARY

Effectively critiquing manuscripts is an essential element of the peer review process. A published manuscript has survived the scrutiny of reviewers who possess some expertise in that relevant area, thus conferring "added value" to the manuscript. (Benos et al., 2003; Hyman, 1995; Seals & Tanaka, 2000). Despite the criticisms and flaws associated with the peer review process, it remains our best option for judging the merits of scientific research. Thus, reviewers must strive to make the process as valid as possible, for "the review process can never be better than the reviewers who participate" (Meier, 1997, p. 563).

While reviewing is a privilege and responsibility, it is important to remember that it is a fallible process (Benos et al., 2003; Rudner & Schafer, 1999). A journal editor thoughtfully considers the feedback received from reviewers and makes a decision about the acceptability of the manuscript for journal publication. In the end, quality reviews will be rewarded by the success of the journal (Young, 2003). Arrington (1995) concludes, "For good or ill, reviewers must make recommendations to the editor about the quality of submissions that may see print, and what they recommend, and how they will have a direct bearing on the profession, the particular journal, and the reviewers themselves" (p. 250).


REFERENCES

American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.).Washington, DC: Author.

Arrington, P. (1995). Some thoughts on changing the review process for academic journals: A personal exploration. Rhetoric Review, 13, 249-253.

Benos, D. J., Kirk, K. L., & Hall, J. E. (2003). How to review a paper. Advances in Physiology Education, 27(3), 47-52.

Berkenkotter, C. (1995). The power and the perils of peer review. Rhetoric Review, 13, 245-248.

Bornstein, R. F. (1990). Manuscript review in psychology: An alternative model. American Psychologist, 45, 672-673.

Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and U.S. science policy. Albany: State University of New York Press.

Coelho, R. J., & LaForge, J. (2000). Manuscript characteristics affecting reviewers' decisions for rehabilitation counseling-related journals. Journal of Rehabilitation, 66(2), 4-8.

Eichorn, D. H., & VandenBos, G. R. (1985). Dissemination of scientific and professional knowledge. American Psychologist, 40, 1309-1316.

Fontaine, S. I. (1995). With writers' eyes: Perception and change in manuscript review procedures. Rhetoric Review, 13(2), 259-264.

Hawkey, M. (2001). Joys, frustrations and concerns of a journal peer reviewer. Journal of Nursing Management, 9, 65-66.

Houlihan, D., Hofschulte, L., Sachau, D., & Patten, C. (1992). Critiquing the peer review process: Examining a potential dual role conflict. American Psychologist, 47, 1679- 1681.

Hyman, R. (1995). How to critique a published article. Psychological Bulletin, 118, 178- 182.

Meier, K. J. (1997). Reforming the review process: Right problem, wrong solution. PS: Political Science and Politics, 30, 561-563.

Moore, G. E., Burnett, M. F., & Moore, B. A. (1986). Approved practices in reporting quantitative research. Journal of Vocational Education Research, 11(4), 1-24.

Murray, F. B., & Raths, J. (1996). Factors in the peer review of reviews. Review of Educational Research, 66, 417-421.

Patterson, K., & Bailar, J. C. (1985). A peer review of journal peer review. In K. S. Warren (Ed.), Selectivity in information systems: Survival of the fittest (pp. 64-82). New York: Praeger.

Ponsi, M. (2003). Reviewing editorial peer review: A further step towards developing common editorial standards. International Journal of Psychoanalysis, 84, 443-445.

Rowney, J. A., & Zenisek, T. J. (1980). Manuscript characteristics influencing reviewers' decisions. Canadian Psychology, 21, 17-21.

Rudner, L. M., & Schafer, W. D. (1999). How to write a scholarly research report. Washington, D.C.: ERIC Clearinghouse on Assessment and Evaluation. (ERIC Document Reproduction Service No. ED435712) Retrieved February 20, 2004, from http://www.ericfacility.net/database

Seals, D. R., & Tanaka, H. (2000). Manuscript peer review: A helpful checklist for students and novice referees. Advances in Physiology Education, 23(1), 52-58.

Sternberg, R. J. (2003). There is no place for hostile reviews. International Journal of Clinical & Health Psychology, 3(1), 159-161.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective [Electronic version]. Studies in the Education of Adults, 33(2).

Uchiyamam, K., Simnone, G., & Borko, H. (1999, Fall). Publishing educational research: Guidelines and tips. Retrieved March 1, 2004 from American Educational Research Association web site: http://www.aera.net/epubs/index.htm

Young, S. N. (2003). Peer review of manuscripts: Theory and practice. Journal of Psychiatry & Neuroscience, 28, 327-330.


APPENDIX

General Guidelines to Consider When Critiquing Manuscripts
Introduction
  • Is what is known or unknown about a topic succinctly stated?
  • Is significance of the topic (research) established?
  • Does the context or introduction lead to a logical end, i.e., the purpose, and clearly tie the investigation to a larger body of literature and an established conceptual or theoretical base?
  • Is purpose of paper (research) clearly described with questions or objectives listed to guide reader?
  • Are previous studies on the same topic cited, including their strengths and delimitations?
Method
  • Are specific subsections identified with headings?
  • Are methods described in sufficient detail for replication?
  • Is the research design described along with accompanying advantages and delimitations?
  • Is population and sample (for quantitative orientation) or group of participants and context (for qualitative orientation) described adequately (including approach used and resultant implications) and appropriate for question(s) posed?
  • Are both the population and sample described so that some
  • reasonable determination can be made regarding the
  • representativeness of the sample to the intended population?
  • Are instruments or other data collection procedures appropriate, valid, and reliable?
  • For experimental studies, is the treatment described in sufficient detail and appropriate, valid, and reliable?
  • Are data analyzed appropriately?
  • Are the statistical techniques used appropriate for the research design?
  • Is the potential of researcher bias addressed?
Results
  • Are findings presented in a well-organized, concise, and straightforward manner?
  • If qualitative, are themes identified and described in detail? Are contradictory findings discussed?
  • If quantitative, are both descriptive and inferential statistics (if appropriate) interpreted correctly and described in detail in the findings section?
  • What is the practical importance of findings as evidenced by effect size and confidence intervals?
  • Are the data results on all measurements included in the methods section described?
  • Do the data appear reasonable?
  • Are tables and figures effectively used to present findings?
  • Have tables, figures, and narrative been used appropriately and effectively?
Discussion
  • Does discussion of results go beyond merely restating findings?
  • Are conclusions and recommendations for practice based on research findings?
  • Do the major findings of the study contribute to the field?
  • Are experimental limitations of the study addressed?
  • Is the significance of the results presented?
  • Are implications of the findings and recommendations for future research discussed?
  • Does the author(s) support statements with appropriate references?

THE AUTHORS

Jay W. Rojewski, Professor and Co-Interim Department Head, Department of Workforce Education, Learning, and Social Foundations, 210 River's Crossing, University of Georgia, Athens, GA 30602-4809. Phone: 706-542-4461. E-mail: rojewski@uga.edu.

Desirae M. Domenico, Doctoral candidate, Department of Workforce Education, Learning, and Social Foundations, 212 River's Crossing, University of Georgia, Athens, GA 30602-4809. Phone: 706-546-0830. E-mail: ddomenico@uga.edu.



DLA Ejournal Home | JCTE Home | Table of Contents for this issue | Search JCTE and other ejournals