Fluent Essays

When you think of gaining knowledge on a topic in order to make an informed decision in social work, why might it be more appropriate to turn to a research article published in a reputable journal than something on the Internet? The answer relates to who evaluates the accuracy of the information and what level of accountability there is in ensuring that it is up-to-date, reliable, and valid. As a social work graduate student, scholar, and emerging researcher, you are asked to base your writings on peer-reviewed research articles. In this Discussion, you explore what peer review means and how a research article differs from other types of writing. To Prepare: Review the Learning Resources on peer-reviewed research and on APA Style. Skim the example peer-reviewed research article, making note of the content and language used, and how it compares to the editorial, blog, and website provided on a similar topic. By Day 3 Post a description of peer review and its importance. Specifically, what makes an article peer reviewed? What is the importance of peer review in the research process and in social work practice? Finally, how does the content and language of a peer-reviewed research article compare to other forms of writing? In this issue: Focusing on Publication Ethics Peer review in scientific publications: benefits, critiques, & a survival guide Jacalyn Kelly1, Tara Sadeghieh1, Khosrow Adeli1,2,3 1 Clinical Biochemistry, Department of Pediatric Laboratory Medicine, The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada 2 Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada 3 Chair, Communications and Publications Division (CPD) , International Federation for Sick Clinical Chemistry (IFCC), Milan, Italy A R T I C L E I N F O A B S T R A C T Corresponding author: Khosrow Adeli Clinical Biochemistry The Hospital for Sick Children University of Toronto Toronto, Ontario Canada, M5G 1X8 E-mail: [email protected] Disclosure The authors declare no conflicts of interest regarding publication of this article. Key words: peer review, manuscript, publication, journal, open access Peer review has been defined as a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It functions to encourage authors to meet the accepted high standards of their discipline and to control the dissemination of research data to ensure that unwarranted claims, unacceptable inter- pretations or personal views are not published without pri- or expert review. Despite its wide-spread use by most jour- nals, the peer review process has also been widely criticised due to the slowness of the process to publish new findings and due to perceived bias by the editors and/or reviewers. Within the scientific community, peer review has become an essential component of the academic writing process. It helps ensure that papers published in scientific journals answer meaningful research questions and draw accurate conclusions based on professionally executed experimen- tation. Submission of low quality manuscripts has become increasingly prevalent, and peer review acts as a filter to prevent this work from reaching the scientific community. The major advantage of a peer review process is that peer- reviewed articles provide a trusted form of scientific com- munication. Since scientific knowledge is cumulative and builds on itself, this trust is particularly important. Despite the positive impacts of peer review, critics argue that the peer review process stifles innovation in experimentation, Peer review in scientific publications: benefits, critiques, & a survival guide Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Page 227 eJIFCC2014Vol25No3pp227-243 mailto:khosrow.adeli%40sickkids.ca?subject= http://en.wikipedia.org/wiki/Scholarly_method http://en.wikipedia.org/wiki/Idea http://en.wikipedia.org/wiki/Expert http://en.wikipedia.org/wiki/Author Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide and acts as a poor screen against plagiarism. Despite its downfalls, there has not yet been a foolproof system developed to take the place of peer review, however, researchers have been looking into electronic means of improving the peer review process. Unfortunately, the recent explosion in online only/electronic journals has led to mass publication of a large number of sci- entific articles with little or no peer review. This poses significant risk to advances in scientific knowledge and its future potential. The current article summarizes the peer review process, highlights the pros and cons associated with dif- ferent types of peer review, and describes new methods for improving peer review. WHAT IS PEER REVIEW AND WHAT IS ITS PURPOSE? Peer Review is defined as “a process of sub- jecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field” (1). Peer review is intended to serve two primary purposes. Firstly, it acts as a filter to ensure that only high quality research is published, especially in reputable journals, by determining the validity, significance and originality of the study. Secondly, peer review is intended to improve the quality of manu- scripts that are deemed suitable for publication. Peer reviewers provide suggestions to authors on how to improve the quality of their manu- scripts, and also identify any errors that need correcting before publication. HISTORY OF PEER REVIEW The concept of peer review was developed long before the scholarly journal. In fact, the peer re- view process is thought to have been used as a method of evaluating written work since an- cient Greece (2). The peer review process was first described by a physician named Ishaq bin Ali al-Rahwi of Syria, who lived from 854-931 CE, in his book Ethics of the Physician (2). There, he stated that physicians must take notes de- scribing the state of their patients’ medical con- ditions upon each visit. Following treatment, the notes were scrutinized by a local medical council to determine whether the physician had met the required standards of medical care. If the medical council deemed that the appropri- ate standards were not met, the physician in question could receive a lawsuit from the mal- treated patient (2). The invention of the printing press in 1453 al- lowed written documents to be distributed to the general public (3). At this time, it became more important to regulate the quality of the written material that became publicly available, and editing by peers increased in prevalence. In 1620, Francis Bacon wrote the work Novum Organum, where he described what eventually became known as the first universal method for generating and assessing new science (3). His work was instrumental in shaping the Scientific Method (3). In 1665, the French Journal des sça- vans and the English Philosophical Transactions of the Royal Society were the first scientific jour- nals to systematically publish research results (4). Philosophical Transactions of the Royal Soci- ety is thought to be the first journal to formalize the peer review process in 1665 (5), however, it is important to note that peer review was ini- tially introduced to help editors decide which manuscripts to publish in their journals, and at that time it did not serve to ensure the valid- ity of the research (6). It did not take long for the peer review process to evolve, and shortly thereafter papers were distributed to reviewers with the intent of authenticating the integrity of the research study before publication. The Roy- al Society of Edinburgh adhered to the following peer review process, published in their Medical Essays and Observations in 1731: “Memoirs sent by correspondence are distributed accord- ing to the subject matter to those members who Page 228 eJIFCC2014Vol25No3pp227-243 http://en.wikipedia.org/wiki/Journal_des_s%C3%A7avans http://en.wikipedia.org/wiki/Journal_des_s%C3%A7avans http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide are most versed in these matters. The report of their identity is not known to the author.” (7). The Royal Society of London adopted this review procedure in 1752 and developed the “Committee on Papers” to review manuscripts before they were published in Philosophical Transactions (6). Peer review in the systematized and institution- alized form has developed immensely since the Second World War, at least partly due to the large increase in scientific research during this period (7). It is now used not only to ensure that a scientific manuscript is experimentally and ethically sound, but also to determine which papers sufficiently meet the journal’s standards of quality and originality before publication. Peer review is now standard practice by most credible scientific journals, and is an essential part of determining the credibility and quality of work submitted. IMPACT OF THE PEER REVIEW PROCESS Peer review has become the foundation of the scholarly publication system because it effective- ly subjects an author’s work to the scrutiny of other experts in the field. Thus, it encourages au- thors to strive to produce high quality research that will advance the field. Peer review also sup- ports and maintains integrity and authenticity in the advancement of science. A scientific hypoth- esis or statement is generally not accepted by the academic community unless it has been pub- lished in a peer-reviewed journal (8). The Insti- tute for Scientific Information (ISI) only considers journals that are peer-reviewed as candidates to receive Impact Factors. Peer review is a well- established process which has been a formal part of scientific communication for over 300 years. OVERVIEW OF THE PEER REVIEW PROCESS The peer review process begins when a scien- tist completes a research study and writes a manuscript that describes the purpose, experi- mental design, results, and conclusions of the study. The scientist then submits this paper to a suitable journal that specializes in a relevant research field, a step referred to as pre-submis- sion. The editors of the journal will review the paper to ensure that the subject matter is in line with that of the journal, and that it fits with the editorial platform. Very few papers pass this ini- tial evaluation. If the journal editors feel the pa- per sufficiently meets these requirements and is written by a credible source, they will send the paper to accomplished researchers in the field for a formal peer review. Peer reviewers are also known as referees (this process is sum- marized in Figure 1). The role of the editor is to select the most appropriate manuscripts for the journal, and to implement and monitor the peer review process. Editors must ensure that peer reviews are conducted fairly, and in an effective and timely manner. They must also ensure that there are no conflicts of interest involved in the peer review process. When a reviewer is provided with a paper, he or she reads it carefully and scrutinizes it to evalu- ate the validity of the science, the quality of the experimental design, and the appropriateness of the methods used. The reviewer also assess- es the significance of the research, and judges whether the work will contribute to advance- ment in the field by evaluating the importance of the findings, and determining the originality of the research. Additionally, reviewers identi- fy any scientific errors and references that are missing or incorrect. Peer reviewers give rec- ommendations to the editor regarding whether the paper should be accepted, rejected, or im- proved before publication in the journal. The editor will mediate author-referee discussion in order to clarify the priority of certain referee requests, suggest areas that can be strength- ened, and overrule reviewer recommenda- tions that are beyond the study’s scope (9). If Page 229 eJIFCC2014Vol25No3pp227-243 Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide Figure 1 Overview of the review process the paper is accepted, as per suggestion by the peer reviewer, the paper goes into the produc- tion stage, where it is tweaked and formatted by the editors, and finally published in the sci- entific journal. An overview of the review pro- cess is presented in Figure 1. WHO CONDUCTS REVIEWS? Peer reviews are conducted by scientific experts with specialized knowledge on the content of the manuscript, as well as by scientists with a more general knowledge base. Peer review- ers can be anyone who has competence and Page 230 eJIFCC2014Vol25No3pp227-243 Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide expertise in the subject areas that the journal covers. Reviewers can range from young and up-and-coming researchers to old masters in the field. Often, the young reviewers are the most responsive and deliver the best quality reviews, though this is not always the case. On average, a reviewer will conduct approximately eight reviews per year, according to a study on peer review by the Publishing Research Consor- tium (PRC) (7). Journals will often have a pool of reviewers with diverse backgrounds to allow for many different perspectives. They will also keep a rather large reviewer bank, so that review- ers do not get burnt out, overwhelmed or time constrained from reviewing multiple articles simultaneously. WHY DO REVIEWERS REVIEW? Referees are typically not paid to conduct peer reviews and the process takes considerable ef- fort, so the question is raised as to what incen- tive referees have to review at all. Some feel an academic duty to perform reviews, and are of the mentality that if their peers are expected to review their papers, then they should review the work of their peers as well. Reviewers may also have personal contacts with editors, and may want to assist as much as possible. Oth- ers review to keep up-to-date with the latest developments in their field, and reading new scientific papers is an effective way to do so. Some scientists use peer review as an opportu- nity to advance their own research as it stimu- lates new ideas and allows them to read about new experimental techniques. Other reviewers are keen on building associations with presti- gious journals and editors and becoming part of their community, as sometimes reviewers who show dedication to the journal are later hired as editors. Some scientists see peer review as a chance to become aware of the latest research before their peers, and thus be first to develop new insights from the material. Finally, in terms of career development, peer reviewing can be desirable as it is often noted on one’s resume or CV. Many institutions consider a researcher’s in- volvement in peer review when assessing their performance for promotions (11). Peer review- ing can also be an effective way for a scientist to show their superiors that they are committed to their scientific field (5). ARE REVIEWERS KEEN TO REVIEW? A 2009 international survey of 4000 peer re- viewers conducted by the charity Sense About Science at the British Science Festival at the University of Surrey, found that 90% of review- ers were keen to peer review (12). One third of respondents to the survey said they were happy to review up to five papers per year, and an ad- ditional one third of respondents were happy to review up to ten. HOW LONG DOES IT TAKE TO REVIEW ONE PAPER? On average, it takes approximately six hours to review one paper (12), however, this num- ber may vary greatly depending on the con- tent of the paper and the nature of the peer reviewer. One in every 100 participants in the “Sense About Science” survey claims to have taken more than 100 hours to review their last paper (12). HOW TO DETERMINE IF A JOURNAL IS PEER REVIEWED Ulrichsweb is a directory that provides informa- tion on over 300,000 periodicals, including in- formation regarding which journals are peer re- viewed (13). After logging into the system using an institutional login (eg. from the University of Toronto), search terms, journal titles or ISSN numbers can be entered into the search bar. The database provides the title, publisher, and country of origin of the journal, and indicates Page 231 eJIFCC2014Vol25No3pp227-243 Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide whether the journal is still actively publishing. The black book symbol (labelled ‘refereed’) re- veals that the journal is peer reviewed. THE EVALUATION CRITERIA FOR PEER REVIEW OF SCIENTIFIC PAPERS As previously mentioned, when a reviewer re- ceives a scientific manuscript, he/she will first determine if the subject matter is well suited for the content of the journal. The reviewer will then consider whether the research question is important and original, a process which may be aided by a literature scan of review articles. Scientific papers submitted for peer review usu- ally follow a specific structure that begins with the title, followed by the abstract, introduction, methodology, results, discussion, conclusions, and references. The title must be descriptive and include the concept and organism inves- tigated, and potentially the variable manipu- lated and the systems used in the study. The peer reviewer evaluates if the title is descriptive enough, and ensures that it is clear and concise. A study by the National Association of Realtors (NAR) published by the Oxford University Press in 2006 indicated that the title of a manuscript plays a significant role in determining reader in- terest, as 72% of respondents said they could usually judge whether an article will be of inter- est to them based on the title and the author, while 13% of respondents claimed to always be able to do so (14). The abstract is a summary of the paper, which briefly mentions the background or purpose, methods, key results, and major conclusions of the study. The peer reviewer assesses whether the abstract is sufficiently informative and if the content of the abstract is consistent with the rest of the paper. The NAR study indicated that 40% of respondents could determine whether an article would be of interest to them based on the abstract alone 60-80% of the time, while 32% could judge an article based on the ab- stract 80-100% of the time (14). This demon- strates that the abstract alone is often used to assess the value of an article. The introduction of a scientific paper presents the research question in the context of what is already known about the topic, in order to identify why the question being studied is of interest to the scientific community, and what gap in knowledge the study aims to fill (15). The introduction identifies the study’s purpose and scope, briefly describes the general methods of investigation, and outlines the hypothesis and predictions (15). The peer reviewer determines whether the introduction provides sufficient background information on the research topic, and ensures that the research question and hy- pothesis are clearly identifiable. The methods section describes the experimen- tal procedures, and explains why each experi- ment was conducted. The methods section also includes the equipment and reagents used in the investigation. The methods section should be detailed enough that it can be used it to re- peat the experiment (15). Methods are written in the past tense and in the active voice. The peer reviewer assesses whether the appropri- ate methods were used to answer the research question, and if they were written with suffi- cient detail. If information is missing from the methods section, it is the peer reviewer’s job to identify what details need to be added. The results section is where the outcomes of the experiment and trends in the data are ex- plained without judgement, bias or interpre- tation (15). This section can include statistical tests performed on the data, as well as figures and tables in addition to the text. The peer re- viewer ensures that the results are described with sufficient detail, and determines their credibility. Reviewers also confirm that the text is consistent with the information presented in Page 232 eJIFCC2014Vol25No3pp227-243 Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide tables and figures, and that all figures and ta- bles included are important and relevant (15). The peer reviewer will also make sure that table and figure captions are appropriate both con- textually and in length, and that tables and fig- ures present the data accurately. The discussion section is where the data is an- alyzed. Here, the results are interpreted and related to past studies (15). The discussion describes the meaning and significance of the results in terms of the research question and hypothesis, and states whether the hypothesis was supported or rejected. This section may also provide possible explanations for unusual results and suggestions for future research (15). The discussion should end with a conclusions section that summarizes the major findings of the investigation. The peer reviewer determines whether the discussion is clear and focused, and whether the conclusions are an appropri- ate interpretation of the results. Reviewers also ensure that the discussion addresses the limi- tations of the study, any anomalies in the re- sults, the relationship of the study to previous research, and the theoretical implications and practical applications of the study. The references are found at the end of the pa- per, and list all of the information sources cited in the text to describe the background, meth- ods, and/or interpret results. Depending on the citation method used, the references are listed in alphabetical order according to author last name, or numbered according to the order in which they appear in the paper. The peer re- viewer ensures that references are used appro- priately, cited accurately, formatted correctly, and that none are missing. Finally, the peer reviewer determines whether the paper is clearly written and if the content seems logical. After thoroughly reading through the entire manuscript, they determine whether it meets the journal’s standards for publication, and whether it falls within the top 25% of papers in its field (16) to determine priority for publica- tion. An overview of what a peer reviewer looks for when evaluating a manuscript, in order of importance, is presented in Figure 2. To increase the chance of success in the peer review process, the author must ensure that the paper fully complies with the journal guide- lines before submission. The author must also be open to criticism and suggested revisions, and learn from mistakes made in previous submissions. ADVANTAGES AND DISADVANTAGES OF THE DIFFERENT TYPES OF PEER REVIEW The peer review process is generally conducted in one of three ways: open review, single-blind review, or double-blind review. In an open re- view, both the author of the paper and the peer reviewer know one another’s identity. Alter- natively, in single-blind review, the reviewer’s identity is kept private, but the author’s iden- tity is revealed to the reviewer. In double-blind review, the identities of both the reviewer and author are kept anonymous. Open peer review is advantageous in that it prevents the reviewer from leaving malicious comments, being care- less, or procrastinating completion of the re- view (2). It encourages reviewers to be open and honest without being disrespectful. Open reviewing also discourages plagiarism amongst authors (2). On the other hand, open peer re- view can also prevent reviewers from being honest for fear of developing bad rapport with the author. The reviewer may withhold or tone down their criticisms in order to be polite (2). This is especially true when younger review- ers are given a more esteemed author’s work, in which case the reviewer may be hesitant to provide criticism for fear that it will damper their relationship with a superior (2). Accord- ing to the Sense About Science survey, editors Page 233 eJIFCC2014Vol25No3pp227-243 Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide find that completely open reviewing decreases the number of people willing to participate, and leads to reviews of little value (12). In the afore- mentioned study by the PRC, only 23% of au- thors surveyed had experience with open peer review (7). Single-blind peer review is by far the most com- mon. In the PRC study, 85% of authors surveyed had experience with single-blind peer review (7). This method is advantageous as the reviewer is more likely to provide honest feedback when their identity is concealed (2). This allows the reviewer to make independent decisions with- out the influence of the author (2). The main disadvantage of reviewer anonymity, howev- er, is that reviewers who receive manuscripts on subjects similar to their own research may be tempted to delay completing the review in order to publish their own data first (2). Double-blind peer review is advantageous as it prevents the reviewer from being biased against the author based on their country of origin or previous work (2). This allows the pa- per to be judged based on the quality of the content, rather than the reputation of the au- thor. The Sense About Science survey indicates that 76% of researchers think double-blind peer review is a good idea (12), and the PRC survey indicates that 45% of authors have had experience with double-blind peer review (7). The disadvantage of double-blind peer review is that, especially in niche areas of research, it can sometimes be easy for the reviewer to determine the identity of the author based on Figure 2 How a peer review evaluates a manuscript Page 234 eJIFCC2014Vol25No3pp227-243 Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli Peer review in scientific publications: benefits, critiques, & a survival guide writing style, subject matter or self-citation, and thus, impart bias (2). Masking the author’s identity from peer review- ers, as is the case in double-blind review, is gen- erally thought to minimize bias and maintain review quality. A study by Justice et al. in 1998 investigated whether masking author identity affected the quality of the review (17). One hun- dred and eighteen manuscripts were random- ized; 26 were peer reviewed as normal, and 92 were moved into the ‘intervention’ arm, where editor quality assessments were completed for 77 manuscripts and author quality assessments were completed for 40 manuscripts (17). There was no perceived difference in quality between the masked and unmasked reviews. Addition- ally, the masking itself was often unsuccessful, especially with well-known authors (17). How- ever, a previous study conducted by McNutt et al. had different results (18). In this case, blind- ing was successful 73% of the time, and they found that when author identity was masked, the quality of review was slightly higher (18). Although Justice et al. argued that this differ- ence was too small to be consequential, their study targeted only biomedical journals, and the results cannot be generalized to journals of a different subject matter (17). Additionally, there were problems masking the identities of well-known authors, introducing a flaw in the methods. Regardless, Justice et al. concluded that masking author identity from reviewers may not improve review quality (17). In addition to open, single-blind and double- blind peer review, there are two experimental forms of peer review. In some cases, following publication, papers may be subjected to post- publication peer review. As many papers are now published online, the scientific commu- nity has the opportunity to comment on these papers, engage in online discussions and post a formal review. For example, online publish- ers PLOS and BioMed Central have enabled scientists to post comments on published pa- pers if they are registered users of the site (10). Philica is another journal launched with this ex- perimental form of peer review. Only 8% of au- thors surveyed in the PRC study had experience with post-publication review (7). Another ex- perimental form of peer review called Dynamic Peer Review has also emerged. Dynamic peer review is conducted on websites such as Naboj, which allow scientists to conduct peer reviews on articles in the preprint media (19). The peer review is conducted on repositories and is a continuous process, which allows the public to see both the article and the reviews as the article is being developed (19). Dynamic peer review helps prevent plagiarism as the scien- tific community will already be familiar with the work before the peer reviewed version appears in print (19). Dynamic review also reduces the time lag between manuscript submission and publishing. An example of a preprint server is the ‘arXiv’ developed by Paul Ginsparg in 1991, which is used primarily by physicists (19). These alternative forms of peer review are still un- established and experimental. Traditional peer review is time-tested and still highly utilized. All methods of peer review have their advantages and deficiencies, and all are prone to error. PEER REVIEW OF OPEN ACCESS JOURNALS Open access (OA) journals are becoming in- creasingly popular as they allow the potential for widespread distribution of publications in a timely manner (20). Nevertheless, there can be issues regarding the peer review process of open access journals. In a study published in Science in 2013, John Bohannon submitted 304 slightly different versions of a fictional sci- entific paper (written by a fake author, working out of a non-existent institution) to a selected group of OA journals. This study was performed in order to determine whether papers sub- mitted to OA journals are properly reviewed Page …