Critical Review of Scientific Manuscripts Symposium

Critical Review of Scientific Manuscripts Symposium | ASCRS

Sunday, June 2, 8:00 – 9:30 am
1.5 CME Credit Hours
Registration is Required (No Charge)

Limit: 100 participants

The peer review process is central to the continued advancement of surgical knowledge. Continuous critical review of new manuscripts ensures that the best available evidence is disseminated within the surgical community. The volume of new material, the complexity of trial design and the increasingly nuanced conclusions require detailed and systematic critical review. While the practicing surgeon relies on the editorial process to a great extent to separate the “wheat from the chaff”, he/she also requires solid critical appraisal skills to ensure that evidence from published studies is relevant and appropriate for individual patient care. While the editor asks, “Does this manuscript add significant knowledge to the literature?”, the surgeon asks, “Does this manuscript add significant knowledge to change my practice?”

There are three generic types of surgical trials: exploratory trials to assess utility, explanatory trials to assess efficacy and pragmatic trials to assess effectiveness. Methodologies include observational studies (cohort or case control), administrative database studies, randomized controlled trials (RCT), structured reviews and meta-analyses. Each methodology has its purpose and place in the investigation of surgical care and its own strengths and weaknesses.

Traditionally, observational studies are viewed as the lowest form of evidence. Yet there are many instances where an observational study is the best and perhaps the only form of evidence that is practical and available especially if a disease entity or outcome is rare. Observational studies may provide relatively strong evidence when there is a large treatment effect, or when confounding factors would bias the results in opposition to the observed effect. They may be subject to significant bias thus the methodology and results must be carefully and critically examined.

Large non-randomized observational studies based on administrative databases have become very popular due to electronic data collection. They have the advantage of reporting on large populations and identifying trends in treatment, outcomes and rare complications. However, data collection may be incomplete or inconsistent and lack the granularity to draw conclusions as to how or why.

The RCT design is least likely to be affected by bias and is the only methodology that can identify cause and effect. Sound knowledge of study-design is needed to evaluate the many variations in structure and primary outcomes (i.e. inferiority, non-inferiority). While no study is completely void of bias it is important to determine whether bias is responsible for a significant portion of the observed effect as there is wide variation in the quality of RCTs.

The sheer volume of primary literature has increased the importance of secondary analysis or literature summaries. A systematic review of the literature may be combined with a meta-analysis to give a best estimate of effect. Although pooling the results of multiple trials increases precision by narrowing confidence intervals, a secondary analysis of poorly designed RCTs may result in a misleading conclusion. Thus, the reviewer must be familiar with the common limitations of secondary analysis and conclusions that can be drawn.

This symposium is aimed at two groups: present and prospective reviewers for Diseases of the Colon & Rectum and the practicing surgeon who wants to increase his/her critical appraisal skills. It is designed to be hands on. Through an interactive symposium we will explore the most common study methodologies, identify appropriate questions for each, identify the advantages and disadvantages and the common mistakes in study conduct, reporting and conclusions. We will also explore essential resources for additional learning in this area.

Previously published representative papers from the four common methodologies will be identified in advance from Diseases of the Colon & Rectum. At the symposium, each participant will be assigned to a small group lead by an editorial board member from DC&R. Following an introduction of the manuscript by the faculty, the editorial board members will facilitate a working discussion and critique of each manuscript within the small groups. Board members will have access to the original editorial comments and the changes that were requested by the editors prior to publication to enhance the discussion. At the end of the discussion period, the faculty will summarize for all participants the most significant concerns from the editorial review, the changes that were made to the manuscript prior to publication and any unresolved issues that were recognized but accepted as they were not felt to have a significantly effect on outcomes.

Gap Analysis

What Is: Evidence is presented in many forms using many methodologies. Familiarity with these methodologies is necessary to evaluate the continued stream of manuscripts with respect to study design, conduct, results and conclusions. The knowledge and ability to analyze these methodologies may not be common to all surgeons in our group.

What Should Be: As colorectal surgeons we should be familiar with the literature not only with respect to content, but with measures of quality. The ability to recognize a quality paper is an essential skill for the journal reviewer and the practicing surgeon alike.


At the conclusion of this session, participants should be able to:

  1. Recognize when observational studies can provide relatively strong evidence and their limitations.
  2. Identify the advantages, limitations and pitfalls of administrative database studies
  3. Recognize potential for bias and methodological issues within randomized controlled trials
  4. Recall the components of a valuable comprehensive systematic review and meta-analysis
  5. Apply resources to enhance your critical appraisal skills.


W. Donald Buie, MD, Calgary, AB, Canada
Susan Galandiuk, MD, Louisville, KY