Abstract
Background: Although several tools to evaluate the credibility of health care guidelines exist, guidance on practical steps for developing guidelines is lacking. We systematically compiled a comprehensive checklist of items linked to relevant resources and tools that guideline developers could consider, without the expectation that every guideline would address each item.
Methods: We searched data sources, including manuals of international guideline developers, literature on guidelines for guidelines (with a focus on methodology reports from international and national agencies, and professional societies) and recent articles providing systematic guidance. We reviewed these sources in duplicate, extracted items for the checklist using a sensitive approach and developed overarching topics relevant to guidelines. In an iterative process, we reviewed items for duplication and omissions and involved experts in guideline development for revisions and suggestions for items to be added.
Results: We developed a checklist with 18 topics and 146 items and a webpage to facilitate its use by guideline developers. The topics and included items cover all stages of the guideline enterprise, from the planning and formulation of guidelines, to their implementation and evaluation. The final checklist includes links to training materials as well as resources with suggested methodology for applying the items.
Interpretation: The checklist will serve as a resource for guideline developers. Consideration of items on the checklist will support the development, implementation and evaluation of guidelines. We will use crowdsourcing to revise the checklist and keep it up to date.
Health care guidelines and their appropriate implementation are of interest to national organizations, professional societies, health care providers, policy-makers, patients and the public. Several tools to evaluate the credibility of existing health care guidelines are available,1–3 but guideline developers worldwide are struggling with the lack of guidance on practical steps, resources and tools to facilitate the development of trustworthy guidelines.
Several overviews describe the complete process for guideline development and follow-up efforts, such as implementation and evaluation.3–19 In 2006, a series of reviews of methods used in guideline development led to the preparation of advice for the World Health Organization (WHO) to improve the use of research evidence in the development of recommendations, guidelines and policies.6 Another project, resulting in a series of 15 manuscripts authored by representatives from 36 international organizations, outlined the steps and processes for guideline development for professional societies.11 The findings of these projects, together with the work leading to the related standards of the Institute of Medicine and the Guidelines International Network, have been included in advice given to guideline developers around the world, including ministries of health,20 through many workshops and counselling.21
However, while providing technical assistance to implement the national guideline development program with the WHO office in Estonia and the ministry of health in Saudi Arabia, and while initiating the DECIDE project (Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence),22 we recognized the need for facilitation of the steps to produce and implement guidelines. Our experience highlighted that, although manuals for single organizations are available, a comprehensive list of items and a portal for resources to consider in the planning, development, implementation, evaluation and updating of guidelines does not exist.
Our objective was to systematically compile a comprehensive checklist of items and a portal for related resources that guideline developers, both newcomers and experts, could consider for all stages of the guideline enterprise, from the planning and formulation of recommendations, to their implementation, evaluation and updating. We also aimed to identify gaps in currently available processes and tools and provide a mechanism for filling these gaps.
Methods
Selection of data sources
We sought to achieve a representative sample of source documents for review that would allow us to reach saturation for the checklist’s topics and items. We established the following eligibility criteria for data sources to be included, without restrictions on the date of publication: (a) documents described as “guidelines for guidelines,” guideline methodology reports or guideline manuals that were available from guideline developers, particularly international and national agencies, and professional societies; (b) documents that described the development process of health care guidelines, including clinical, public health and health systems guidelines; and (c) documents that, altogether, encompassed a global geographic perspective, including North America, South America, Europe, Asia and Australasia, and described the development process of guidelines produced for high-, middle- or low-income countries.
We used an iterative process that began with 2 of us (H.J.S. and E.A.A.) identifying an initial list of key guideline methodology reports and guidelines for guidelines to review.1–3,6–9,11,23 A comprehensive list of guideline developers was then compiled through group discussion and email correspondence. The discussion and identification of guideline developers was informed by our collective awareness of guideline developers through work in the field for over a decade, membership on key committees and attendance of meetings in the field (e.g., the Guidelines International Network) and a recent systematic search for guideline manuals (the search strategy is available from the authors upon request). Based on our systematic search, we also generated a secondary list of potentially eligible source documents that would be reviewed to determine whether saturation of items was reached following data extraction from the primary sources. The secondary list specifically included manuals and methodology reports from professional societies.11
In May 2013, we electronically searched the websites of guideline developers for methodology reports, guideline manuals and relevant supplementary manuals (e.g., guideline writing manuals, manuals for updating guidelines). When guideline manuals were not found online, we contacted the organizations directly to obtain them. As a final step, expert colleagues, researchers and other stakeholders in guideline development in our group (G.G., J.T., M.R., P.A.-C., R.H., S.L.N., S.T., A.O.) were involved in identifying important omissions and suggesting additional sources for inclusion. We did not conduct our own systematic review of electronic databases on guideline methodology because the guidelines for guidelines and methodology reports included in our list of sources were already informed by systematic reviews.
Data extraction and analysis
One of us (W.W.) reviewed the initial list of sources for key guideline methodology reports and guidelines for guidelines. He extracted items and generated overarching topics to create a preliminary checklist. The checklist was reviewed for completeness (by H.J.S.). We then developed a form based on the preliminary checklist to extract data from each source document.
Data abstractors (W.W., I.E., M.F., M.V., R.B.-P., K.-T.L., S.K., T.B., Y.Z., U.R., I.N.) worked in pairs to review the documents. They checked off and referenced items that were already included in the preliminary checklist, provided applicable revisions to the items and extracted new items. Given that our aim was comprehensiveness, any item discussed was eligible for inclusion in the checklist.
The data abstractors were provided with an instruction sheet on how to review the source documents and a copy of the standardized abstraction form. The instructions and form were reviewed in a group discussion with the abstractors and then pilot tested in one round based on 4 examples before abstraction of all source documents. The instructions included notes to look for figures, flow diagrams, bullet points and descriptive text about the steps to follow to develop guidelines. The data abstractors were instructed to be inclusive when considering any new items to add to the checklist.
The data abstractors reviewed the source documents, checking off existing items and suggesting new items, with a page reference. In addition, they identified references and links to available training tools and materials for the items, as well as resources with suggested methodology for applying the items in a guideline development process. The abstractors used the form to make comments about the wording of items, provide additional details for items and suggest placement of items under the specific topics. When one of a pair of abstractors suggested an item or checked off an existing item, it was included in the draft list of items. A third data abstractor confirmed that the item was discussed in the source document.
Following data extraction, we were all invited to review the draft checklist, suggest revisions and approve the final draft. We consulted the guideline development experts in our group to provide additional suggestions and feedback about completeness of the checklist using a pilot-tested feedback form that contained the draft checklist. As a final step, 2 of us (M.V. and W.W.) reviewed 8 documents24–31 to determine whether saturation of checklist topics and items was reached. No new topics or items were identified, which indicated that we had reached saturation.
Results
Figure 1 summarizes the identification and selection of source documents. The initial list of 43 documents included key methodology reports,1–3,6,7,11,23 sources identified through discussions and email correspondence,20,24,25,27,28,30–55 and sources recommended by experts.5,26,29,56,57 Eight documents were excluded because the full-text version or manual could not be obtained.48–55 Of the remaining 35 documents, data were extracted from 27,1–3,5–7,11,20,23,32–47,56,57 and 8 were reviewed for saturation of topics and items.24–31 Potential checklist topics and items were obtained from 9 guideline methodology documents or document series and 18 guideline developer manuals (Box 1).
Search and selection process.
Sources for data extraction
Guideline development manuals
North America
American Academy of Otolaryngology — Head and Neck Surgery Clinical Practice Guideline Development Manual, third edition, 201356
American College of Cardiology — American Heart Association Task Force on Practice Guidelines Methodology Manual and Policies, 2010,32 and supplementary documents58–60
Canadian Task Force on Preventive Health Care Procedure Manual, 201133
Cancer Care Ontario Program in Evidence-Based Care Handbook, 2012,34 and supplementary documents61,62
US Centers for Disease Control and Prevention (CDC) Guidelines and Recommendations: a CDC Primer, 201235
Transparency Matters: Kaiser Permanente’s National Guideline Program Methodological Processes, 201236
US Preventive Services Task Force Procedure Manual, 200837
Europe
Estonian Handbook for Guidelines Development, 201120
European Society of Cardiology Recommendations for Guidelines Production, 201041
National Institute for Health and Care Excellence Guidelines Manual, 2012,42 and supplementary documents63–65
SIGN (Scottish Intercollegiate Guidelines Network) 50: a Guideline Developer’s Handbook, 201143
Spain Ministry of Health Development of Clinical Practice Guidelines in the National Health System: Methodological Manual, 200744
World Health Organization Handbook for Guideline Development, 201245
South America
Argentina National Academy of Medicine Guide to Adaptation of Clinical Practice Guidelines, 200838
Colombia Ministry of Health and Social Security Methodological Guide for Developing Integrated Care Guidelines in the Colombian System of Health and Social Security, 201039
Peru Ministry of Health Technical Standards for the Development of Clinical Practice Guidelines, 200640
Australasia
National Health and Medical Research Council (NHMRC) Procedures and Requirements for Meeting the 2011 NHMRC Standard for Clinical Practice Guidelines, 2011,46 and supplementary document66
New Zealand Guidelines Group Handbook for the Preparation of Explicit Evidence-Based Clinical Practice Guidelines, 200147
Guideline methodology reports
The ADAPTE process: resource toolkit for guideline adaptation, 200957
AGREE II: advancing guideline development, reporting and evaluation in health care, 20101,67,68
Methodology for the development of antithrombotic therapy and prevention of thrombosis guidelines: antithrombotic therapy and prevention of thrombosis, 9th edition, 20125
Conference on Guideline Standardization: Standardized Reporting of Clinical Practice Guidelines, 200323,69
Guidelines International Network: Toward International Standards for Clinical Practice Guidelines, 20122
Health Research Policy and Systems Series: Improving the Use of Research Evidence in Guideline Development, 20066,70–85
Implementation Science Series: Developing Clinical Practice Guidelines, 20127–9
Institute of Medicine: Clinical Practice Guidelines We Can Trust, 20113
Proceedings of the American Thoracic Society Series: a Guide to Guidelines for Professional Societies and Other Developers of Recommendations, 201211,86–99
Note: AGREE = Appraisal of Guidelines, Research and Evaluation.
The list of items based on the initial abstraction included 19 topics and 123 items. Duplicate review of the 18 manuals revealed an additional 15 items, for a total of 138. After several iterations of the checklist were reviewed by the research team and the guideline development experts, we merged 2 topics, for a total of 18, and identified 8 additional items, for a total of 146. Box 2 lists the 18 topics with a description of how they fit into the guideline development process. The topics and items cover all stages of the guideline enterprise, from the planning and formulation of guidelines, to their dissemination, evaluation and updating.
| Topic | Description |
|---|---|
| 1. Organization, budget, planning and training | Involves laying out a general but detailed plan describing what is feasible, how it will be achieved and what resources are required to produce and use the guideline. The plan should refer to a specific period and be expressed in formal, measurable terms. |
| 2. Priority setting | Refers to the identification, balancing and ranking of priorities by stakeholders. Priority setting ensures that resources and attention are devoted to those general areas (e.g., chronic obstructive pulmonary disease, diabetes, cardiovascular disease, cancer, prevention) where health care recommendations will provide the greatest benefit to the population, a jurisdiction or a country. A priority-setting approach needs to contribute to future plans while responding to existing, potentially difficult circumstances.100,101 |
| 3. Guideline group membership | Defines who is involved, in what capacity, and how the members are selected for the guideline development and at other steps of the guideline enterprise. |
| 4. Establishing guideline group processes | Defines the steps to be followed, how those involved will interact and how decisions will be made. |
| 5. Identifying target audience and topic selection | Involves describing the potential users or consumers of the guideline and defining the topics to be covered in the guideline (e.g., diagnosis of chronic obstructive pulmonary disease). |
| 6. Consumer and stakeholder involvement | Describes how relevant people or groups who are not necessarily members of the panel but are affected by the guideline (e.g., as target audience or users) will be engaged. |
| 7. Conflict of interest considerations | Focuses on defining and managing the potential divergence between an individual’s interests and his or her professional obligations that could lead to questioning whether the actions or decisions are motivated by gain, such as financial, academic advancement, clinical revenue streams or community standing. Financial or intellectual or other relationships that may affect an individual’s or organization’s ability to approach a scientific question with an open mind are included. |
| 8. Question generation | Focuses on defining key questions the recommendations should address using the PICO (patient/problem, intervention, comparison, outcome) framework, including the detailed population, intervention (including diagnostic tests and strategies) and outcomes that will be relevant for decision-making (e.g., should test A be used, or should treatments B, C, D or E be used in chronic obstructive pulmonary disease?). |
| 9. Considering importance of outcomes and interventions, values, preferences and utilities | Includes integrating, in the process of developing the guidelines, how those affected by its recommendations assess the possible consequences. These include patient, caregiver and health care provider knowledge, attitudes, expectations, moral and ethical values, and beliefs; patient goals for life and health; prior experience with the intervention and the condition; symptom experience (e.g., breathlessness, pain, dyspnea, weight loss); preferences for and importance of desirable and undesirable outcomes; perceived impact of the condition or interventions on quality of life, well-being or satisfaction, and interactions between the work of implementing the intervention, the intervention itself, and other contexts the patient may be experiencing; preferences for alternative courses of action; and preferences relating to communication content and styles, information and involvement in decision-making and care. This can be related to what in the economic literature is considered utilities. An intervention itself can be considered a consequence of a recommendation (e.g., the burden of taking a medication or undergoing surgery) and a level of importance or value is associated with that. |
| 10. Deciding what evidence to include and searching for evidence | Focuses on laying out inclusion and exclusion criteria based on types of evidence (e.g., rigorous research, informally collected), study designs, characteristics of the population, interventions and comparators, and deciding how the evidence will be identified and obtained. It also includes but is not limited to evidence about values and preferences, local data and resources. |
| 11. Summarizing evidence and considering additional information | Focuses on presenting evidence in a synthetic format (e.g., tables or brief narratives) to facilitate the development and understanding of recommendations. It also involves identifying and considering additional information relevant to the question under consideration. |
| 12. Judging quality, strength or certainty of a body of evidence | Includes assessing the confidence one can place in the obtained evidence by transparently evaluating the obtained research (individual studies and across studies) and other evidence applying structured approaches. This may include, but is not limited to, evidence about baseline risk or burden of disease, importance of outcomes and interventions, values, preferences and utilities, resource use (cost), estimates of effects and accuracy of diagnostic tests. |
| 13. Developing recommendations and determining their strength | Developing recommendations involves use of a structured analytic framework and a transparent and systematic process to integrate the factors that influence a recommendation. Determining the strength of the recommendations refers to judgments about how confident a guideline panel is that the implementation of a recommendation exerts more desirable than undesirable consequences. |
| 14. Wording of recommendations and of considerations about implementation, feasibility and equity | Refers to choosing syntax and formulations that facilitate understanding and implementation of the recommendations. Such wording is connected to considerations about implementation, feasibility and equity, which refer to the guideline panel’s considerations about how the recommendation will be used and what impact it may have on the factors described. |
| 15. Reporting and peer review | Reporting refers to how a guideline will be made public (e.g., print, online). Peer review refers to how the guideline document will be reviewed before its publication and how it can be assessed (e.g., for errors), both internally and externally, by stakeholders who were not members of the guideline development group. |
| 16. Dissemination and implementation | Focuses on strategies to make relevant groups aware of the guidelines and to enhance their uptake (e.g., publications and tools such as mobile applications). |
| 17. Evaluation and use | Refers to formal and informal strategies that allow judgments about: evaluation of the guidelines as a process and product; evaluation of the use or uptake, or both; and evaluation of impact and whether or not the guideline leads to improvement in patient or population health or other consequences. |
| 18. Updating | Refers to how and when a guideline requires revision because of changes in the evidence or other factors that influence the recommendations. |
Where identified, the checklist includes links to learning tools and training materials, as well as resources with suggested methodology for applying the items; it also identifies where gaps exist. Examples of learning tools and training materials include tutorials, readings and references (e.g., how certain organizations accomplish the step) that will assist guideline developers in aspects such as outlining the factors that may be considered in priority setting and topic selection, adhering to a conflict-of-interest policy and establishing an appropriate code of conduct for the guideline development group. Examples of resources for applying the items include methods, tools and references to assist developers in carrying out the steps, such as providing a disclosure-of-interest form, rating the strength of recommendations and adapting toolkits for guideline dissemination.
Box 3 describes the final checklist (available at the end of the article). Although organization and priority setting precede most other steps, we agreed that a strict ordering of topics and items would be difficult to achieve. Guideline developers will often need to return to items mentioned in preceding topics or look ahead to subsequent items to enhance their understanding of a specific step in the guideline process. Figure 2 describes the interrelation of the various steps and topics and provides a graphical presentation of the guideline development process. In addition, we have compiled a glossary of key terms to help with the interpretation of the checklist (see Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.131237/-/DC1). Finally, we have created an interactive version of the checklist for use by guideline developers (available at http://cebgrade.mcmaster.ca/guidecheck.html) that includes the suggested learning tools and resources for implementation that were identified.
Diagram of the guideline development process. The steps and involvement of various members of the guideline development group are interrelated and not necessarily sequential. The guideline panel and supporting groups (e.g., methodologist, health economist, systematic review team and secretariat for administrative support) work collaboratively, informed through consumer and stakeholder involvement. They typically report to an oversight committee or board overseeing the process. For example, while deciding how to involve stakeholders early for priority setting and topic selection, the guideline group must also consider how developing formal relationships with the stakeholders will enable effective dissemination and implementation to support uptake of the guideline. Furthermore, considerations for organization, planning and training encompass the entire guideline development project, and steps such as documenting the methodology used and the decisions made, as well as considering conflicts of interest, occur throughout the process.
| The checklist is organized into 18 topics, each with corresponding items to consider. Users of the checklist should review all topics and items before applying them, because the items are not necessarily sequential and many are interconnected. The brief examples included with some items are for clarification and elaboration; they are not meant to be extensive instructions for how to accomplish the steps. Instructions and suggestions for accomplishing the steps can be found in the source documents referenced and in the resources suggested in the interactive online version of the checklist (http://cebgrade.mcmaster.ca/guidecheck.html). See Appendix 1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.131237/-/DC1) for a glossary of terms appearing in the checklist. | |||
|---|---|---|---|
| Completed | Not applicable | Guideline development steps | Sources, reference nos. |
| 1. Organization, budget, planning and training | |||
| □ | □ | 1. Establish the structure of the guideline development group and determine the roles, tasks and relationships among the various groups to be involved (e.g., oversight committee or body to direct guideline topic selection and group membership, a working group consisting of experts and methodologists to synthesize evidence, a secretariat to provide administrative support, a guideline panel to develop recommendations, and stakeholders and consumers for consultation). See also topics 3, 4 and 6 | 20, 32–36, 39, 42–45, 56, 57, 72, 88 |
| □ | □ | 2. Perform a thorough assessment of the proposed guideline development project with respect to financial and feasibility issues concerning the guideline development group (e.g., availability of resources to complete the project, expected commitment from guideline panel and staff). | 20, 32, 35, 37, 40, 42–46, 47, 57, 58, 71, 88 |
| □ | □ | 3. Obtain organizational approval to proceed with the guideline project. | 2, 20, 33–35, 37, 42, 43, 45, 46, 58 |
| □ | □ | 4. Prepare a budget for the development of the guideline, outlining the estimated costs for each step (e.g., remuneration of working group members and staff, cost of outsourcing certain tasks to outside organizations or groups, travel expenses, and publication and dissemination expenses). | 3, 37, 41, 45, 71 |
| □ | □ | 5. Determine whether guideline panel members will be provided any payment or reimbursement for their time or will work as volunteers. | 3, 34, 47, 57 |
| □ | □ | 6. Obtain or secure funding for the development of the guideline, with attention to conflict-of-interest considerations. See also topic 7 | 1, 20, 43, 45, 46, 56, 57, 89 |
| □ | □ | 7. Outline and arrange the administrative support that will be required to facilitate the guideline development process (e.g., a secretariat of the working group to organize and obtain declaration of interests and to arrange group meetings). | 20, 32, 37, 40, 42, 43, 45, 56, 57, 88 |
| □ | □ | 8. Plan and prepare for training and support that will be required for those involved in the guideline development process (e.g., conflict-of-interest–related education or training for guideline panel members, and teaching sessions for patients to be involved in the guideline group). See also topics 4 and 6 | 2, 3, 39, 42–45, 60, 64, 70, 72, 88 |
| □ | □ | 9. Set a timeline for the completion of the guideline and target dates for the completion of milestones in the guideline development process. | 20, 33, 34, 37, 40, 42, 43, 45, 46, 56, 58, 71, 88 |
| □ | □ | 10. Determine what, if any, legal considerations are relevant for the planned guideline (e.g., reimbursement policies for orphan drugs). | 2, 3, 20, 37, 42, 43, 45, 46, 81, 94 |
| □ | □ | 11. To keep the guideline development group on track, prepare a protocol for the entire guideline that can be completed as the project progresses, including an outline of the overall goals and objectives for the guideline, the timeline, task assignments, steps that will require documentation of decisions and the proposed methods for all steps (i.e., those covered in this checklist, such as the methods for forming the guideline group, selection of topics to be covered in the guideline, consensus methods, consultation methods, and methods for the evidence search and selection). | 1, 2, 23, 32–35, 37, 39, 42, 44, 45, 56, 57, 70, 74, 90 |
| 2. Priority setting | |||
| □ | □ | 1. Decide on a process for priority setting of guideline topics and who will be responsible for directing the process (e.g., priorities set by oversight committee at headquarters of sponsoring organization, or priorities referred by government ministries of health or by professional societies). | 7, 20, 32–37, 39, 41–43, 45, 47, 56, 71, 87 |
| □ | □ | 2. Apply a systematic and transparent process with specific criteria for the proposal of a guideline topic during priority setting (e.g., high prevalence and burden of disease, avoidable mortality and morbidity, high cost, emerging diseases or emerging care options, variation in clinical practice, and rapidly changing evidence). | 7, 20, 32–37, 39, 41–43, 47, 56, 57, 71, 87 |
| □ | □ | 3. Involve appropriate stakeholders in the priority setting and selection of guideline topics (e.g., clinicians, professional societies, policy-makers, payers and the public). See also topic 6 | 32–34, 36, 37, 39, 42–45, 71, 72, 87 |
| □ | □ | 4. Consider and decide how different perspectives on the importance and resources required for implementing the guideline recommendations will be considered (e.g., perspectives of patients, payers, clinicians and public health programs). See topic 11 | 1, 23, 32, 34, 36, 37, 42, 46, 47, 57, 71, 94, 99 |
| □ | □ | 5. Search for any existing up-to-date guidelines covering the proposed topic and assess their credibility (e.g., AGREE II). Determine whether the existing guideline(s) can be adapted or whether a completely new guideline should be developed. See also topic 10 | 20, 32–39, 42, 43, 45, 47, 56, 57, 71, 99 |
| □ | □ | 6. Discuss the need or opportunity to partner with other organizations that develop guidelines to determine whether a collaborative effort will be sought for the development of the guideline, or any part of the guideline. | 3, 11, 32, 33, 37, 41, 45 |
| □ | □ | 7. Perform a scoping exercise for the proposed guideline topic with respect to implementation issues and barriers to change (e.g., if developed, the guideline is likely to improve health outcomes, implementation of health care recommendations is feasible, resources are available). | 1, 3, 9, 20, 23, 32, 33, 35, 37, 39, 42–47, 56, 57, 71, 94 |
| □ | □ | 8. Select or provide a consensus method to be used to agree on the priorities set and the guideline topic selected (e.g., voting, Delphi consensus). See also topic 4 | 20, 33, 37, 42, 43, 71, 87 |
| □ | □ | 9. To ensure transparency, document the processes used to set priorities and to select guideline topics. | 20, 33, 35, 37, 42, 43, 71, 87 |
| 3. Guideline group membership | |||
| □ | □ | 1. Seek multidisciplinary representation for the guideline development group, including members from the target audience, patients, caregivers, frontline clinicians, content experts, methodology experts and experts in health economics, to fulfill the roles required (e.g., for the working group, guideline panel). See also topic 6 | 1–3, 5, 7, 20, 33–35, 37–47, 56, 57, 72, 86 |
| □ | □ | 2. Decide on methods for recruitment and enrolment of members for the guideline development group (e.g., widespread advertising of posts, competitive appointment by interview). | 3, 5, 33–35, 37, 41–43, 56, 72, 88 |
| □ | □ | 3. Achieve a topic-appropriate balance of expertise and adequate representation for the guideline panel (e.g., experts and primary care physicians who form the target audience, and sex and geographic distribution of panel members), which may be iterative if additional members are required as the target audience and topics within the guideline are refined. See also topic 5 | 1, 3, 5, 7, 20, 32–34, 37–39, 41–45, 57, 72, 88 |
| □ | □ | 4. Consider the optimum size for the guideline development group, particularly the guideline panel (e.g., if too small, the group may lack sufficient experience, content expertise and wide representation; if too large, it may lack cohesiveness and effective group interaction). | 2, 3, 5, 7, 20, 33, 37, 39, 42–45, 56, 72, 88 |
| □ | □ | 5. Outline roles for the group members and the tasks they will be responsible for (e.g., forming a writing team, taking meeting minutes and documenting decisions, providing methodology consultation, conducting systematic reviews and obtaining other evidence, providing perspective of patients, and providing perspective of specialist clinicians). | 3, 5, 7, 20, 32–34, 37– 40, 42–45, 56, 72, 88 |
| □ | □ | 6. Select group leader(s) or chair(s) experienced in group facilitation, maintaining constructive dynamics, and identifying and resolving conflicts who will remain neutral and objective and who have methodologic expertise and content expertise. | 2, 3, 5, 7, 20, 32, 33, 37–39, 41–45, 56, 72, 88 |
| □ | □ | 7. To ensure transparency, document the process used to select guideline group members and their roles. | 1, 2, 5, 32, 33, 37, 42, 43, 45, 46, 72, 88 |
| 4. Establishing guideline group processes | |||
| □ | □ | 1. Establish how and how often communication with guideline panel members and other groups will take place, establish who will be responsible for making the arrangements, and consider when to deviate from this approach. | 3, 5, 20, 32, 33, 37, 42–44, 56, 72, 88 |
| □ | □ | 2. Set expectations and awareness of the group process through an introduction, training and support for the guideline development group members (e.g., setting ideal conditions for group discussion and decision-making). | 2, 3, 7, 20, 32, 39, 43, 45, 47, 56, 63, 72, 88 |
| □ | □ | 3. As part of the training for the guideline development group, ensure that group members understand what the process and proposed methods will be and that they need to be adhered to (e.g., consensus methods that may be used, anonymous or non-anonymous voting, assessment of evidence, group discussion and contribution of ideas). | 3, 20, 32, 39, 43–45, 47, 56, 63, 72, 74, 88 |
| □ | □ | 4. Aim to set optimal conditions for group members to be provided equal opportunities to contribute ideas and for their ideas and arguments to be given appropriate consideration (e.g., during group discussion, decision-making and formulation of recommendations). | 2, 3, 7, 32, 33, 43, 45, 47, 56, 63, 72, 74, 88 |
| □ | □ | 5. Establish methods for dealing with conflict or disputes among group members and dysfunction in the group process. | 32, 33, 44, 45, 47, 56, 63, 74, 88 |
| □ | □ | 6. Provide opportunities for discussion and feedback about the group process throughout the guideline development project. | E, 43 |
| □ | □ | 7. Establish a method for structured and timely distribution and archiving of documents used and produced in the guideline development. | 32, 39, 42, 44, 56, 88 |
| □ | □ | 8. Set a quorum for meetings (e.g., 75% of group must be present to formulate recommendations), but expect that all group members attend all meetings as much as possible. | 20, 32, 33, 37, 42 |
| □ | □ | 9. Set or plan times and locations for meetings (virtual or in person) in advance and prepare a scope and specific agenda for each meeting. | 5, 20, 33, 37, 39, 42, 44, 45, 47, 56, 72, 88 |
| □ | □ | 10. Keep a record of all meetings with minutes (e.g., who attended, what was the agenda, what decisions were made, what the next steps will be) and determine whether not to make the minutes publicly or internally available. | 5, 20, 32, 42, 44, 88 |
| 5. Identifying target audience and topic selection | |||
| □ | □ | 1. Identify, define and/or review the primary audience (e.g., primary care physicians, health program managers) and secondary audience(s) (e.g., hospital administrators) for the guideline and determine how many audiences can be addressed with the guideline. | 1, 2, 7, 20, 23, 32, 35, 37–39, 42, 44, 45, 47, 66, 86 |
| □ | □ | 2. Consult appropriate stakeholders about the identified target audiences to ensure they are applicable for the guideline topic and that no relevant audience is missed. See also topic 6 | 20, 39, 44, 86 |
| □ | □ | 3. Establish a method and criteria to generate and prioritize a candidate list of topics to be addressed within the guideline (e.g., where evidence is most confusing or controversial, where there is currently uncertainty or inconsistency in practice, where questions exist about screening, diagnosis and treatment). | 3, 8, 20, 32–34, 36–39, 42–45, 47, 56, 57, 66, 87 |
| □ | □ | 4. Consult appropriate stakeholders to ensure all relevant topics for the guideline have been identified and will meet the needs of the target audience(s). See also topic 6 | 3, 20, 33, 36, 37, 39, 42–44, 87, 93 |
| □ | □ | 5. Select or provide a consensus development method to be used by the group in agreeing on the final topics to be addressed within the guideline (e.g., Delphi method, nominal group technique). | 37, 42, 56, 87 |
| □ | □ | 6. To ensure transparency, document the processes used to identify the target audience(s) and to select the topics for the guideline. | 1, 23, 33, 37, 39, 44, 66, 86, 87 |
| 6. Consumer and stakeholder involvement | |||
| □ | □ | 1. Identify the appropriate stakeholders to involve and consult with in the development of the guideline to incorporate views of all those who might be affected by the guideline (e.g., professional groups, health managers, policy-makers, industry representatives). | 1–3, 33, 35, 37–39, 43, 44, 46, 47, 56, 57, 60, 64, 72, 94 |
| □ | □ | 2. Identify the appropriate consumers to involve and consult with in the development of the guideline (e.g., individual patients, people who provide nonreimbursed care and support to patients, members of the public as potential patients and as funders of health care through taxation, community organizations that represent the interests of patients, and advocates representing the interests of patients and people who care for patients). | 1–3, 7, 35, 38, 39, 43–46, 56, 60, 64, 72, 93 |
| □ | □ | 3. Establish methods for consumer and stakeholder involvement and maintain a registry of stakeholders for the guideline (e.g., enroll consumer and stakeholder members to participate directly on the guideline panel, announce call for separate consumer and stakeholder meeting[s] or workshop[s], distribute documents and feedback electronically, have an open period for review of documents and feedback). | 2, 3, 5, 7, 33, 35, 37–39, 42–44, 46, 47, 56, 60, 64, 72, 93, 94 |
| □ | □ | 4. Provide information (e.g., training and introduction sessions) for consumers and stakeholders involved directly on the guideline panel to clarify roles and maximize contributions (e.g., information on evaluating evidence objectively and avoiding recommendations based on self-interests). | 2, 3, 7, 43, 44, 64, 72, 93, 94 |
| □ | □ | 5. Determine the roles, tasks and timing for consultation with consumers and stakeholders not directly participating on the guideline panel (e.g., at specific milestones during the guideline development process, include opportunities to comment on priority setting, suggest topics for the guideline, identify target audience, identify patient-important outcomes, identify additional evidence, point to consequences that the panel has not considered and review the final guideline draft). | 3, 7, 35, 38, 39, 43, 44, 46, 47, 64, 72, 79, 94 |
| □ | □ | 6. Develop or adopt standard templates for consumer and stakeholder input and comments during consultation, with clear instructions or training modules to ensure effective input. | 33, 42, 43, 44, 94 |
| □ | □ | 7. Offer adequate time for consumer and stakeholder feedback and consultation. | 33, 44, 64, 94 |
| □ | □ | 8. Set a policy and process for handling consumer and stakeholder feedback and dealing with different perspectives (e.g., ensure that diverse perspectives are taken into account in making decisions, provide transparent rationale for judgments made, provide an appeal process for stakeholders, and publish consultation comments and the guideline development panel’s responses). | 33, 42, 43, 45, 64, 79, 94 |
| □ | □ | 9. To ensure explicit and transparent methods, document the enrolment and selection of consumers and stakeholders for the guideline panel and the involvement and consultation with all other consumers and stakeholders. | 1, 33, 42, 44, 46, 64, 72, 94 |
| 7. Conflict-of-interest considerations | |||
| □ | □ | 1. Set a policy for declaration of interests by individual participants at admission to the project, including potential guideline panel members before their involvement (e.g., what interests should be disclosed [e.g., financial, intellectual, academic/clinical, competitive interests of the professional society]). | 2, 3, 5, 7, 20, 33, 35, 37–39, 41–46, 56, 58, 62, 66, 73, 89 |
| □ | □ | 2. Set a policy for determining conflicts of interest and an approach for collecting and updating declarations of conflicts of interest (e.g., how and what level of financial interest should be disclosed, how long the period covered by the disclosure should be, who will judge what constitutes a conflict). | 2, 3, 5, 7, 20, 33, 35, 37, 39, 41–46, 56, 58, 62, 66, 73, 89 |
| □ | □ | 3. Provide clear instructions and training to the potential guideline group members on how to complete the conflict-of-interest disclosure, including a list of the members who must declare conflicts of interest and the types of interests to declare (include examples). | 3, 7, 20, 33, 37, 39, 42– 45, 56, 58, 62, 66, 73, 89 |
| □ | □ | 4. Set a policy for managing conflicts of interest (e.g., allow individuals with conflicts of interest to participate in the guideline development but exclude them from voting on specific recommendations related to the area of conflict; ensure that the chair has no conflicts of interest; have evidence summaries prepared by unconflicted methodologists). | 2, 3, 5, 7, 20, 33, 35, 37, 39, 41–46, 56, 58, 62, 66, 73, 89 |
| □ | □ | 5. Set a policy for managing conflicts of interest with respect to funding of the guideline development activities (e.g., advocate for public funding, no commercial sponsorship, commercial sponsorship from entities unrelated to topic of guideline, commercial support for nondirect activities such as translation, no single-source sponsor). | 1, 45, 46, 60, 73, 89 |
| □ | □ | 6. Disclose and publish the funding source and describe the role of the sponsors and support provided for the development of the guideline. | 1–3, 20, 23, 38, 45, 46, 56, 57, 60, 83, 89 |
| □ | □ | 7. Explicitly disclose, publish and describe conflicts of interest of the guideline group members, particularly where the conflicts bear on specific recommendations. | 1–3, 5, 7, 20, 23, 35, 37, 39, 42–46, 56, 57, 62, 83, 89 |
| 8. Question generation | |||
| □ | □ | 1. Establish methods for generating the questions for the guideline, prioritizing questions, and selecting and ranking outcomes. | 2, 3, 5, 8, 20, 33, 35–39, 42–45, 47, 56, 57, 60, 66, 75, 90 |
| □ | □ | 2. Generate and document the key questions (e.g., clinical, health, policy, cost-effectiveness) to be answered in the guideline using a standard format (e.g., PICO [patient/problem, intervention, comparison, outcome] framework). If it is not feasible to answer all questions (e.g., because of limited time or resources), determine the criteria by which the questions generated will be prioritized (e.g., by surveying guideline panel members and other stakeholders). | 1–3, 5, 8, 20, 33–39, 42–45, 47, 56, 57, 60, 66, 75, 90 |
| □ | □ | 3. Explicitly describe the population to whom the guideline is meant to apply. Take into consideration specific characteristics of the population, such as prevalence of multiple comorbidities in the population, geographic setting and equity issues (e.g., plausible reasons for anticipating differential relative effects across disadvantaged and advantaged populations). | 1–3, 5, 9, 20, 23, 32, 33, 41–45, 48, 51–54, 57, 73, 87 |
| □ | □ | 4. Determine whether or not regulatory approval is required for considering interventions in guidelines (e.g., for international guidelines, this may be not relevant because regulatory approval may not be available or present in all target countries). | 3, 20, 32, 39, 42, 45, 46, 81, 94 |
| □ | □ | 5. Explicitly describe the intervention(s) and comparator(s) to be considered in the guideline and develop an analytic framework depicting the relationships among interventions and outcomes. Identify whether or not multiple (treatment) comparisons should be included. | 2, 3, 5, 8, 20, 23, 32, 33, 35–37, 39, 42–47, 56, 76, 90 |
| □ | □ | 6. Identify the important outcomes (e.g., outcomes along the clinical pathway; morbidity, quality of life, mortality), including both desirable (e.g., benefits, less burden, savings) and undesirable (e.g., harm, burden, costs and decreased patient autonomy) effects. Do not ignore important outcomes for which evidence may be lacking, | 1–3, 5, 8, 20, 32, 33, 35–39, 42–47, 56, 75, 90 |
| □ | □ | 7. Determine the setting (e.g., countries, hospitals) or include it in the considerations about the population (i.e., population cared for in tertiary care hospitals). | 3, 20, 37, 42, 43, 45, 47, 66, 75, 90 |
| □ | □ | 8. Mandate a preference for patient-important outcomes over surrogate, indirect outcomes. Consider appropriateness of surrogate outcomes along the causal pathway when data for a patient-important outcome is lacking. | 3, 5, 8, 20, 37, 39, 42, 43, 47, 75, 90 |
| □ | □ | 9. Rank the relative importance of the outcomes, taking into consideration the values and preferences of the target population. | 3, 5, 20, 33, 35, 37, 39, 42, 44, 45, 75, 90 |
| □ | □ | 10. Determine or develop a process for determining a priori the magnitude of effect for the individual outcomes that is judged as important to the target population. | 5, 90 |
| □ | □ | 11. Involve all guideline group members and consult consumers and stakeholders to ensure broad representation from the target population in generating the questions and selecting and rating the important outcomes. | 3, 20, 37, 39, 42, 45, 72, 75, 94 |
| □ | □ | 12. To ensure explicit and transparent methods, document the generation and prioritization of questions, the selection and ranking of outcomes, and stakeholder and consumer consultation. | 20, 33, 42, 45, 66, 90 |
| □ | □ | 13. To help direct the evidence review, ensure that the guideline protocol outlines the target population, target condition, outcomes and key questions considered. | 1, 2, 5, 23, 33, 35, 42, 44, 66, 90 |
| 9. Considering the importance of outcomes and interventions, values, preferences and utilities | |||
| □ | □ | 1. Decide whether the relative importance of outcomes and interventions, values, preferences or utilities of consumers and stakeholders (e.g., patients and target audience) to inform decisions and deliberations during the guideline development will be elicited indirectly or directly (e.g., review of the published literature v. consultation with consumers). | 1–3, 5, 7, 20, 33, 35, 37, 43–46, 56, 72, 79, 93 |
| □ | □ | 2. Establish methods for consultation with consumers and stakeholders to obtain information about the relative importance of outcomes and interventions, values, preferences or utilities (e.g., involvement of consumers on guideline panel, surveys or focus groups with broader representation of consumers). | 3, 7, 43, 44, 79, 93 |
| □ | □ | 3. Determine whether a structured approach will be used to assess the confidence (i.e., quality of the evidence) in the obtained importance ratings, values, preferences and utilities. | E, 79 |
| □ | □ | 4. Determine whether modelling will be used to integrate the relative importance of outcomes and interventions, values, preferences or utilities and how modelling will be done. | E, 8, 79 |
| □ | □ | 5. Determine whose perspective(s) will be considered when obtaining information about the relative importance of outcomes and interventions, values, preferences or utilities and when making decisions or formulating recommendations (e.g., patients, public, society, clinicians). | 7, 20, 37, 79, 93 |
| □ | □ | 6. Consider and document approaches for dealing with conflicting ratings of relative importance for outcomes and interventions, values, preferences or utilities (e.g., patient v. caregiver or health care provider, patient v. public). | 5, 37, 44, 79, 93 |
| □ | □ | 7. To ensure explicit and transparent methods, document how information was obtained about the relative importance of outcomes and interventions, values, preferences or utilities. | 1, 5, 35, 43, 44, 72, 79, 93 |
| □ | □ | 8. Document whether ethical considerations, such as whether recommendations should give special consideration to certain patient groups or conditions (e.g., older patients, rare diseases, patients affected by health inequalities). | 93 |
| □ | □ | 9. Decide how to consider ethical or moral values in making health care recommendations (e.g., by considering religious, social, or cultural convictions). | 102 |
| 10. Deciding what evidence to include and searching for evidence | |||
| □ | □ | 1. Follow methods for systematic reviews (either full systematic reviews or rapid systematic reviews depending on the topic and organization’s framework) or provide a rationale for why this is not done. | 1–3, 5, 8, 20, 32–47, 56, 57, 77, 92 |
| □ | □ | 2. Develop a protocol for locating, selecting and synthesizing the evidence (e.g., conduct a search for existing systematic reviews, conduct a new systematic review and search the grey literature) and determine the types of evidence to include (e.g., databases searched, types of studies, inclusion and exclusion criteria, searching for specific studies on adverse effects or deciding to abstract information on adverse effects from studies on benefit). | 1–3, 5, 8, 20, 23, 32–47, 56, 57, 77, 90 |
| □ | □ | 3. Decide who will develop the search strategies and perform the search and selection of evidence (e.g., working group of guideline development group, outsource to external agency, form a relationship between guideline development group and external agency to collaborate on development of the guideline). | 3, 5, 20, 32–35, 42–45, 56, 77, 88 |
| □ | □ | 4. Using a validated tool, critically appraise existing systematic reviews selected to be included to ensure they are of adequate quality and appropriate for use in the guideline. | 3, 5, 8, 20, 34–39, 42–45, 56, 77, 92 |
| □ | □ | 5. If an existing systematic review is updated or requires updating, determine how new evidence will be included and how those who conducted the review will be contacted and possibly involved in the update. | 37, 45, 57, 92 |
| □ | □ | 6. If a new systematic review is required, conduct an assessment to determine whether adequate resources (e.g., time and funding) are available to conduct a full systematic review. | 37, 39, 45, 77 |
| □ | □ | 7. If resources are limited, consider applying a rapid assessment methodology and explicitly describe the methodology, noting important limitations, uncertainties, and the need and urgency to undertake a full systematic review. | 32, 37, 45, 77 |
| □ | □ | 8. Establish methods for identifying additional evidence and unpublished data (e.g., obtain suggestions from guideline panel members, consult with stakeholders). | 32, 33, 35, 42, 47, 77, 90 |
| □ | □ | 9. Set a policy for handling expert input (i.e., expert opinion is not evidence per se and should not be used as evidence; rather, experience or observations that support expert opinions should be described, identified and, if possible, appraised in a systematic and transparent way [e.g., in the conceptual framework]). | 3, 32, 34, 35, 76 |
| □ | □ | 10. To ensure explicit and transparent methods, document and publish the search and selection of evidence, the judging of eligibility, the range of evidence included and the search strategies used. | 1, 2, 20, 23, 32, 33, 35, 37, 39, 42, 46, 47, 57, 92 |
| 11. Summarizing evidence and considering additional information | |||
| □ | □ | 1. Summarize the evidence using a concise summary (e.g., evidence table, evidence profile or summary of findings table) of the best available evidence for each important outcome, including accuracy of diagnostic tests, anticipated benefits, harms, resources (costs), rating of the quality of evidence, and a summary of the relative and absolute results/estimate of effect for each outcome. | 1, 3, 5, 8, 20, 23, 32–39, 42–47, 77, 92 |
| □ | □ | 2. Provide a summary of the additional information needed to inform recommendations (e.g., qualitative narrative summary, evidence table), including values and preferences, factors that might modify the expected effects, need (prevalence, baseline risk or status), effects on equity, feasibility and the availability of resources. | 1–3, 5, 8, 20, 33–35, 37, 39, 42–45, 57, 65, 66, 77, 80–82, 91, 93 |
| □ | □ | 3. Establish methods for obtaining information about resource use and cost (e.g., searching for existing economic evaluations, developing economic model, performing cost-effectiveness analysis). | 1–3, 5, 8, 20, 33, 35, 37, 39, 42–47, 80, 91 |
| □ | □ | 4. Identify the costs, resource use and, if applicable, cost-effectiveness and describe the nature of the costs (patient, community, society) (e.g., affordability considerations, estimates of resource use and acquisition costs weighed directly against evidence of benefits and harms of an intervention). | 1–3, 5, 8, 20, 33, 35, 37, 39, 42–47, 80, 91 |
| □ | □ | 5. To ensure transparent methods, document how the additional information is to be incorporated with the synthesized evidence (e.g., formal consensus on patient values, consensus on equity issues, formal economic analysis, consideration of disaggregated resource use data in a qualitative manner). | 3, 5, 8, 20, 23, 33–35, 37, 39, 42, 44–47, 77, 81, 91, 93 |
| □ | □ | 6. Provide training about the use of the evidence tables and opportunities for discussion to ensure that all members of the guideline panel are familiar with the tables and use them in the appropriate manner. | 44, 70, 88 |
| □ | □ | 7. In addition to the evidence summary, make available the full systematic review(s) and the original studies and other sources of evidence for the guideline panel to inform deliberations (e.g., set up a collaborative website or make sources of evidence available at meetings and via electronic communication). | 44, 77 |
| 12. Judging quality, strength or certainty of a body of evidence | |||
| □ | □ | 1. Select a framework outlining the criteria to be considered in rating the quality of evidence (e.g., GRADE, US Preventive Services Task Force). Avoid modifying grading tools. | 2, 3, 5, 8, 20, 32–37, 39–47, 56, 78, 92 |
| □ | □ | 2. Decide who will be responsible for appraising the quality of evidence (e.g., unconflicted methodologists participating in the working group). | 20, 34, 39, 43, 44, 97 |
| □ | □ | 3. Assess the quality of evidence for each important outcome. | 5, 8, 20, 33, 35, 36, 39, 42, 44, 45, 78, 92 |
| □ | □ | 4. Assess the overall quality of evidence (e.g., lowest quality of evidence from outcomes rated as most important or critical, or highest quality of evidence when all outcomes point in the same direction). | 5, 8, 20, 32, 33, 35–37, 39, 42, 44–47, 56, 78, 92 |
| □ | □ | 5. Report the quality of evidence assessed for the outcomes and the body of evidence. | 2, 8, 20, 32, 36, 37, 39, 44–46, 83, 97 |
| □ | □ | 6. To ensure they are explicit and transparent, document the judgments made in appraising the quality of evidence. | 2, 5, 20, 32, 33, 35, 37, 39, 42, 44–47, 78, 92 |
| 13. Developing recommendations and determining their strength | |||
| □ | □ | 1. Apply a framework outlining the factors to be considered to arrive at a recommendation. | 1, 3, 5, 8, 23, 32–39, 42–47, 56, 57, 78, 96 |
| □ | □ | 2. Plan and share the logistical details of the consensus meeting(s) during which recommendations will be formulated with the participants, including distribution of documents required for the meeting (e.g., evidence summaries, evidence-to-recommendation tables), setting an agenda for the meeting(s) and selecting a consensus development method to be used by the group in agreeing on judgments (e.g., Delphi method, nominal group technique). | 5, 34, 45, 47, 56 |
| □ | □ | 3. Review the factors of the framework that influence the recommendation, including the direction and strength (e.g., the types of evidence and information relevant to the analysis, focusing on the balance between desirable and undesirable consequences informed by the quality of evidence, the magnitude of the difference between the benefits and harms, the certainty about or variability in values and preferences, resource use, equity and other factors). | 1–3, 5, 8, 20, 23, 32, 35–39, 42–45, 47, 57, 65, 78, 96 |
| □ | □ | 4. If applicable, make provisions for formulating recommendations in situations where there is insufficient evidence or very-low-quality evidence (e.g., conditional recommendation with judgments laid out transparently; no recommendation if the guideline panel feels there is substantial risk that their decision may be wrong; recommendation that the intervention be used in the context of research complemented by guidance for what are the best management options until further research becomes available). | 8, 20, 33, 35, 37, 39, 42, 44, 45, 96 |
| □ | □ | 5. Make provisions for formulating research recommendations and decide where to report them (e.g., in the guideline appendix, suggest the specific research questions, specific patient-important outcomes to measure and other relevant aspects of what research is needed to reduce the uncertainty about the benefits and/or undesirable downsides of the intervention). | 8, 39, 42–45, 85, 96 |
| □ | □ | 6. Formulate the recommendations and summarize the rationale for each recommendation (e.g., narratively or in a table), including details about the judgments made by the group and the explicit link between the recommendation and evidence supporting the recommendation. | 1, 3, 8, 20, 23, 33, 35– 38, 42–46, 83, 96, 97 |
| □ | □ | 7. Select a method for rating the strength of the formulated recommendations to inform the audience of the guideline about the degree of the guideline group’s confidence about following that recommendation. | 2, 3, 5, 8, 20, 32, 33, 36, 37, 39, 40, 43–47, 56, 78, 96 |
| □ | □ | 8. Select a consensus development method to be used by the group in rating the strength of recommendations (e.g., Delphi method, nominal group technique, voting). | 5, 20, 37, 43–45, 88 |
| □ | □ | 9. Provide suggestions about whether the recommendations are appropriate to serve as performance measures/quality criteria (e.g., management options associated with strong recommendations based on high- or moderate-quality evidence are particularly good candidates for quality criteria; when a recommendation is weak, discussing with patients the relative merits of the alternative management strategies and appropriate documentation of this interaction may become a quality criterion). | 3, 9, 20, 33, 56, 96 |
| □ | □ | 10. To ensure they are explicit and transparent, document the judgments made in formulating the recommendations and determining their strength. | 1–3, 5, 20, 23, 32, 33, 35–37, 43, 45–47, 57, 83, 96 |
| 14. Wording of recommendations and of considerations of implementation, feasibility and equity | |||
| □ | □ | 1. Decide on standardized wording to use for recommendation statements to ensure clarity and to maintain consistency throughout the guideline. Avoiding wording that may be vague and nonspecific. | 1–3, 5, 8, 20, 32, 33, 37, 39, 42, 45–47, 56, 96 |
| □ | □ | 2. Write the recommendations in a way that is actionable. Provide sufficient information so that it is not necessary for guideline users to refer to other material in order to understand the recommendation. | 1–3, 8, 20, 23, 32, 37, 39, 42, 45, 46, 56, 83, 97 |
| □ | □ | 3. Provide clear direction or an interpretation aid to describe the implication of the strength of recommendation for clinicians, patients, policy-makers and other target audience groups. | 8, 20, 23, 32, 33, 37, 39, 47, 56, 96 |
| □ | □ | 4. Indicate in the recommendation statements the population for which the recommendation is intended, the intervention being recommended and the alternative approach(es) or intervention(s). | 8, 20, 23, 32, 33, 37, 42, 56, 96, 97 |
| □ | □ | 5. Include remarks that describe the context, feasibility and applicability of the recommendation and highlight key considerations such as equity issues and specific conditions that might apply to the recommendation (e.g., whether the conditions outlined apply to a specific subpopulation, specific types of the intervention, for certain values and preferences, when certain resources are available). | 3, 5, 8, 20, 23, 32, 33, 37, 42, 56, 57, 65, 79, 81, 82, 96 |
| □ | □ | 6. Report the quality of evidence and the strength of recommendation in proximity to the recommendation statement. | 3, 8, 23, 32, 33, 37, 43, 45–47, 56, 83 |
| □ | □ | 7. Establish methods to be used by the group in agreeing on the final wording of recommendation statements (e.g., review and approval, formal consensus). | 5, 20, 45, 46, 56, 96 |
| □ | □ | 8. Report the recommendations in a way that is comprehensible and visible (e.g., do not embed recommendations within long paragraphs, group recommendations together in a summary section). | 1, 3, 8, 39, 42, 44, 56, 97 |
| 15. Reporting and peer review | |||
| □ | □ | 1. Develop or adopt a standardized format for reporting the guideline, with specific structure, headings and content. | 5, 23, 34, 35, 37–46, 56, 58, 83, 97 |
| □ | □ | 2. Decide on the format(s) to be prepared for the guideline product(s) (e.g., full guideline, full guideline with technical report/systematic reviews, brief guideline for clinicians or policy-makers, consumer version for patients) that will correspond to the dissemination plan. See also topic 16 | 3, 9, 20, 32, 34–37, 39–46, 56, 57, 83, 97 |
| □ | □ | 3. Decide who will be responsible for writing the guideline product(s) (e.g., subcommittee of the guideline working group) and decide on authorship (e.g., individual authors, organization as author, working group as author). See also topic 1 | 5, 20, 32, 33, 35, 37, 39, 42, 44, 45, 57, 97 |
| □ | □ | 4. Conduct a review of the final draft of the guideline report(s) by all members of the guideline development group, allowing sufficient opportunity for feedback, editing and revisions. | 5, 32, 33, 42, 43, 45, 85, 97 |
| □ | □ | 5. Seek approval from all members of the guideline development group for the final document(s). | 3, 5, 32, 35, 36, 42, 45, 46, 56, 72 |
| □ | □ | 6. Initiate organizational (i.e., internal) peer review. | 5, 9, 32–37, 41–45, 56, 85, 97 |
| □ | □ | 7. Decide on the method(s) for external peer review to review the final document(s) for accuracy, practicality, clarity, organization and usefulness of the recommendations, as well as to ensure input from broader and important perspectives that the guideline group did not encompass (e.g., invited peer review, public consultation period with incorporation of feedback and responses from the guideline development group, peer review provided by peer-reviewed journal). | 1–3, 5, 9, 32–35, 37–46, 56, 57, 70, 85, 97 |
| □ | □ | 8. Document the internal and external peer review process and, if applicable, publish consultation comments and the guideline development group’s responses. | 2, 9, 23, 32–34, 41–43, 45, 46, 56, 85, 97 |
| 16. Dissemination and implementation | |||
| □ | □ | 1. Prepare an active dissemination plan with various approaches to enhance the adoption of the guideline (e.g., make guideline available online, develop formal relationships with those in health care systems responsible for guideline dissemination and implementation to support guideline uptake, conduct press conference, develop social media strategy, disseminate guideline at professional society meetings, publish guideline in journal that is accessed by target audience). | 3, 5, 9, 20, 34–40, 42, 44–47, 56, 57, 84, 98 |
| □ | □ | 2. Develop or adapt tools, support and derivative products to provide guidance on how the recommendations can be implemented into practice (e.g., develop mobile applications, integrate guideline with clinical decision support systems, make guideline adaptable as an educational resource for target audience for education outreach). | 1, 3, 5, 9, 20, 23, 32, 35–37, 41–46, 82, 97 |
| □ | □ | 3. Make considerations for adaptation of the guideline and provide specific instructions for how target end users who would like to adapt the guidelines to other contexts can do so in a systematic and transparent way (e.g., modifying a recommendation based on local resources and baseline risk, implications that deviate from the judgments made by the guideline panel). | 1, 44–47, 82, 99 |
| □ | □ | 4. Set rules and regulations for translation of the guideline into other languages (e.g., allow translation by third-party organizations following approval by the guideline group, include staff responsible for translation in guideline working group). | 41, 45, 99 |
| 17. Evaluation and use | |||
| □ | □ | 1. Conduct an internal evaluation (i.e., self-assessment) of the guideline development process, including the guideline panel meeting(s) held to formulate recommendations, by asking guideline group members for feedback. | E, 85 |
| □ | □ | 2. Consider pilot testing the guideline with the target end users (e.g., members of target audience and stakeholders who participated in the guideline development group). | 1, 23, 43, 44, 47, 85 |
| □ | □ | 3. Provide criteria and tools for target end users to monitor and audit the implementation and use of the guideline recommendations (e.g., identify outcomes that should change with implementation and suggest methods for measuring the outcomes). | 1, 3, 9, 20, 23, 35, 36, 39, 42–47, 56, 57, 85, 99 |
| □ | □ | 4. Provide support and tools for prospective evaluation of the guideline to determine its effectiveness after implementation (e.g., use randomized evaluations where possible, use before–after evaluations cautiously because of uncertainties regarding the effects of implementation). | 20, 35, 42, 44–47, 84, 85 |
| □ | □ | 5. Consider the potential involvement of the guideline development group in prospective evaluation(s) of the guideline (e.g., partnering with organizations that implement the guideline to plan evaluation studies). | 3, 41, 42, 44, 45, 84, 85 |
| □ | □ | 6. Plan to collect feedback and evaluations from end users to identify how to improve the intrinsic implementability of the recommendations in subsequent versions of the guideline. | 3, 20, 35, 42, 43, 47, 85, 99 |
| 18. Updating | |||
| □ | □ | 1. Set a policy, procedure and timeline for routinely monitoring and reviewing whether the guideline needs to be updated (e.g., update systematic review every 3 years to determine whether new evidence is available). | 1–3, 5, 9, 20, 23, 32, 35–37, 39–47, 56, 57, 61, 85, 99 |
| □ | □ | 2. Decide who will be responsible for routinely monitoring the literature and assessing whether new evidence is available (e.g., consider involvement of experts not previously involved in the guideline development group to periodically review the guideline). | 3, 5, 32, 37, 39, 42–45, 56, 57, 85, 99 |
| □ | □ | 3. Set the conditions that will determine when a partial or a full update of the guideline is required (e.g., whether only certain recommendation statements need to be updated; whether many recommendations are out of date, making the entire guideline invalid; or whether recommendations need to be added for newly available treatments). | 2, 3, 9, 20, 35, 37, 40, 42–45, 56, 57, 61, 85 |
| □ | □ | 4. Make arrangements for guideline group membership and participation after completion of the guideline (e.g., rotating membership every 1–2 years, selection of a new group at time of updating, continuing participation by guideline panel chair). | 33, 37, 41, 42, 56, 85, 99 |
| □ | □ | 5. Plan the funding and logistics for updating the guideline in the future (e.g., secure ongoing funding, form a standing oversight committee to oversee the updating process). | 44, 85 |
| □ | □ | 6. Document the plan and proposed methods for updating the guideline to ensure they are followed. | 1, 23, 61, 44, 57 |
Note: E = item informed by expert consultation, GRADE = Grading of Recommendations Assessment, Development and Evaluation.
Interpretation
Based on a thorough review of available sources for guideline developers, we have developed a comprehensive checklist of practical steps with 146 items arranged under 18 topics for consideration in the context of guideline development. The list, which defines the complete guideline process, is not intended to evaluate the credibility of a guideline or to describe the minutiae of the steps to take (e.g., how to conduct a systematic review to support a specific recommendation). Its purpose is to function as a publicly available and interactive resource, with links to learning tools and training materials, for those interested in beginning, enhancing or evaluating their guideline development process.
Strengths and limitations
Although many manuals for guideline development exist, our checklist of items is free of institutional or organizational constraints. The main strength of this study is the extent of our review of key sources in this field. In addition, the experience some of us had over the years in using various manuals for guideline development facilitated the identification of gaps and relevant resources.
A potential limitation of our checklist is that it is lengthy. However, the list is intended to be generic, to cover all topics and items that should be considered when developing a guideline, and to be applicable to a variety of guidelines, including clinical, policy and public health guidelines. Those planning to use the checklist should keep that in mind and ignore steps that might not be relevant to their topic or area, although many of the items will be essential for producing guidelines that will be judged trustworthy.
Another potential limitation is that the checklist appears to give equal weight to each step. Guideline developers face challenges of limited time, funding and human resources and often need to prioritize one step over another. Having a comprehensive checklist will allow developers to be explicit about which steps they want to prioritize and why.
The field of guideline development is constantly evolving, particularly in areas that lack more formal guidance, such as incorporation of the relative importance of outcomes and interventions, and patient values and preferences. We used current guideline manuals and methodology reports to develop the checklist; therefore, areas that lack formal guidance may not be well covered in the checklist. In addition, we have only begun to evaluate the checklist with guideline developers who were not involved in its development (e.g., in a project focusing on rare diseases [www.rarebestpractices.eu]). However, we will solicit feedback from users of the online version of the checklist and use it to revise the checklist and keep it up to date.
Conclusion
Our checklist represents a comprehensive list of items to consider during the development of guidelines. Although the checklist covers all stages in the guideline process, from planning to implementation of a guideline, there are several areas in the guideline enterprise for which further guidance is needed, as shown by the deficiency of detail about specific items we identified. Current guidelines for guidelines and guideline development manuals place a heavy focus on systematic reviews of the evidence and clinical outcomes, whereas less attention is paid to providing guidance on considering resource use, patients’ and consumers’ values and preferences, and equity.
Similarly, there are several areas for which training tools and resources for implementation of the steps are lacking. Manuals describe the overall steps and considerations needed in developing guidelines, but specific details about how the process is accomplished are often omitted. Details about guideline group interaction and group processes such as conflict resolution may be internally documented, but support through tools or learning materials would be useful to individuals seeking instruction for developing their own guidelines. Moreover, instructions about the practical steps and processes for guideline development would ensure consistency among different guideline groups within an organization. Our checklist focuses on the processes and practical steps of guideline development.
The checklist is intended for use by guideline developers to plan and track the process of guideline development and to help the developers ensure that no key steps are missed. Guideline developers may, with justification, skip items that may not be applicable to the specific guideline or the organization. However, following the steps will ensure that key items are covered and increase the likelihood of the guideline achieving higher scores when evaluated with credibility assessment tools. Users of the checklist should become familiar with the topics and items before applying them and realize that the items are interrelated and not necessarily sequential (Figure 2).
As part of our work with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group (www.gradeworkinggroup.org) and the DECIDE project (www.decide-collaboration.eu), we will implement this checklist in our electronic, Web-based Guideline Development Tool (www.guidelinedevelopment.org), freely available to nonprofit organizations.
Future work will focus on our obtaining additional feedback through crowdsourcing, keeping this checklist up to date and providing further practical, training and learning resources on our interactive website (http://cebgrade.mcmaster.ca/guidecheck.html) and the electronic Guideline Development Tool. This work will include more formal testing and assessment of which steps are proving to be most time and resource intensive.
Acknowledgement
The authors thank Charmaine Fraser for her administrative assistance.
Footnotes
Competing interests: None declared. Authors of this manuscript have been involved in the development of various guideline manuals which are referenced in this article.
This article has been peer reviewed.
Contributors: Holger Schünemann was the study supervisor, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Holger Schünemann, Wojtek Wiercioch, Itziar Etxeandia, Maicon Falavigna, Reem Mustafa, Matthew Ventresca, Romina Brignardello-Petersen, Kaja-Triin Laisaar, Sérgio Kowalski, Tejan Baldeh, Yuan Zhang, Ulla Raid, Ignacio Neumann, Susan Norris, Judith Thornton, Robin Harbour, Shaun Treweek, Gordon Guyatt, Pablo Alonso-Coello, Andrew Oxman and Elie Akl contributed to the acquisition of data. All of the authors contributed to the analysis and interpretation of data. Holger Schünemann and Wojtek Wiercioch drafted the manuscript; all of the authors revised the manuscript critically for important intellectual content and approved the final version submitted for publication.
Funding: This study was supported by funds from McMaster University. Holger Schünemann, Judith Thornton, Robin Harbour, Shaun Treweek, Pablo Alonso-Coello, Jan Brošek and Andrew Oxman received funding from the European Union FP7 Programme (grant no. 258583) as part of the DECIDE (Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence) project.
References
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- Values and preferences in COVID-19 public health guidelines: A systematic review
- Measuring evidence-based clinical guideline compliance in the paediatric intensive care unit
- ESPGHAN and NASPGHAN 2023 protocol for paediatric FAPD treatment guidelines (standard operating procedure)
- Implementation of the updated NICE haematological cancers (NG47) improving outcomes guidelines across Specialist Integrated Haematological Malignancy Diagnostic Services (SIHMDS) in England: a UK NEQAS LI survey
- Recommandations sur le depistage pour la prevention primaire des fractures de fragilisation
- Investigation of bias due to selective inclusion of study effect estimates in meta-analyses of nutrition research
- Recommendations on screening for primary prevention of fragility fractures
- Health equity considerations in guideline development: a rapid scoping review
- Canadian Association of Radiologists Diagnostic Imaging Referral Guidelines: a guideline development protocol
- BSG 2024 IBD guidelines protocol (standard operating procedures)
- Which actionable statements qualify as good practice statements In Covid-19 guidelines? A systematic appraisal
- Investigation of bias due to selective inclusion of study effect estimates in meta-analyses of nutrition research
- Evaluation of Conflicts of Interest among Participants of the Japanese Nephrology Clinical Practice Guideline
- Abdominal aortic aneurysm clinical practice guidelines: a methodological assessment using the AGREE II instrument
- Quality assessment of clinical practice guidelines for perioperative care and use of GRADE: a systematic review protocol
- Methods used to select results to include in meta-analyses of nutrition research: a meta-research study
- Clinical practice guideline adaptation methods in resource-constrained settings: four case studies from South Africa
- Recommandations canadiennes pour les pratiques optimales de prise en charge de la sclerose laterale amyotrophique
- Canadian best practice recommendations for the management of amyotrophic lateral sclerosis
- Financial conflict of interest among clinical practice guideline-producing organisations
- Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development
- Diagnosing acute aortic syndrome: a Canadian clinical practice guideline
- 'Building on shaky ground--challenges to and solutions for primary care guideline implementation in four provinces in South Africa: a qualitative study
- Management of the neurologically deceased organ donor: A Canadian clinical practice guideline
- Extending the RIGHT statement for reporting adapted practice guidelines in healthcare: the RIGHT-Ad@pt Checklist protocol
- Reporting, presentation and wording of recommendations in clinical practice guideline for gout: a systematic analysis
- American Society of Hematology 2018 guidelines for management of venous thromboembolism: diagnosis of venous thromboembolism
- American Society of Hematology 2018 Guidelines for management of venous thromboembolism: treatment of pediatric venous thromboembolism
- American Society of Hematology 2018 guidelines for management of venous thromboembolism: prophylaxis for hospitalized and nonhospitalized medical patients
- American Society of Hematology 2018 guidelines for management of venous thromboembolism: optimal management of anticoagulation therapy
- American Society of Hematology 2018 guidelines for management of venous thromboembolism: heparin-induced thrombocytopenia
- Strengthening the quality of paediatric primary care: protocol for the process evaluation of a health systems intervention in South Africa
- PACK Child: the development of a practical guide to extend the scope of integrated primary care for children and young adolescents
- A systematic review of how patients value COPD outcomes
- Choosing guidelines to use in your practice
- Choisir les guides de pratique clinique a utiliser
- Deprescribing benzodiazepine receptor agonists: Evidence-based clinical practice guideline
- Deprescription des agonistes des recepteurs des benzodiazepines: Lignes directrices de pratique clinique fondees sur les donnees probantes
- Deprescribing antipsychotics for behavioural and psychological symptoms of dementia and insomnia: Evidence-based clinical practice guideline
- Deprescription des antipsychotiques pour les symptomes comportementaux et psychologiques de la demence et linsomnie: Guide de pratique clinique fonde sur des donnees probantes
- Deprescribing antihyperglycemic agents in older persons: Evidence-based clinical practice guideline
- Deprescription des antihyperglycemiants chez les personnes agees: Guide de pratique clinique fonde sur les donnees probantes
- Perspective: Improving Nutritional Guidelines for Sustainable Health Policies: Current Status and Perspectives
- Deprescrire les inhibiteurs de la pompe a protons: Lignes directrices de pratique clinique fondees sur les donnees probantes
- Deprescribing proton pump inhibitors: Evidence-based clinical practice guideline
- World Health Organization treatment guidelines for drug-resistant tuberculosis, 2016 update
- The Saudi clinical practice guideline for the prophylaxis of venous thromboembolism in medical and critically ill patients
- GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines
- Do "Evidence-Based Recommendations" Need to Reveal the Evidence? Minimal Criteria Supporting an "Evidence Claim"
- The development of guideline implementation tools: a qualitative study
- Do guidelines offer implementation advice to target users? A systematic review of guideline applicability
More in this TOC Section
Similar Articles
Collections
