RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions

Hyun-Ju Seo; Soo Young Kim; Yoon Jae Lee; Ji-Eun Park

doi:10.4082/kjfm.23.0034

Korean J Fam Med > Volume 44(5); 2023 > Article

Seo, Kim, Lee, and Park: RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions

Review Article

Korean Journal of Family Medicine 2023;44(5):249-260.

Published online: July 7, 2023

DOI: https://doi.org/10.4082/kjfm.23.0034

RoBANS 2: A Revised Risk of Bias Assessment Tool for Nonrandomized Studies of Interventions

Hyun-Ju Seo¹

, Soo Young Kim^2,^*

, Yoon Jae Lee³

, Ji-Eun Park⁴

¹College of Nursing, Chungnam National University, Daejeon, Korea

²Department of Family Medicine, College of Medicine, Hallym University, Seoul, Korea

³Jaseng Spine and Joint Research Institute, Jaseng Medical Foundation, Seoul, Korea

⁴Center for Global Collaboration, Korea Institute of Oriental Medicine, Daejeon, Korea

^*Corresponding Author: Soo Young Kim
Tel: +82-2-2224-2406, Fax: +82-2-2224-2409, E-mail: hallymfm@gmail.com

Received April 6, 2023 Revised April 21, 2023 Accepted April 26, 2023

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Assessment of the risk of bias is an essential component of any systematic review. This is true for both nonrandomized studies and randomized trials, which are the main study designs of systematic reviews. The Risk of Bias Assessment Tool for Nonrandomized Studies (RoBANS) was developed in 2013 and has gained wide usage as a risk-of-bias assessment tool for nonrandomized studies. Four risk-of-bias assessment experts revised it by reviewing existing assessment tools and user surveys. The main modifications included additional domains of selection and detection bias susceptible to nonrandomized studies of interventions, a more detailed consideration of the comparability of participants, and more reliable and valid outcome measurements. A psychometric assessment of the revised RoBANS (RoBANS 2) revealed acceptable inter-rater reliability (weighted kappa, 0.25 to 0.49) and construct validity in which intervention effects of studies with an unclear or high risk of bias were overestimated. The RoBANS 2 has acceptable feasibility, fair-to-moderate reliability, and construct validity. It provides a comprehensive framework for allowing authors to assess and understand the plausible risk of bias in nonrandomized studies of interventions.

Keywords: Systematic Review; Risk Assessment; Bias; Nonrandomized Studies

INTRODUCTION

The findings of nonrandomized studies of interventions (NRSI) can be generalized to the population and provide clinical evidence regarding the benefits or risks of healthcare interventions [1]. Researchers have increasingly included NRSI in systematic reviews to examine various interventions such as medications, hospital procedures, community health interventions, and health systems [2]. Moreover, these reviews may allow for the evaluation of adverse events and long-term effects after exposure to healthcare interventions [3].

Real-world evidence has become increasingly critical for identifying the effects and safety of healthcare interventions. The inclusion of nonrandomized studies and randomized controlled trials (RCTs) in systematic reviews to address these effects is becoming increasingly essential. Actual evidence, including nonrandomized studies, is provided by practitioners, investigators, and regulatory and health technology assessments in real-world setting [4]. However, a crucial limitation of observational studies on intervention effects and adverse reactions is that the intervention of interest is not randomly assigned, blinding is lacking, and there is often no comparison. Thus, the study findings are susceptible to confounding and selection biases, which could result in biased estimates of intervention effects compared with smaller RCTs [5,6]. Therefore, the risk of bias of NRIS must be assessed when undertaking a systematic review while considering the strengths and weaknesses of real-world evidence research.

The Risk of Bias Assessment Tool for Nonrandomized Studies (RoBANS), published in 2013 [7], is a widely used bias tool. Since its publication, several critiques and users provided feedback on the instrument. We decided to reflect on the following feedbacks: simplification of the domain from a question to an item format, judgment criteria, and guidance by the study design of the NRSI. Moreover, advancements in risk of bias science necessitate the revision and updating of the original RoBANS tool.

DEVELOPMENT OF THE REVISED ROBANS (ROBANS 2)

To revise the RoBANS, we reviewed the previous risk of bias or critical appraisal checklists for nonrandomized studies, such as the Scottish Intercollegiate Guidelines Network [8], Newcastle-Ottawa Scale [9], Agency for Healthcare Research and Quality checklist [10], and the RTI (Research Triangle Institute) Item Bank’s risk of bias tool [11]. Several consultative meetings with five systematic review methodologists who are experts in the fields of evidence-based medicine, epidemiology, and biostatistics and who are users of the original RoBANS were held to provide feedback and advice on the plausible risk of bias when conducting nonrandomized studies.

A sample of various types of nonrandomized studies for the assessment of the risk of bias using the revised version of RoBANS (RoBANS 2) was compiled by contacting the National Evidence-based Healthcare Collaborating Agency, Department of Evidence-based Health in Health Insurance Review, and Assessment Service funded by the Korean government. Additionally, PubMed and the Cochrane Library were searched to retrieve systematic reviews of NRSI.

The inclusion criteria for nonrandomized studies to evaluate interrater reliability and construct validity were as follows: (1) studies in which the control group had no intervention or placebo control; (2) studies with dichotomous outcome data, except before-and-after studies; and (3) studies included in systematic reviews of cohort studies, case-control studies, and cross-sectional or before-and-after studies.

The minimum number of studies required by the two raters was 85, based on a dichotomous variable with 80% power, to detect a kappa of 0.70, at a proportion of positive ratings of 0.70. The null hypothesis value of kappa was 0.40 [12]. Consequently, we selected 112 studies to cover all relevant nonrandomized study designs, including 45 cohort studies, 16 case-control studies, 25 cross-sectional studies, and 26 before-and-after studies (Appendix 1).

Paired assessors of the review team independently evaluated the risk of bias of the included studies using RoBANS 2 after pilot testing a sample of included studies. All assessors had doctoral degrees and at least 10 years of experience in conducting systematic reviews. Each study was randomly assigned to paired assessors using computer-based random number generation. The software packages SAS ver. 9.4 (SAS Institute Inc., Cary, NC, USA) and Stata SE ver. 16.0 (Stata Corp., College Station, TX, USA) were used for the statistical analyses. All statistical tests were two-sided, with a significance level of 0.05.

1. The Key Changes of the Revised RoBANS 2

Similar to the original version, RoBANS 2 is an outcome-based checklist. Additionally, the domains of blinding of outcome assessors, outcome assessment, and incomplete outcome data can be treated as result-based evaluations because they are classified as patient-reported outcomes or objective outcome measures.

In nonrandomized studies, selection bias occurs when participants chosen for the intervention of interest have different characteristics from those allocated to the alternative intervention (or not treated) because the choice of a given intervention might be affected by the discretion of the treating clinician or patient preference, patient characteristics, and clinical history [13]. This might result in incomparable comparison groups. Consequently, confounding by indication or severity introduces systematic bias, leading to either over- or underestimation of treatment effects depending on the treatment decision mechanism [14]. Therefore, we separated the existing domain of participant selection into the comparability of the participants and target group selection in RoBANS 2. These revised items may address confounders by indication or severity and evaluate the inadequate selection of participants, including the absence of outcomes among the study participants at the beginning of the study and being representative of the population between the treatment groups.

Differential or non-differential misclassification of the outcome data could introduce detection bias in NSRI [15]. Bias can occur when outcome assessors are aware of the intervention status, if different methods are used to assess outcomes in different intervention groups, or if measurement errors are related to the intervention status or effects [16]. In RoBANS 2, we revised the domain of blinding of outcome assessments to blinding of outcome assessors and reliable and valid outcome assessment methods to consider biases related to the ascertainment of outcomes and measurement methods in the NRSI (Table 1) [17,18].

2. Psychometric Characteristics of RoBANS 2

1) Feasibility

To evaluate the ease of use of RoBANS 2, independent assessors of the paired team measured the time to complete the risk of bias assessment and then calculated the mean time. The time spent assessing each study ranged from 20 seconds to 36.35 minutes, with a mean of 8.72±5.00 minutes per article. As the nonrandomized studies included in the risk of bias assessment not only covered a variety of research topics but also had diverse study designs, the time required to conduct the evaluation varied.

2) Inter-rater reliability

To determine the interrater reliability of RoBANS 2, we calculated the weighted kappa (κ) statistics for each domain of the risk-of-bias tool [19]. The agreement was categorized as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00) [20]. A summary of the inter-rater reliability of the RoBANS 2 is presented in Table 2. All domains of eight RoBANS 2 had fair agreement or higher, ranging from 0.25 to 0.49 κ statistics.

3) Validity

Construct validity was examined by comparing the effect size of each domain of the risk of bias assessed using RoBANS 2. Among 112 nonrandomized studies included for inter-rater reliability, 77 studies excluding 26 before-and-after studies without comparison and nine studies unable to extract data were included for construct validity.

The effect sizes were calculated using Cohen’s d statistic for continuous outcomes. For dichotomous outcomes, the odds ratios (ORs) were converted into effect sizes using Hasselblad and Hedges’s transformation method [21,22]. The risk of bias was classified as low, unclear, or high risk of bias [7]. We then explored the association between the effect size of the primary outcomes and domain-specific risk of bias using ORs. Most included nonrandomized studies were comparative studies with no intervention, except before-and-after studies. The primary outcomes were objective and unintended outcomes of the intervention such as mortality, head injury, and influenza-like illnesses. Hence, a lower ORs indicates a greater effect of the intervention than in the control group. Specifically, ORs less than 1 indicated that the pooled effect sizes showed a protective effect of the intervention on unintended outcomes, such as mortality. Statistical analyses were conducted to identify the association between the risk of bias domain and the effect size using the Review Manager 5 software package (RevMan version 5.4; Cochrane, London, UK) [23]. Our findings revealed that studies conducted inadequately for each domain of the risk of bias were likely to report low ORs in seven of eight domains (Table 3). In other words, intervention effect studies with an unclear or high risk of bias were overestimated. Therefore, the RoBANS 2 has construct validity and can detect significant differences in effect size estimates according to the risk of bias.

DISCUSSION

We revised the RoBANS tool to assess the risk of bias in the results of nonrandomized studies, including cohort studies, case-control studies, cross-sectional studies, and before and after intervention studies. Our aim was to address the limitations identified since its publication in 2013. The main modifications included additional domains of selection and detection bias susceptible to the NRSI, a more detailed consideration of the comparability of participants, and more reliable and valid outcome measurements. Similar to the original RoBANS, the assessments in RoBANS 2 were related to the risk of bias in the estimates of the intervention effect for a single outcome or endpoint rather than at the study level. We recommend that the overall risk of bias in the results or outcomes assessed using the RoBANS 2 generally yields the worst risk of bias in any of the domains or certain critical domains. In other words, the assessors of risk of bias could justify and choose critical domains, such as the selection of participants, confounders, and measurement of exposure. Additionally, the assessors can assess the susceptibility to bias in the observational epidemiology of the research question of interest to reach a consensus on the overall risk of bias judgments. The overall judgments can then be incorporated to rate the confidence of the conclusions and be compatible with the grading of recommendations, assessment, development, and evaluations [24].

The RoBANS 2 is a comprehensive checklist instrument for assessing the risk of bias in cohort studies, case-control studies, cross-sectional studies, and before and after studies of interventions with user guidance to support educational purposes and improve inter-rater agreement of the assessment results (Appendix 2). It is expected to gain wide usage in systematic reviews and in clinical practice guideline development [25]. However, when applied to the risk of bias assessment of controlled before-after studies, interrupted time series, and interrupted time series with comparisons, assessors need to determine how to judge the risk of bias in each domain, considering the nature of study designs from epidemiological experts. We recommend using the Cochrane revised risk-of-bias tool for RCTs, non-RCTs, or quasi-experimental trials [26].

Further research is needed to compare the inter-rater agreement and usability of both the RoBANS 2 and Risk of Bias In Nonrandomized Studies of Interventions tools, specifically for studies with a cohort-type design [16]. However, the tools overlap substantially in terms of the risk of bias domains (Appendix 3).

CONCLUSION

In conclusion, RoBANS 2 had acceptable feasibility, fair to moderate reliability, and construct validity. Although further refinement and extensive feedback from RoBANS 2 users are required, we expect RoBANS 2 to be useful for review authors since it provides a comprehensive framework for assessing and understanding the plausible risk of bias in NRSI.

Notes

CONFLICT OF INTEREST

No potential conflict of interest relevant to this article was reported.

FUNDING

This study was supported by a grant from the Health Insurance Review and Assessment Service (HIRA) of the Republic of Korea.

ACKNOWLEDGMENTS

We are grateful to Prof. Seokyung Hahn, Prof. Seung-Soo Sheen, Prof. Juneyoung Lee, Prof. Sung-Il Cho, and Prof. Sun-Young Jung for their advice and comments regarding the revision of RoBANS.

Table 1.

Revised version of the risk of bias assessment tool for nonrandomized studies (RoBANS 2)

Domain	Details	Risk of bias
Comparability of the target group	Selection bias due to the selection of an inappropriate comparison target group	□ Low
		□ High
		□ Unclear
Target group selection	Selection bias due to inappropriate intervention or inappropriate selection of exposure group or patient group	□ Low
		□ High
		□ Unclear
Confounders	Selection bias due to inappropriate confounder confirmation and consideration	□ Low
		□ High
		□ Unclear
Measurement of intervention/exposure	Performance bias due to inappropriate intervention or inappropriate exposure measurement	□ Low
		□ High
		□ Unclear
Blinding of assessors	Detection bias due to inappropriate blinding of assessors	□ Low
		□ High
		□ Unclear
Outcome assessment	Detection bias due to inappropriate outcome assessment methods	□ Low
		□ High
		□ Unclear
Incomplete outcome data	Attrition bias due to inappropriate handling of incomplete data	□ Low
		□ High
		□ Unclear
Selective outcome reporting	Reporting bias due to selective outcome reporting	□ Low
		□ High
		□ Unclear

RoBANS 2, revised version of Risk of Bias Assessment tool for Nonrandomized Studies.

Table 2.

Inter-rater agreement between the two raters for RoBANS 2 (N=112)

Domain	Weighted κ statistics (95% CI)	Interpretation
Comparability of the target group	0.37 (0.23–0.51)	Fair agreement
Target group selection	0.36 (0.26–0.43)	Fair agreement
Confounders	0.49 (0.42–0.59)	Moderate agreement
Measurement of intervention/exposure	0.32 (0.20–0.38)	Fair agreement
Blinding of assessors	0.25 (0.21–0.30)	Fair agreement
Outcome assessment	0.45 (0.33–0.55)	Moderate agreement
Incomplete outcome data	0.43 (0.33–0.54)	Moderate agreement
Selective outcome reporting	0.45 (0.25–0.58)	Moderate agreement

RoBANS 2, revised version of Risk of Bias Assessment tool for Nonrandomized Studies; CI, confidence interval.

Table 3.

Effect estimates for studies categorized as low and unclear or high risk of bias by domain (N=77)

Domain	Risk of bias assessments (OR, 95% CI)		Between P-value
Domain	Low risk of bias	Unclear or high risk of bias	Between P-value
Comparability of the target group	0.64 (0.59–0.68)	0.53 (0.51–0.55)	<0.001
Target group selection	0.65 (0.61–0.69)	0.52 (0.50–0.54)	<0.001
Confounders	0.63 (0.61–0.66)	0.47 (0.45–0.49)	<0.001
Measurement of intervention/exposure	0.59 (0.56–0.63)	0.53 (0.51–0.55)	0.003
Blinding of assessors	0.62 (0.57–0.67)	0.53 (0.52–0.55)	0.001
Outcome assessment	0.59 (0.56–0.62)	0.52 (0.50–0.54)	0.001
Incomplete outcome data	0.56 (0.53–0.60)	0.54 (0.52–0.56)	0.31
Selective outcome reporting	0.51 (0.48–0.54)	0.56 (0.54–0.58)	0.01

OR, odds ratio; CI, confidence interval.

REFERENCES

1. Chodankar D. Introduction to real-world evidence studies. Perspect Clin Res 2021;12:171-4.

2. Reves BC, Deeks JJ, Higgins JP, Shea B, Tugwell P, Wells GA, et al. Including non-randomized studies on intervention effects. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane handbook for systematic reviews of interventions. 2nd ed. Hoboken (NJ): Wiley-Blackwell; 2019. p. 595-619

3. Monti S, Grosso V, Todoerti M, Caporali R. Randomized controlled trials and real-world data: differences and similarities to untangle literature data. Rheumatology (Oxford) 2018;57(57 Suppl 7):vii54-8.

4. Kim HS, Lee S, Kim JH. Real-world evidence versus randomized controlled trial: clinical research based on electronic medical records. J Korean Med Sci 2018;33:e213.

5. Tang M, Pearson SA, Simes RJ, Chua BH. Harnessing real-world evidence to advance cancer research. Curr Oncol 2023;30:1844-59.

6. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008.

7. Kim SY, Park JE, Lee YJ, Seo HJ, Sheen SS, Hahn S, et al. Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. J Clin Epidemiol 2013;66:408-14.

8. Scottish Intercollegiate Guidelines Network (SIGN). Checklists [Internet]. Edinburgh: SIGN; c2023 [cited 2023 Mar 24]. Available from: https://www.sign.ac.uk/what-we-do/methodology/checklists/

9. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses [Internet]. Ottawa (ON): Ottawa Hospital Research Institute; c2021 [cited 2023 Mar 29]. Available from: https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

10. Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Validity and inter-rater reliability testing of quality assessment instruments [Internet]. Rockville (MD): Agency for Healthcare Research and Quality; 2012 [cited 2023 Mar 29]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK92293/

11. Viswanathan M, Berkman ND. Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol 2012;65:163-78.

12. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005;85:257-68.

13. Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:1-173.

14. Kyriacou DN, Lewis RJ. Confounding by indication in clinical research. JAMA 2016;316:1818-9.

15. Desai RJ, Levin R, Lin KJ, Patorno E. Bias implications of outcome misclassification in observational studies evaluating association between treatments and all-cause or cardiovascular mortality using administrative claims. J Am Heart Assoc 2020;9:e016906.

16. Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919.

17. Groenwold RHH, Dekkers OM. Measurement error in clinical research, yes it matters. Eur J Endocrinol 2020;183:E3-5.

18. Wirtz HS, Calip GS, Buist DSM, Gralow JR, Barlow WE, Gray S, et al. Evidence for detection bias by medication use in a cohort study of breast cancer survivors. Am J Epidemiol 2017;185:661-72.

19. Warrens MJ. Weighted kappas for 3×3 tables. J Probab Stat 2013;2013:325831.

20. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.

21. Hasselblad V, Hedges LV. Meta-analysis of screening and diagnostic tests. Psychol Bull 1995;117:167-78.

22. Sanchez-Meca J, Marin-Martínez F, Chacon-Moscoso S. Effect-size indices for dichotomized outcomes in meta-analysis. Psychol Methods 2003;8:448-67.

23. Cochrane. Review Manager (RevMan): version 5.4 [Internet]. London, The Cochrane Collaboration. [date unknown] [cited 2023 Mar 24]. Available from: https://training.cochrane.org/online-learning/coresoftware/revman

24. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64:383-94.

25. Kim SY. Recent advance in clinical practice guideline development methodology. Korean J Fam Med 2022;43:347-52.

26. Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomized trials. BMJ 2019;366:l4898.

Appendices

Appendix 1.

The list of non-randomized studies included for assessment

Roder D, Houssami N, Farshid G, Gill G, Luke C, Downey P, et al. Population screening and intensity of screening are associated with reduced breast cancer mortality: evidence of efficacy of mammography screening in Australia. Breast Cancer Res Treat 2008;108:409-16.

Allgood PC, Warwick J, Warren RM, Day NE, Duffy SW. A case-control study of the impact of the East Anglian breast screening programme on breast cancer mortality. Br J Cancer 2008;98:206-9.

Fielder HM, Warwick J, Brook D, Gower-Thomas K, Cuzick J, Monypenny I, et al. A case-control study to estimate the impact on breast cancer death of the breast screening programme in Wales. J Med Screen 2004;11:194-8.

Gabe R, Tryggvadottir L, Sigfusson BF, Olafsdottir GH, Sigurdsson K, Duffy SW. A case-control study to estimate the impact of the Icelandic population-based mammography screening program on breast cancer death. Acta Radiol 2007;48:948-55.

Miltenburg GA, Peeters PH, Fracheboud J, Collette HJ. Seventeen-year evaluation of breast cancer screening: the DOM project, The Netherlands. Diagnostisch Onderzoek (investigation) Mammacarcinoom. Br J Cancer 1998;78:962-5.

Puliti D, Miccinesi G, Collina N, De Lisi V, Federico M, Ferretti S, et al. Effectiveness of service screening: a case-control study to assess breast cancer mortality reduction. Br J Cancer 2008;99:423-7.

van Schoor G, Moss SM, Otten JD, Donders R, Paap E, den Heeten GJ, et al. Increasingly strong reduction in breast cancer mortality due to screening. Br J Cancer 2011;104:910-4.

Otto SJ, Fracheboud J, Verbeek AL, Boer R, Reijerink-Verheij JC, Otten JD, et al. Mammography screening and risk of breast cancer death: a population-based case-control study. Cancer Epidemiol Biomarkers Prev 2012;21:66-73.

Ahmed AE, Nicholson KG, Nguyen-Van-Tam JS. Reduction in mortality associated with influenza vaccine during 1989-90 epidemic. Lancet 1995;346:591-5.

Ahmed AH, Nicholson KG, Nguyen-van Tam JS, Pearson JC. Effectiveness of influenza vaccine in reducing hospital admissions during the 1989-90 epidemic. Epidemiol Infect 1997;118:27-33.

Foster DA, Talsma A, Furumoto-Dawson A, Ohmit SE, Margulies JR, Arden NH, et al. Influenza vaccine effectiveness in preventing hospitalization for pneumonia in the elderly. Am J Epidemiol 1992;136:296-307.

Jordan RE, Hawker JI, Ayres JG, Tunnicliffe W, Adab P, Olowokure B, et al. A case-control study of elderly patients with acute respiratory illness: effect of influenza vaccination on admission to hospital in winter 2003-2004. Vaccine 2007;25:7909-13.

Mullooly JP, Bennett MD, Hornbrook MC, Barker WH, Williams WW, Patriarca PA, et al. Influenza vaccination programs for elderly persons: cost-effectiveness in a health maintenance organization. Ann Intern Med 1994;121:947-52.

Ohmit SE, Monto AS. Influenza vaccine effectiveness in preventing hospitalization among the elderly during influenza type A and type B seasons. Int J Epidemiol 1995;24:1240-8.

Puig-Barbera J, Diez-Domingo J, Varea AB, Chavarri GS, Rodrigo JA, Hoyos SP, et al. Effectiveness of MF59-adjuvanted subunit influenza vaccine in preventing hospitalisations for cardiovascular disease, cerebrovascular disease and pneumonia in the elderly. Vaccine 2007;25:7313-21.

Puig-Barbera J, Diez-Domingo J, Perez Hoyos S, Belenguer Varea A, Gonzalez Vidal D. Effectiveness of the MF59-adjuvanted influenza vaccine in preventing emergency admissions for pneumonia in the elderly over 64 years of age. Vaccine 2004;23:283-9.

Rudenko LG, Arden NH, Grigorieva E, Naychin A, Rekstin A, Klimov AI, Donina S, Desheva J, Holman RC, DeGuzman A, Cox NJ, Katz JM. Immunogenicity and efficacy of Russian live attenuated and US inactivated influenza vaccines used alone and in combination in nursing home residents. Vaccine 2000;19:308-18.

Brandt MM, Ahrns KS, Corpron CA, Franklin GA, Wahl WL. Hospital cost is reduced by motorcycle helmet use. J Trauma Acute Care Surg 2002;53:469-71.

Conrad P, Bradshaw YS, Lamsudin R, Kasniyah N, Costello C. Helmets, injuries and cultural definitions: motorcycle injury in urban Indonesia. Accid Anal Prev 1996;28:193- 200.

Deutermann W. Motorcycle helmet effectiveness revisited. Washington (DC): National Center for Statistics and Analysis; 2004.

Diemath HE. Head injuries due to motorcycle accidents: crash helmets and alcoholism. Neurosurg Rev 1989;12 Suppl 1:458-64.

Eastridge BJ, Shafi S, Minei JP, Culica D, McConnel C, Gentilello L. Economic impact of motorcycle helmets: from impact to discharge. J Trauma 2006;60:978-84.

Gopalakrishna G, Peek-Asa C, Kraus JF. Epidemiologic features of facial injuries among motorcyclists. Ann Emerg Med 1998;32:425-30.

Hundley JC, Kilgo PD, Miller PR, Chang MC, Hensberry RA, Meredith JW, et al. Non-helmeted motorcyclists: a burden to society?: a study using the National Trauma Data Bank. J Trauma 2004;57:944-9.

Javouhey E, Guérin AC, Chiron M. Incidence and risk factors of severe traumatic brain injury resulting from road accidents: a population-based study. Accid Anal Prev 2006;38:225-33.

Johnson RM, McCarthy MC, Miller SF, Peoples JB. Craniofacial trauma in injured motorcyclists: the impact of helmet usage. J Trauma 1995;38:876-8.

Kelly P, Sanson T, Strange G, Orsay E. A prospective study of the impact of helmet usage on motorcycle trauma. Ann Emerg Med 1991;20:852-6.

Keng SH. Helmet use and motorcycle fatalities in Taiwan. Accid Anal Prev 2005;37:349-55.

Kraus JF, Peek C. The impact of two related prevention strategies on head injury reduction among nonfatally injured motorcycle riders, California, 1991-1993. J Neurotrauma 1995;12:873-81.

LaTorre G, Bertazzoni G, Zotta D, van Beeck E, Ricciardi G. Epidemiology of accidents among users of two-wheeled motor vehicles: a surveillance study in two Italian cities. Eur J Public Health 2002;12:99-103.

Luna GK, Copass MK, Oreskovich MR, Carrico CJ. The role of helmets in reducing head injuries from motorcycle accidents: a political or medical issue? West J Med 1981;135:89-92.

O’Connor PJ, Kloeden C, McLean AJ. Do full-face helmets offer greater protection against cervical spinal cord injury than open-face helmets? Traffic Inj Prev 2002;3:247- 50.

Nakahara S, Chadbunchachai W, Ichikawa M, Tipsuntornsak N, Wakai S. Temporal distribution of motorcyclist injuries and risk of fatalities in relation to age, helmet use, and riding while intoxicated in Khon Kaen, Thailand. Accid Anal Prev 2005;37:833-42.

Orsay EM, Muelleman RL, Peterson TD, Jurisic DH, Kosasih JB, Levy P. Motorcycle helmets and spinal injuries: dispelling the myth. Ann Emerg Med 1994;23:802-6.

Petridou E, Skalkidou A, Ioannou N, Trichopoulos D. Fatalities from non-use of seat belts and helmets in Greece: a nationwide appraisal. Hellenic Road Traffic Police. Accid Anal Prev 1998;30:87-91.

Rowland J, Rivara F, Salzberg P, Soderberg R, Maier R, Koepsell T. Motorcycle helmet use and injury outcome and hospitalization costs from crashes in Washington State. Am J Public Health 1996;86:41-5.

Rutledge R, Stutts J. The association of helmet use with the outcome of motorcycle crash injury when controlling for crash/injury severity. Accid Anal Prev 1993;25:347-53.

Sarkar S, Peek C, Kraus JF. Fatal injuries in motorcycle riders according to helmet use. J Trauma 1995;38:242-5.

Sauter C, Zhu S, Allen S, Hargarten S, Layde PM. Increased risk of death or disability in unhelmeted Wisconsin motorcyclists. WMJ 2005;104:39-44.

Shankar BS, Ramzy AI, Soderstrom CA, Dischinger PC, Clark CC. Helmet use, patterns of injury, medical outcome, and costs among motorcycle drivers in Maryland. Accid Anal Prev 1992;24:385-96.

Shibata A, Fukuda K. Risk factors of fatality in motor vehicle traffic accidents. Accid Anal Prev 1994;26:391-7.

Weiss AA. The effects of helmet use on the severity of head injuries in motorcycle accidents. J Am Stat Assoc 1992;87:48-56.

Daly LE, Mulcahy R, Graham IM, Hickey N. Long term effect on mortality of stopping smoking after unstable angina and myocardial infarction. Br Med J (Clin Res Ed) 1983;287:324-6.

Greenwood DC, Muir KR, Packham CJ, Madeley RJ. Stress, social support, and stopping smoking after myocardial infarction in England. J Epidemiol Community Health 1995;49:583-7.

Hallstrom AP, Cobb LA, Ray R. Smoking as a risk factor for recurrence of sudden cardiac arrest. N Engl J Med. 1986 Jan 30;314(5):271-5.

Hasdai D, Garratt KN, Grill DE, Lerman A, Holmes DR Jr. Effect of smoking status on the long-term outcome after successful percutaneous coronary revascularization. N Engl J Med 1997;336:755-61.

Herlitz J, Bengtson A, Hjalmarson A, Karlson BW. Smoking habits in consecutive patients with acute myocardial infarction: prognosis in relation to other risk indicators and to whether or not they quit smoking. Cardiology 1995;86:496-502.

Perkins J, Dick TB. Smoking and myocardial infarction: secondary prevention. Postgrad Med J 1985;61:295-300.

Sato I, Nishida M, Okita K, Nishijima H, Kojima S, Matsumura N, et al. Beneficial effect of stopping smoking on future cardiac events in male smokers with previous myocardial infarction. Jpn Circ J 1992;56:217-22.

Sparrow D, Dawber TR. The influence of cigarette smoking on prognosis after a first myocardial infarction: a report from the Framingham study. J Chronic Dis 1978;31:425-32

Tofler GH, Muller JE, Stone PH, Davies G, Davis VG, Braunwald E. Comparison of long-term outcome after acute myocardial infarction in patients never graduated from high school with that in more educated patients: Multicenter Investigation of the Limitation of Infarct Size (MILIS). Am J Cardiol 1993;71:1031-5.

van Domburg RT, Meeter K, van Berkel DF, Veldkamp RF, van Herwerden LA, Bogers AJ. Smoking cessation reduces mortality after coronary artery bypass surgery: a 20-year follow-up study. J Am Coll Cardiol 2000;36:878-83.

Voors AA, van Brussel BL, Plokker HW, Ernst SM, Ernst NM, Koomen EM, Tijssen JG, Vermeulen FE. Smoking and cardiac events after venous coronary bypass surgery: a 15-year follow-up study. Circulation 1996;93:42-7.

Kawai N, Ikematsu H, Iwaki N, Satoh I, Kawashima T, Tsuchimoto T, et al. A prospective, Internet-based study of the effectiveness and safety of influenza vaccination in the 2001-2002 influenza season. Vaccine 2003;21:4507-13.

Arden NH, Patriarca PA, Fasano MB, Lui KJ, Harmon MW, Kendal AP, et al. The roles of vaccination and amantadine prophylaxis in controlling an outbreak of influenza A (H3N2) in a nursing home. Arch Intern Med 1988;148:865-8.

Arroyo JC, Postic B, Brown A, Harrison K, Birgenheier R, Dowda H. Influenza A/Philippines/2/82 outbreak in a nursing home: limitations of influenza vaccination in the aged. Am J Infect Control 1984;12:329-34.

Cartter ML, Renzullo PO, Helgerson SD, Martin SM, Jekel JF. Influenza outbreaks in nursing homes: how effective is influenza vaccine in the institutionalized elderly? Infect Control Hosp Epidemiol 1990;11:473-8.

Coles FB, Balzano GJ, Morse DL. An outbreak of influenza A (H3N2) in a well immunized nursing home population. J Am Geriatr Soc 1992;40:589-92.

Consonni S, Sandrini C, Segato E, Perucchini E, Bergamaschini L, Vergani C. Tolerability and efficacy of anti-influenza vaccination alone and associated with antipneumococcal vaccination in an elderly ambulatory population and adherence to the vaccination campaign. J Prev Med Hyg 2004;45:45-50.

D’Alessio DJ, Cox PM Jr, Dick EC. Failure of inactivated influenza vaccine to protect an aged population. JAMA 1969;210:485-9.

Hara M, Sakamoto T, Tanaka K. Effectiveness of influenza vaccination in preventing influenza-like illness among community-dwelling elderly: population-based cohort study in Japan. Vaccine 2006;24:5546-51.

Horman JT, Stetler HC, Israel E, Sorley D, Schipper MT, Joseph JM. An outbreak of influenza A in a nursing home. Am J Public Health 1986;76:501-4.

Leung JC. Effectiveness of influenza vaccination among elderly home residents in Hong Kong: a retrospective cohort study. Hong Kong Pract 2007;29:123-33.

Meiklejohn G, Hall H. Unusual outbreak of influenza A in a Wyoming nursing home. J Am Geriatr Soc 1987;35:742-6.

Monto AS, Hornbuckle K, Ohmit SE. Influenza vaccine effectiveness among elderly nursing home residents: a cohort study. Am J Epidemiol 2001;154:155-60.

Morens DM, Rash VM. Lessons from a nursing home outbreak of influenza A. Infect Control Hosp Epidemiol 1995;16:275-80.

Mukerjee A. Spread of influenza: a study of risk factors in homes for the elderly in Wales. J Epidemiol Community Health 1994;48:602-3.

Patriarca PA, Weber JA, Parker RA, Hall WN, Kendal AP, Bregman DJ, et al. Efficacy of influenza vaccine in nursing homes: reduction in illness and complications during an influenza A (H3N2) epidemic. JAMA 1985;253:1136-9.

Ruben FL, Johnston F, Streiff EJ. Influenza in a partially immunized aged population. Effectiveness of killed Hong Kong vaccine against infection with the England strain. JAMA 1974;230:863-6.

Saito R, Suzuki H, Oshitani H, Sakai T, Seki N, Tanabe N. The effectiveness of influenza vaccine against influenza a (H3N2) virus infections in nursing homes in Niigata, Japan, during the 1998-1999 and 1999-2000 seasons. Infect Control Hosp Epidemiol 2002;23:82-6.

Strassburg MA, Greenland S, Sorvillo FJ, Lieb LE, Habel LA. Influenza in the elderly: report of an outbreak and a review of vaccine effectiveness reports. Vaccine 1986;4:38-44

Taylor JL, Dwyer DM, Coffman T, Groves C, Patel J, Israel E. Nursing home outbreak of influenza A (H3N2): evaluation of vaccine efficacy and influenza case definitions. Infect Control Hosp Epidemiol 1992;13:93-7.

Bookbinder M, Blank AE, Arney E, Wollner D, Lesage P, McHugh M, et al. Improving end-of-life care: development and pilot-test of a clinical pathway. J Pain Symptom Manage 2005;29:529-43.

Brattebo G, Hofoss D, Flaatten H, Muri AK, Gjerde S, Plsek PE. Effect of a scoring system and protocol for sedation on duration of patients’ need for ventilator support in a surgical intensive care unit. BMJ 2002;324:1386-9.

Chadha Y, Mollison J, Howie F, Grimshaw J, Hall M, Russell I. Guidelines in gynaecology: evaluation in menorrhagia and in urinary incontinence. BJOG 2000;107:535-43.

Macartney KK, Gorelick MH, Manning ML, Hodinka RL, Bell LM. Nosocomial respiratory syncytial virus infections: the cost-effectiveness and cost-benefit of infection control. Pediatrics 2000;106:520-6.

Pang X, Zhu Z, Xu F, Guo J, Gong X, Liu D, et al. Evaluation of control measures implemented in the severe acute respiratory syndrome outbreak in Beijing, 2003. JAMA 2003;290:3215-21.

Ryan MA, Christian RS, Wohlrabe J. Handwashing and respiratory illness among young adults in military training. Am J Prev Med 2001;21:79-83.

Simon A, Khurana K, Wilkesmann A, Muller A, Engelhart S, Exner M, et al. Nosocomial respiratory syncytial virus infection: impact of prospective surveillance and targeted infection control. Int J Hyg Environ Health 2006;209:317-24.

Krasinski K, LaCouture R, Holzman RS, Waithe E, Bonk S, Hanna B. Screening for respiratory syncytial virus and assignment to a cohort at admission to reduce nosocomial transmission. J Pediatr 1990;116:894-8.

Pelke S, Ching D, Easa D, Melish ME. Gowning does not affect colonization or infection rates in a neonatal intensive care unit. Arch Pediatr Adolesc Med 1994;148:1016-20.

Heymann A, Chodick G, Reichman B, Kokia E, Laufer J. Influence of school closure on the incidence of viral respiratory diseases among children and on health care utilization. Pediatr Infect Dis J 2004;23:675-7.

Krilov LR, Barone SR, Mandel FS, Cusack TM, Gaber DJ, Rubino JR. Impact of an infection control program in a specialized preschool. Am J Infect Control 1996;24:167-73.

Moulopoulos S, Stamatelopoulos S, Petrou P. Intraaortic balloon assistance in intractable cardiogenic shock. Eur Heart J 1986;7:396-403.

Waksman R, Weiss AT, Gotsman MS, Hasin Y. Intra-aortic balloon counterpulsation improves survival in cardiogenic shock complicating acute myocardial infarction. Eur Heart J 1993;14:71-4.

Anderson RD, Ohman EM, Holmes DR Jr, Col I, Stebbins AL, Bates ER, et al. Use of intraaortic balloon counterpulsation in patients presenting with cardiogenic shock: observations from the GUSTO-I Study: Global Utilization of Streptokinase and TPA for Occluded Coronary Arteries. J Am Coll Cardiol 1997;30:708-15.

Sanborn TA, Sleeper LA, Bates ER, Jacobs AK, Boland J, French JK, et al. Impact of thrombolysis, intra-aortic balloon pump counterpulsation, and their combination in cardiogenic shock complicating acute myocardial infarction: a report from the SHOCK Trial Registry. SHould we emergently revascularize Occluded Coronaries for cardiogenic shocK? J Am Coll Cardiol 2000;36(3 Suppl A):1123-9.

Barron HV, Every NR, Parsons LS, Angeja B, Goldberg RJ, Gore JM, et al. The use of intra-aortic balloon counterpulsation in patients with cardiogenic shock complicating acute myocardial infarction: data from the National Registry of Myocardial Infarction 2. Am Heart J 2001;141:933-9.

Vis MM, Sjauw KD, van der Schaaf RJ, Baan J Jr, Koch KT, DeVries JH, et al. In patients with ST-segment elevation myocardial infarction with cardiogenic shock treated with percutaneous coronary intervention, admission glucose level is a strong independent predictor for 1-year mortality in patients without a prior diagnosis of diabetes. Am Heart J 2007;154:1184-90.

Aberg A, Bergstrand R, Johansson S, Ulvenstam G, Vedin A, Wedel H, et al. Cessation of smoking after myocardial infarction: effects on mortality after 10 years. Br Heart J 1983;49:416-22.

Hedback B, Perk J, Wodlin P. Long-term reduction of cardiac mortality after myocardial infarction: 10-year results of a comprehensive rehabilitation programme. Eur Heart J 1993;14:831-5.

Johansson S, Bergstrand R, Pennert K, Ulvenstam G, Vedin A, Wedel H, et al. Cessation of smoking after myocardial infarction in women: effects on mortality and reinfarctions. Am J Epidemiol 1985;121:823-31.

Salonen JT. Stopping smoking and long-term mortality after acute myocardial infarction. Br Heart J 1980;43:463-9.

Vlietstra RE, Kronmal RA, Oberman A, Frye RL, Killip T 3rd. Effect of cigarette smoking on survival of patients with angiographically documented coronary artery disease: report from the CASS registry. JAMA 1986;255:1023-7.

Jianping H, Xin F, Changshun L, Bo Z, Linxiu G, Wei X, et al. Assessment of effectiveness of Vaxigrip. Vaccine 1999;17 Suppl 1:S57-8.

Fujieda M, Maeda A, Kondo K, Kaji M, Hirota Y. Inactivated influenza vaccine effectiveness in children under 6 years of age during the 2002-2003 season. Vaccine 2006;24:957-63.

Appendix 2.

User guidance for the revised version of Risk of Bias Assessment tool for Nonrandomized Studies

1. Comparability of the target group
Selection bias due to the selection of inappropriate comparison target group
Criteria for ‘low’ risk of bias	Cohort study
		Exposure group and comparison group for interventions are comparable population groups that don’t differ regarding adaptation syndrome and the severity of the disease.
	Cross sectional study
		The two groups to be compared are comparable population groups since they do not differ regarding adaptation syndrome and the severity of the disease.
	Case-control study
		The patient group and comparison group are comparable population groups since they don’t have differences regarding the possibility of exposure to interventions.
	Before-after study
		The population group is the same for both before and after the exposure to interventions.
Criteria for 'high’ risk of bias	Cohort study
		The exposure group and comparison group for interventions are not comparable population groups since they differ regarding adaptation syndrome and the severity of the disease.
	Cross sectional study
		The two groups to be compared are not comparable population groups since they differ regarding adaptation syndrome or the severity of the disease.
	Case-control study
		The patient group and comparison group are not comparable population groups since they have differences regarding the possibility of exposure to interventions.
	Before-after study
		The population groups are different for before and after the exposure to interventions.
Criteria for ‘unclear’ risk of bias	If it is unclear whether the risk of bias belongs to ‘low’ or ‘high’ regarding the possibility of the target group comparisons.

2. Target group selection
Selection bias due to inappropriate intervention or inappropriate selection of exposure group or patient group
Criteria for ‘low’ risk of bias	Target group selection for each study design has been met by two of the below criteria.
	Cohort study
		(1) A confirmed absence of outcomes from study participants at the point of enrollment for the study.
		(2) Participant recruitment strategy (standard of inclusion/exclusion, selection method) was the same for all the target groups.
	Cross sectional study
		(1) It was confirmed that the participant selection was not influenced by the outcome occurrence at the point of enrollment for the study.
		(2) The participant recruitment strategy (standard of inclusion/exclusion, selection method) was the same for all the target groups.
	Case-control study
		(1) Confirmed absence of disease for the control group.
		(2) The sample was collected from the general population group.
	Before-after study
		(1) Target group was recruited consecutively.
		(2) The data was collected prospectively.
Criteria for ‘high’ risk of bias	If one or more of the following criteria are met.
	Cohort study
		(1) An unconfirmed absence of outcomes from study participants at the point of enrollment for the study.
		(2) Participant recruit strategy (standard of inclusion/exclusion, selection method) was not the same as the target groups.
	Cross sectional study
		(1) It was not confirmed if the participant selection was not influenced by the outcome occurrence (for study participants) at the point of enrollment for the study.
		(2) The participant recruit strategy (standard of inclusion/exclusion, selection method) differs for each target group.
	Case-control study
		(1) The sample was not collected from the general population group.
		(2) An unconfirmed absence of the disease in the control group.
	Before-after study
		(1) The target group was not recruited consecutively.
		(2) The data was collected retrospectively.
Criteria for ‘unclear’ risk of bias	If it is unclear whether the risk of bias belongs to ‘low’ or ‘high’ regarding the target group selection.

3. Confounders
Selection bias due to inappropriate confounder confirmation and consideration
Criteria for ‘low’ risk of bias	If one or more of below criteria were met.
	Non-randomized study other than before-after study
		(1) Main confounders were confirmed and considered properly during the planning stage (matching, restriction on participation).
		(2) Main confounders were confirmed and properly revised during the analysis stage (stratification, propensity score) and the statistical revision stage (regression analysis, etc.).
	Before-after study
		(1) Due to the characteristics of a disease or interventions, before and after differences according to the time elapsed (natural course) can be excluded.
		(2) Exclusion was not possible but the data were revised during the analysis stage (regression analysis, etc.).
Criteria for ‘high’ risk of bias	If one or more of below criteria were met.
	Non-randomized study other than before-after study
		(1) Main confounders were not handled during the stages of planning or analysis.
		(2) Main confounders were confirmed, but not considered properly during the stages of design or analysis.
	Before-after study
		(1) Considering the characteristics of a disease or interventions, before and after differences according to the time elapsed (natural course) cannot be excluded from influencing main outcomes and they were not considered during the stage of analysis.
Criteria for ‘unclear’ risk of bias	If it is uncertain whether the risk of bias belongs to ‘low’ or ‘high’ regarding the confounder.

4. Measurement of intervention/exposure
Performance bias due to inappropriate intervention or inappropriate exposure measurement
Criteria for ‘low’ risk of bias	If both of the following two criteria are met.
		(1) Confirmed from a trustworthy source such as medical records or structured interviews.
		(2) Measurements were objectified and standardized properly by utilizing multiple measurements (2 times or more), independent measurements by multiple investigators, or using a standardized measurement of exposure.
Criteria for ‘high’ risk of bias	If one or more of the following criteria are met.
		(1) Exposures were measured by simple self-response.
		(2) Exposures were measured by unstructured interviews.
		(3) Recall bias is relatively clear.
		(4) Measurements were not objectified or standardized properly by utilizing multiple measurements (2 times or more), independent measurements by multiple investigators, or using a standardized measurement of exposure, even with no effort to do so.
Criteria for ‘unclear’ risk of bias	If it is unclear whether risk of bias belongs to ‘low’ or ‘high’ regarding the exposure measurement.

5. Blinding of assessors
Detection bias due to inappropriate blinding of assessors
Criteria for ‘low’ risk of bias	If one or more of the following criteria are met.
	Non-randomized study other than case-control study
		(1) The blinding of outcome assessors was properly done and the blinding is judged to be unbreakable.
		(2) No blinding was done for the outcome assessors, but the fact that blinding does not exist is not judged as affecting outcomes.
	Case-control study
		(1) The blinding of exposure assessors was properly done and judged as unbreakable.
		(2) No blinding was done for exposure assessors, but the fact that blinding does not exist is not judged as affecting outcomes.
Criteria for ‘high’ risk of bias	If one or more of the following criteria are met.
	Non-randomized study other than case-control study
		(1) Blinding was not done for outcome assessors.
		(2) The blinding was done for outcome assessors, but it is uncertain whether the blinding is intact. The blinding is judged as affecting outcome measurement.
	Case-control study
		(1) Blinding was not done for exposure assessors.
		(2) The blinding was done for exposure assessors, but it is uncertain whether the blinding is intact. The blinding is judged as affecting outcome measurement.
Criteria for ‘unclear’ risk of bias	If it is unclear whether the risk of bias belongs to ‘low’ or ‘high’ regarding the blinding of assessors.

6. Outcome assessment
Detection bias due to inappropriate outcome assessment methods
Criteria for ‘low’ risk of bias	If one or more of the following criteria are met.
		(1) Patient-reported outcomes were assessed using tools that have proven reliability and validity.
		(2) If outcomes were collected using an equipment-based measurement method, such as test results or blood pressure checks, the accuracy certification of the measuring equipment was implemented.
		(3) Outcomes such as death or disease were confirmed with either medical records or reliable data sources.
		(4) The outcome assessment is judged as being handled in a trustworthy manner using tools with proven reliability and validity or utilizing an objective measuring method.
Criteria for ‘high’ risk of bias	If one or more of the following criteria are met.
		(1) Patient-reported outcome was assessed using only simple self-response.
		(2) When outcomes were collected using an equipment-based measurement method, such as test results or blood pressure checks, the accuracy certification of the measuring equipment was not implemented.
		(3) Outcomes such as death or disease were not confirmed with either medical records or reliable data sources.
		(4) The outcome assessment is judged as being handled in a non-trustworthy manner, the tools do not have proven reliability or validity, and the measuring method was not objective.
Criteria for ‘unclear’ risk of bias	If it is unclear whether risk of bias belongs to ‘low’ or ‘high’ regarding the outcome assessment.

7. Incomplete outcome data
Attrition bias due to inappropriate handling of incomplete data
Criteria for ‘low’ risk of bias	If one or more of the following criteria are met.
	Non-randomized study other than before-after study
		(1) No missing data.
		(2) The reason for missing data is judged to not affect outcomes.
		(3) The missing data occurred similarly between the intervention exposure group and the control group. The reasons given for the missing data are similar.
	Before-after study
		(1) Dropouts and those who have completed the study had no difference in the baselines.
Criteria for ‘high’ risk of bias	If one or more of the following criteria are met.
	Non-randomized study other than before-after study
		(1) Due to the differences in ratio or the reason for incomplete data between two groups, the reason for missing data is judged as affecting outcomes.
	Before-after study
		(1) Dropouts and those who have completed the study had a difference in the baselines.
Criteria for ‘unclear’ risk of bias	If it is unclear whether the risk of bias belongs to ‘low’ or ‘high’ regarding the incomplete outcome data.

8. Selective outcome reporting
Reporting bias due to selective outcome reporting
Criteria for ‘low’ risk of bias	If one or more of the following criteria are met.
		(1) The protocol that previously determined primary and secondary outcomes was described as planned.
		(2) Although there was no protocol, most of the expected main outcomes were included.
Criteria for ‘high’ risk of bias	If one or more of the following criteria are met.
		(1) Some of the previously determined primary and secondary outcomes were not reported.
		(2) Reporting was done using a method that was not previously determined.
		(3) The outcomes that were not previously determined were reported (exception: when a clear explanation for reporting is provided).
		(4) The expected main outcomes for the respective study were not reported.
Criteria for ‘unclear’ risk of bias	If it is unclear whether the risk of bias belongs to ‘low’ or ‘high’ regarding the selective outcome reporting.

Appendix 3.

Comparison of the risk of bias domain and study designs covered by RoBANS, RoBINS-I, and RoBANS 2

Bias type	RoBANS: Kim et al. [7] (2013)	ROBINS-I: Sterne et al. [16] (2016)	RoBANS 2
Selection	- Selection of participants	- Bias in selection of participants into the study	- Comparability of the target group
	- Confounding variables	- Bias due to confounding	- Target group selection
	- Confounding variables	- Bias due to confounding	- Confounders
Performance	- Measurement of exposure	- Bias due to deviations from intended interventions	- Measurement of intervention/exposure
Performance	- Measurement of exposure	- Bias in classification of interventions	- Measurement of intervention/exposure
Detection	- Blinding of outcome assessments	- Bias in measurement of outcomes	- Blinding of assessors
Detection	- Blinding of outcome assessments	- Bias in measurement of outcomes	- Outcome assessment
Attrition	- Incomplete outcome data	- Bias due to missing data	- Incomplete outcome data
Reporting	-Selectiveoutcomereporting	- Bias in selection of the reported result	-Selectiveoutcomereporting
Study designs covered by the tool	- Nonrandomized trials	- Cohort-like designs, such as cohort studies, quasi-randomized trials, and other concurrently controlled studies	- Cohort study
	- Cohort study		- Case-control study
	- Case-control study		- Cross-sectional study
	- Before and after study		- Before-after study

RoBANS, Risk of Bias Assessment tool for Nonrandomized Studies; ROBINS-I, Risk of Bias in Non-randomized Studies of Interventions.