Our main objective is to assess the inter-reviewer reliability (IRR) reported in published systematic literature reviews (SLRs). Our secondary objective is to determine the expected IRR by authors of SLRs for both human and machine-assisted reviews.

We performed a review of SLRs of randomised controlled trials using the PubMed and Embase databases. Data were extracted on IRR by means of Cohen’s kappa score of abstract/title screening, full-text screening and data extraction in combination with review team size, items screened and the quality of the review was assessed with the A MeaSurement Tool to Assess systematic Reviews 2. In addition, we performed a survey of authors of SLRs on their expectations of machine learning automation and human performed IRR in SLRs.

After removal of duplicates, 836 articles were screened for abstract, and 413 were screened full text. In total, 45 eligible articles were included. The average Cohen’s kappa score reported was 0.82 (SD=0.11, n=12) for abstract screening, 0.77 (SD=0.18, n=14) for full-text screening, 0.86 (SD=0.07, n=15) for the whole screening process and 0.88 (SD=0.08, n=16) for data extraction. No association was observed between the IRR reported and review team size, items screened and quality of the SLR. The survey (n=37) showed overlapping expected Cohen’s kappa values ranging between approximately 0.6–0.9 for either human or machine learning-assisted SLRs. No trend was observed between reviewer experience and expected IRR. Authors expect a higher-than-average IRR for machine learning-assisted SLR compared with human based SLR in both screening and data extraction.

Currently, it is not common to report on IRR in the scientific literature for either human and machine learning-assisted SLRs. This mixed-methods review gives first guidance on the human IRR benchmark, which could be used as a minimal threshold for IRR in machine learning-assisted SLRs.

CRD42023386706.

Integrated care is seen as an enabling strategy in organising healthcare to improve quality, finances, personnel and sustainability. Developments in the organisation of maternity care follow this trend. The way care is organised should support the general aims and outcomes of healthcare systems. Organisation itself consists of a variety of smaller ‘elements of organisation’. Various elements of organisation are implemented in different organisations and networks. We will examine which elements of integrated maternity care are associated with maternal and neonatal health outcomes, experiences of women and professionals, healthcare spending and care processes.

We will conduct this review using the JBI methodology for scoping reviews and the reporting guideline PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews). We will undertake a systematic search in the databases PubMed, Scopus, Cochrane and PsycINFO. A machine learning tool, ASReview, will be used to select relevant papers. These papers will be analysed and classified thematically using the framework of the Rainbow Model of Integrated Care (RMIC). The Population Concept Context framework for scoping reviews will be used in which ‘Population’ is defined as elements of the organisation of integrated maternity care, ‘Context’ as high-income countries and ‘Concepts’ as outcomes stated in the objective of this review. We will include papers from 2012 onwards, in Dutch or English language, which describe both ‘how the care is organised’ (elements) and ‘outcomes’.

Since this is a scoping review of previously published summary data, ethical approval for this study is not needed. Findings will be published in a peer-reviewed international journal, discussed in a webinar and presented at (inter)national conferences and meetings of professional associations.

The findings of this scoping review will give insight into the nature and effectiveness of elements of integrated care and will generate hypotheses for further research.