FreshRSS

🔒
❌ Acerca de FreshRSS
Hay nuevos artículos disponibles. Pincha para refrescar la página.
AnteayerTus fuentes RSS

Inter-reviewer reliability of human literature reviewing and implications for the introduction of machine-assisted systematic reviews: a mixed-methods review

Por: Hanegraaf · P. · Wondimu · A. · Mosselman · J. J. · de Jong · R. · Abogunrin · S. · Queiros · L. · Lane · M. · Postma · M. J. · Boersma · C. · van der Schans · J.
Objectives

Our main objective is to assess the inter-reviewer reliability (IRR) reported in published systematic literature reviews (SLRs). Our secondary objective is to determine the expected IRR by authors of SLRs for both human and machine-assisted reviews.

Methods

We performed a review of SLRs of randomised controlled trials using the PubMed and Embase databases. Data were extracted on IRR by means of Cohen’s kappa score of abstract/title screening, full-text screening and data extraction in combination with review team size, items screened and the quality of the review was assessed with the A MeaSurement Tool to Assess systematic Reviews 2. In addition, we performed a survey of authors of SLRs on their expectations of machine learning automation and human performed IRR in SLRs.

Results

After removal of duplicates, 836 articles were screened for abstract, and 413 were screened full text. In total, 45 eligible articles were included. The average Cohen’s kappa score reported was 0.82 (SD=0.11, n=12) for abstract screening, 0.77 (SD=0.18, n=14) for full-text screening, 0.86 (SD=0.07, n=15) for the whole screening process and 0.88 (SD=0.08, n=16) for data extraction. No association was observed between the IRR reported and review team size, items screened and quality of the SLR. The survey (n=37) showed overlapping expected Cohen’s kappa values ranging between approximately 0.6–0.9 for either human or machine learning-assisted SLRs. No trend was observed between reviewer experience and expected IRR. Authors expect a higher-than-average IRR for machine learning-assisted SLR compared with human based SLR in both screening and data extraction.

Conclusion

Currently, it is not common to report on IRR in the scientific literature for either human and machine learning-assisted SLRs. This mixed-methods review gives first guidance on the human IRR benchmark, which could be used as a minimal threshold for IRR in machine learning-assisted SLRs.

PROSPERO registration number

CRD42023386706.

Impaired health-related quality of life due to elevated risk of developing diabetes: A cross-sectional study in Indonesia

by M. Rifqi Rokhman, Bustanul Arifin, Benedetta Broggi, Anne-Fleur Verhaar, Zulkarnain Zulkarnain, Satibi Satibi, Dyah Aryani Perwitasari, Cornelis Boersma, Qi Cao, Maarten J. Postma, Jurjen van der Schans

Background

This study investigated the association between elevated risk of developing diabetes and impaired health-related quality of life (HRQoL) in the Indonesian population.

Methods

A cross-sectional study was conducted on 1,336 Indonesians from the general population who had no previous diagnosis of diabetes. Utility score to represent HRQoL was measured using the EuroQol 5-dimension, while the risk for developing diabetes was determined using the Finnish Diabetes Risk Score (FINDRISC) instrument. All participants underwent a blood glucose test after fasting for 8 hours. The association between FINDRISC score and HRQoL adjusted for covariates was analysed using multivariate Tobit regression models. Minimal clinically important differences were used to facilitate interpretation of minimal changes in utility score that could be observed.

Results

The median (interquartile range) of the overall FINDRISC score was 6 (7), while the mean (95% confidence intervals) of the EQ-5D utility score was 0.93 (0.93–0.94). Once adjusted for clinical parameters and socio-demographic characteristics, participants with a higher FINDRISC score showed a significantly lower HRQoL. No significant association was detected between fasting blood glucose level categories and HRQoL. A difference of 4–5 points in the FINDRISC score was considered to reflect meaningful change in HRQoL in clinical practice.

Conclusion

An elevated risk of developing diabetes is associated with a lower HRQoL. Therefore, attention should be paid not only to patients who have already been diagnosed with diabetes, but also to members of the general population who demonstrate an elevated risk of developing diabetes. This approach will assist in preventing the onset of diabetes and any further deterioration of HRQoL in this segment of the Indonesian population.

❌