Background: Integrating algorithm-based clinical decision support (CDS) systems poses significant challenges in evaluating their actual clinical value. Such CDS systems are traditionally assessed via controlled but resource-intensive clinical trials. Objective: This paper presents a review protocol for preimplementation in silico evaluation methods to enable broadened impact analysis under simulated environments before clinical trials. Methods: We propose a scoping review protocol that follows an enhanced Arksey and O’Malley framework and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines to investigate the scope and research gaps in the in silico evaluation of algorithm-based CDS models—specifically CDS decision-making end points and objectives, evaluation metrics used, and simulation paradigms used to assess potential impacts. The databases searched are PubMed, Embase, CINAHL, PsycINFO, Cochrane, IEEEXplore, Web of Science, and arXiv. A 2-stage screening process identified pertinent articles. The information extracted from articles was iteratively refined. The review will use thematic, trend, and descriptive analyses to meet scoping aims. Results: We conducted an automated search of the databases above in May 2023, with most title and abstract screenings completed by November 2023 and full-text screening extended from December 2023 to May 2024. Concurrent charting and full-text analysis were carried out, with the final analysis and manuscript preparation set for completion in July 2024. Publication of the review results is targeted from July 2024 to February 2025. As of April 2024, a total of 21 articles have been selected following a 2-stage screening process; these will proceed to data extraction and analysis. Conclusions: We refined our data extraction strategy through a collaborative, multidisciplinary approach, planning to analyze results using thematic analyses to identify approaches to in silico evaluation. Anticipated findings aim to contribute to developing a unified in silico evaluation framework adaptable to various clinical workflows, detailing clinical decision-making characteristics, impact measures, and reusability of methods. The study’s findings will be published and presented in forums combining artificial intelligence and machine learning, clinical decision-making, and health technology impact analysis. Ultimately, we aim to bridge the development-deployment gap through in silico evaluation-based potential impact assessments.
Background: evaluation (ISE) methods create a digital twin or a computer simulation of actual care pathways, enabling a broader assessment of the potential impact of algorithm-based clinical decision support systems (CDSS) before implementation. A programmatic search of several academic research databases showed at least 886 CDSS development and evaluation studies in the past 3 decades. However, fewer than 3% applied ISE to evaluate the potential impact on broader clinical care pathways. Objective: This study aims to review the scope of proposed ISE methods to evaluate CDSS, with a focus on simulation modeling approaches, care pathway parameters considered, and outcomes evaluated within the ISE methodological domain. Methods: This review followed the established scoping review methodological guidelines. We conceptualized a tailored search framework and conducted a 2-stage screening process on studies identified through automated searches of selected databases. Relevant information on CDSS study characteristics and the application of care pathway simulation modeling in CDSS evaluation was subsequently extracted. Results: A small subset of studies on CDSS development conducted ISE. Most ISE studies were published after 2019, reflecting a more recent increase in the application of ISE. These studies frequently emphasized patient, process, and cost-effectiveness outcomes. Notably, the evaluation of outcomes directly related to care providers’ well-being is lacking, highlighting a critical gap in current ISE applications. Among the studies included in this review, various simulation modeling paradigms were used, including dynamic simulations and state-based models. Three themes were found among the different motivations and objectives for using ISE: (1) outcome comparison, (2) outcome comparison with sensitivity analysis, and (3) simulation-based optimization of the proposed CDSS. The first two approaches considered a decoupled CDSS model training followed by simulation-based evaluation of the trained CDSS model. The third approach iteratively improved the CDSS model’s decision-support capabilities through optimization based on care pathway simulations. These approaches can be broadly categorized as (1 and 2), where CDSS models are evaluated within seperately designed clinical workflow simulations, and (3), where simulation iteratively optimizes CDSS performance. Conclusions: The growing body of algorithm-based CDSS research underscores the need for evaluation approaches that are resource-efficient and account for systems-level workflow implications. This review highlights both the gaps and the potential of ISE, particularly care pathway simulation-based approaches, as a preimplementation strategy to strengthen evidence before significant resource allocations to pilot or full-scale implementations. ISE presents as a promising intermediary evaluation approach that bridges model-level performance and clinical workflow impact, allowing more contextualized and resource-efficient assessments prior to implementation. International Registered Report Identifier (IRRID): RR2-10.2196/63875