ABSTRACT

The operating room is evolving into an information-rich environment where surgical care is increasingly shaped by integrated digital architectures. The proliferation of minimally invasive and robotic platforms has generated a massive influx of high-resolution video, shifting the field toward real-time computational analysis. Artificial intelligence, particularly computer vision, is moving beyond retrospective analysis to extract structured intelligence from the operative field, redefining how procedures are analyzed and supported. This review focuses on the clinical applications of computer vision with the greatest translational relevance: Computer-aided detection, instrument tracking, procedural phase recognition, and video-based skill assessment. However, clinical translation remains constrained by practice variability, a lack of standardized open datasets, ethical and regulatory considerations, and the difficulty of ensuring model generalizability across diverse institutions. This review examines the current evidence for computer-aided detection, instrument tracking, phase recognition, and video-based skill assessment in surgery, and discusses their clinical readiness and key limitations.

INTRODUCTION

More than 300 million major and minor surgical procedures are performed worldwide each year (1). During the perioperative period, over one-third of patients experience complications such as bleeding, organ injury, and infection, posing substantial challenges for patients, healthcare institutions, and health systems globally. Approximately half of these complications are classified as major, and many are considered potentially preventable, underscoring the urgent need for innovations that enhance procedural safety and surgical quality (2).

The widespread adoption of minimally invasive and robotic techniques has fundamentally transformed modern surgical practice. Compared with open surgery, minimally invasive approaches are associated with faster recovery, reduced postoperative pain, shorter hospital stays, and lower complication rates, with outcomes improving further with increasing surgeon and institutional experience (3-5). These realities contribute to variability in surgical performance and highlight the need for scalable systems capable of supporting decision-making, technical precision, and intraoperative situational awareness.

Concurrently, the operating room is undergoing a profound digital transformation. Fiber-optic cameras, robotic platforms, and advanced imaging systems now generate unprecedented volumes of high-resolution intraoperative video (6). Beyond serving as visualization tools, these technologies function as continuous data sources, enabling computational analysis of surgical activity (7). Artificial intelligence (AI), particularly through computer vision (CV) techniques capable of extracting structured information from visual data, offers new ways to analyze, quantify, and support surgical performance.

This review focuses on the four areas with the most direct evidence of clinical translation: Computer-aided detection, instrument tracking, phase recognition, and video-based skill assessment. While many of these applications have initially focused on retrospective analysis, they increasingly point toward real-time intraoperative assistance, workflow optimization, and context-aware decision support.

CLINICAL AND RESEARCH CONSEQUENCES

CV Applications in the General Surgery Domain

Currently applicable technologies

While the current surgical landscape features only a limited selection of commercially available and clinically validated applications, the high degree of technological promise and robust ongoing research suggest that an imminent expansion of regulatory approvals is on the horizon. Currently, the most fundamental and extensively validated application of this technology is computer-aided detection in endoscopy [exemplified by platforms such as GI Genius™ (Medtronic, USA), CAD EYE (Fujifilm, Japan), and Endo-AID (Olympus, Japan)] which has been demonstrated in multiple randomized controlled trials to significantly increase the detection rate of colorectal neoplasia (8). Another fundamental application is estimating blood loss through image processing (previously known as Triton, currently part of SurgiCount+, Stryker, USA), a method shown to be more accurate than visual estimation (9).

There is an upsurge of regulatory approvals spearheaded by robotic companies transitioning from recent clearance to imminent clinical deployment, most notably the da Vinci 5 system (Intuitive Surgical Inc, USA), which recently received U.S. FDA 510(k) clearance for systems such as a computational observer, which analyzes intraoperative data to provide surgeons with feedback on their technique and procedural efficiency (“Case Insights”, K251739); and tool for analysis and reconstruction of DICOM images into high-fidelity 3D models for preoperative planning and intraoperative display for rectum, prostate, and kidney (“IRISeg”, K251763). This trajectory is further bolstered by a growing cohort of manufacturers developing proprietary intraoperative analytics and digital surgery ecosystems [Touch Surgery (Medtronic, USA, K250725), Intelligent Surgical Unit (Asensus, USA, K212450)].

Complementing commercial systems, significant research efforts have established the scientific proof-of-concept for advanced analytics across these primary categories:

Instrument detection and tracking: Modern technical approaches to surgical instrument detection and tracking primarily leverage CV and deep learning to process high-resolution frames from robotic endoscopes. These systems utilize convolutional neural networks (CNNs) or vision transformers to perform real-time semantic segmentation, where each pixel is classified to distinguish the instrument from complex backgrounds like blood, tissue, and smoke. The technical pipeline typically involves extracting spatial coordinates and orientation vectors (pose estimation) to map the instrument’s kinematic chain within a 3D coordinate system. By analyzing the temporal consistency across video frames, these algorithms generate precise motion trajectories that serve as raw data for recurrent neural networks or long short-term memory units, which evaluate the “smoothness” and efficiency of the surgeon’s movements. This automated labeling of the “action space” eliminates the need for manual frame-by-frame annotation, facilitating the creation of large-scale datasets for objective skill benchmarking. Validated examples report high performance yielding ~70-90% detection accuracy on public tool-localization datasets (M2CAI 2016) (10).

Phase recognition for workflow analysis: Surgical phase recognition is essential because it transforms raw video into context-aware data, enabling the operating room to function as an intelligent, synchronized environment. By identifying the specific step of a procedure in real time, these systems can automate administrative tasks like operative note generation, optimize hospital logistics by predicting the exact time until the next case, and provide intraoperative “co-piloting” to enhance patient safety. A recent systemic review analyzed 35 studies and found that AI models can achieve recognition accuracy rates often exceeding 90% (Cholec80-gold standard for surgical workflow analysis in laparoscopic cholecystectomy; EndoVis, Heidi & GTEA) (11).

Video-based skill assessment: AI has emerged as a robust tool for the objective quantification of surgical technical skill, utilizing deep learning architectures to analyze intraoperative video data. These models are typically validated against gold-standard clinical metrics, including the objective structured assessment of technical Skills, the global operative assessment of laparoscopic skills, and fundamentals of laparoscopic surgery scores. Recent advancements employ 2D and 3D CNNs, often integrated with attention mechanisms, to evaluate either full-length procedures or specific task snippets. The performance of these systems consistently aligns with expert human judgment. For instance, the VBA-Net architecture demonstrated 97.5% accuracy in binary pass/fail classification for laparoscopic pattern-cutting tasks, achieving high Spearman rank correlations with human observers (12). By meeting or exceeding the inter-rater reliability of expert surgeons, these automated systems offer a scalable, unbiased alternative for surgical residency training and longitudinal competency tracking.

AI based analytics for education quality improvement: Surgical education has traditionally relied on apprenticeship-based models with limited objective performance metrics and structured feedback mechanisms. However, the modern operating room represents a data-rich environment that offers significant potential for personalized, AI-assisted surgical training (13). In a randomized study involving brain tumor surgery simulation, AI-based coaching systems significantly improved the learning outcomes and performance of medical students (14). Beyond training, AI algorithms have also demonstrated the ability to objectively evaluate surgical skill from laparoscopic colorectal surgery videos using metrics comparable to established assessment methods (15). Combining automated analysis of surgical videos with expert human annotations may further enable fair and reliable feedback across surgeons with varying levels of experience (16, 17). The summary of applications are provided in Table 1.

Emerging and Future Applications

Emerging and future applications of surgical AI represent a paradigm shift toward proactive, context-aware assistance and the eventual realization of the intelligent operating room ecosystem (Figure 1). These technologies, currently at lower technology readiness levels, move beyond simple detection toward a “human-in-the-loop” partnership where digital systems actively anticipate surgical needs. A critical frontier is real-time anatomical recognition, which serves as the foundation for intraoperative alerts, such as identifying the critical view of safety during cholecystectomy or flagging high-risk zones near vascular landmarks (18). These context-aware decision support systems aim to reduce cognitive load by providing timely guidance tailored to the specific phase of the procedure. Furthermore, the integration with robotic platforms is paving the way for autonomous or semi-autonomous surgical actions, where AI handles repetitive tasks (such as suturing or suctioning) under human supervision, thereby enhancing precision and reducing fatigue. To streamline clinical workflows, automated reporting and documentation tools are being developed to synthesize intraoperative events into structured medical records, potentially reclaiming significant administrative time for surgical teams (19).

The future of surgical excellence lies in multimodal fusion, where high-resolution video is integrated with patient-specific genomics, imaging, and electronic health record data. This holistic data stream enables predictive intraoperative risk modeling, allowing systems to forecast complications before they manifest clinically (20). Central to this vision is the surgical digital twin, a virtual counterpart that mirrors the patient’s live physiology and anatomy. These twins allow for patient-specific simulation, where surgeons can rehearse complex maneuvers on a digital replica or test interventions in a virtual space before executing them in the physical OR (21). By unifying these disparate technologies into a cohesive ecosystem, the “intelligent OR” transitions from a reactive site of intervention to a proactive, networked environment defined by shared intelligence, global equity, and continuous learning (Figure 2).

Key Challenges in the Clinical Integration of CV in Surgery

Despite significant technical progress, important ethical, cultural, and educational challenges must be addressed before CV systems can be widely implemented in surgical care. Beyond technical validation and operating room integration, adoption requires collaboration among clinicians, regulators, and technology developers. Key concerns include data privacy, security, transparency, and fairness in algorithm development. Questions surrounding informed consent, data ownership, and patient control over surgical recordings remain central, with many patients expressing support for video recording if it contributes to quality improvement and objective documentation of care (22). Bias and limited transparency in training datasets are additional concerns, as many current datasets originate from laparoscopic and robotic platforms primarily available in high-resource settings, potentially limiting generalizability (23). Collaborative initiatives, such as international research partnerships, are emerging to expand data diversity and facilitate responsible model development. At the same time, improving education in surgical data science is essential so clinicians can understand and responsibly use AI tools. Growing efforts to promote digital health literacy and interdisciplinary training between surgeons, engineers, and data scientists highlight the importance of integrating clinical expertise with data science to ensure safe and effective adoption of AI in surgery.

CONCLUSION

AI and CV are creating new opportunities to analyze and support surgical care through vast amount of surgical data. Early studies have demonstrated proof of concept applications such as anatomical recognition, instrument tracking, workflow analysis, and objective assessment of surgical performance, particularly in minimally invasive surgery. These developments highlight the potential of computational systems to improve surgical safety, training, and procedural quality. Continued progress will depend on access to high-quality surgical data, development of robust and generalizable models, and careful consideration of ethical and regulatory challenges. With interdisciplinary collaboration and rigorous clinical validation CV technologies may become an important component of the intelligent operating room.