Enterprise AI Analysis
Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology
Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delayed in their detection, but gaps remain in clinical practice integration. In this study, we present a human-centered approach to the problem and describe insights derived following contextual inquiry and in-depth interviews with 15 clinical stakeholders. The interviews helped understand challenges in existing workflows, and how best to align technical capabilities with user needs and expectations. We discovered the tradeoffs and complexities that need consideration when choosing suitable workflow stages, target users, and design configurations for different AI proposals. We explored how to balance AI benefits and risks for healthcare staff and patients within broader organizational, technical, and medical-legal constraints. We also identified data issues related to edge cases and data biases that affect model training and evaluation; how data documentation practices influence data preparation and labeling; and how to measure relevant AI outcomes reliably in future evaluations. We discuss how our work informs design and development of AI applications that are clinically useful, ethical, and acceptable in real-world healthcare services.
Executive Impact
Key metrics highlighting the critical need for improved NGT placement verification in healthcare.
0
NGT Misplacement Rate
0
Complication Rate from Misplacement
0
NGT Template Completion
0
NGTs Used Annually (Europe)
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
2
Critical NGT lung misplacements in 6-week ICU period
Enterprise Process Flow
Patient has NGT fitted
→
First line test: Aspirate
→
Placement check via CXR
→
Radiographer takes X-ray at bedside
→
Doctor assesses and documents NGT on EHR
→
Radiologist/radiographer reports NGT
→
Patient can be fed via NGT
The "Whoosh Test" & Documentation Friction (D5, ICU)
One ICU registrar questioned the mandatory NGT documentation process as "a lot of over-doing" for clear cases, recalling how in other places they would use the "whoosh test"—blowing air into the tube and listening for gurgling in the stomach—and wouldn't document it unless there was doubt. They emphasized that, while current protocols enforce visual checkpoints and detailed entries (Figure 2), this can create friction and compete with the desire to speed up clinical decisions, potentially leading to 'lazy' practices if AI directly auto-populates notes, bypassing clinician review. This highlights the tension between strict safety protocols and the practical need for efficiency, especially for experienced staff who feel confident in their assessments.
Quote (D5, ICU): "(...) In my personal opinion only, it's a lot of over-doing. I've worked in other places and it wasn't done like that. We would put the NG feed in, we would use what is known as the whoosh⁹, which is basically blowing air inside and listening to see if it, or feeling that it's in the stomach, if it's there and if you can aspirate anything out of it, then it's fine. We wouldn't document it. It's like a routine procedure that we do, we wouldn't document it. (...) I'm saying if there's a doubt and we would also order a Chest X-ray and we will always document what we saw in the X-ray. But we don't have like a special... I never used a special entry template. I never had to go through like a test to make sure I can do that. It's taken for granted that if you are a doctor that you are supposed to be able to look at a Chest X-ray, you don't need to be tested for it."
| AI Capability & User | Workflow Stage | Key Benefits | Key Risks |
|---|---|---|---|
| Detection of (critical) NGT misplacement (Imaging radiographer) |
CXR preview image |
|
|
| Critical findings alert in EPIC (ICU doctor) |
CXR image review |
|
|
| CXR image prioritized in PACS reading list (Reporting radiographer/radiologist) |
CXR reporting |
|
|
Unusual Patient Anatomy: Hiatus Hernia & Situs Inversus
Clinicians highlighted that patients with unusual anatomy, such as hiatus hernia (where the stomach moves into the chest) or situs inversus (major visceral organs in a reverse position), present significant challenges for NGT placement verification. In these cases, standard protocols and visual checkpoints (Figure 2) may not apply, requiring more individual assessments, consultations with specialists (e.g., gastrologists), and review of patient history. AI models trained on 'normal' anatomies may struggle with these edge cases, leading to misclassifications or requiring human oversight to interpret deviations from typical placement. This underscores the need for AI development to account for diverse patient populations and to incorporate relevant clinical context (EHR data) to avoid biases and ensure safe outcomes.
Quote (RR1, ID): "(...) one of the things that is hard to bottom out is: at what point in the stomach is it safe to feed? Because if it's at the gastroesophageal junction, do you run the risk that any patient head movement will then dislodge the tube that becomes oesophageal? And how do you say, you know, it's 5 centimeters past the GOJ16? Part of the problem about standardizing the interpretation; expectation and assessment for radiographers is, is it 5 or 10 centimeters passed the GOJ, because once you've set that standard, that's it. For me, as long as it appears radiographically clear of the GOJ, obviously the Chest X-ray is 2D flat. You know about where the GOJ is, if it is over the stomach. It's safe to feed. It's OK.”
Quantify Your AI Impact
Tailored for healthcare enterprises, this tool helps you quantify the potential ROI of integrating AI solutions. By adjusting key variables, you can visualize the impact on operational efficiency and cost savings.
Projected Annual Savings
Total Hours Reclaimed
AI Implementation Roadmap
A phased approach to responsibly integrate AI in healthcare, ensuring clinical utility and patient safety.
Phase 1: Assessment & Strategy
Conduct in-depth user research to understand current workflows, challenges, and stakeholder needs. Systematically map AI capabilities to clinical utility, defining clear goals and success metrics. Align AI strategy with organizational constraints and ethical guidelines.
Phase 2: Data Foundation & Model Development
Leverage existing standardized clinical data (e.g., NGT templates) for efficient label extraction. Address data biases and edge cases (e.g., unusual anatomy, image quality) through careful dataset curation and advanced training techniques. Develop models that prioritize patient safety and minimize false positives for critical findings.
Phase 3: Human-AI Integration & Pilot Deployment
Design AI interactions that integrate seamlessly within existing human safeguarding processes, rather than requiring separate 'verification' steps. Pilot AI solutions with specific user groups (e.g., ICU doctors, imaging radiographers) in controlled environments. Gather feedback to iteratively refine the system, balancing efficiency gains with human skill acquisition and trust.
Ready to Transform Your Healthcare AI Strategy?
Connect with our experts to explore how human-centered AI design can unlock significant value for your organization, improve patient safety, and streamline clinical workflows.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat