You have a governance framework. What you may not have is governance. One lives in a document. The other lives in what happens the Tuesday after your AI updates its model in production.
Three sessions into Clearwater's AI Healthcare Summit, one message kept surfacing from every angle: implementing AI is no longer the hard part. Governing it is. Session four picked up exactly there, with Clearwater's Dave Bailey and Guidehouse's Cate Ciccolone drawing a distinction that reframes the whole problem. A framework tells you what should happen. Governance tells you what did happen: who approved it, who's watching it, who gets called when there's an incident.
"You have a governance framework," Bailey told attendees. "What you may not have is governance." Most healthcare organizations already have committees, policies, and stakeholders in place. What's missing, more often than not, is the operational discipline that keeps a framework alive after the AI it governs goes into production and starts changing.
Three ways governance quietly breaks down after go-live
Bailey opened with three patterns he sees repeatedly once an AI tool clears initial approval and moves into daily use.
Everyone owns AI risk in theory, which means no one owns the model in practice.
Risk gets assessed at intake, but the AI keeps changing long after that snapshot was taken.
The inventory lives in a document nobody updates, and the SIEM doesn't even know the tool exists.
"You can't protect, manage, or govern what you don't know," Bailey said, tying the point directly back to what the summit's earlier sessions had already established about shadow AI and unmapped attack surface. The fix, he and Ciccolone argued, comes down to four pillars.
The four pillars of operational AI oversight
Continuous Inventory & Risk Tiering
Know every AI in your environment, and rank it by impact.
Monitoring & Drift Detection
Set a baseline. Know the moment it shifts.
Human-in-the-Loop Controls
Someone must be able to say stop, with real authority.
Metrics, Reporting & Cadence
If leadership can't see it, it won't get fixed.
Pillar 1: the inventory is the keystone control
"The biggest misconception I run into," Ciccolone said, "is that organizations think they need a complete operating model before they can govern AI. In reality, governance should start before you have all the answers." Her first move is always ownership: a named executive sponsor, because AI touches clinical operations, compliance, cybersecurity, privacy, enterprise risk, revenue cycle, and quality all at once. "Without a named executive sponsor, governance becomes everyone's job, which means it's nobody's job."
The second move is a one-page charter answering three questions: what falls under AI governance, who has decision authority, and how decisions get escalated. Then comes the inventory itself, which Ciccolone called a keystone control: "You can't assess risk, monitor performance, conduct an audit, review a vendor, or investigate an incident if you don't know the AI exists in the first place." Healthcare organizations that think they have a handful of AI tools routinely discover AI embedded in Epic, Oracle Health, radiology platforms, revenue cycle tools, contact center software, and Microsoft Copilot, often without anyone having explicitly approved it.
Her starting point for building that inventory: three sources. Procurement contracts, the EHR vendor's AI roadmap and feature list, and an organizational amnesty survey, simply asking every department what AI they're using, testing, piloting, or considering, with no blame attached. Connect that inventory to systems already in place, a CMDB, a GRC platform, vendor risk management, rather than building a new spreadsheet nobody will maintain.
On tiering, Ciccolone's advice was to keep it simple. "I've seen healthcare governance groups build twenty-page risk matrices that slow every single project to a crawl. One question does most of the work: how close is this AI to patient care, and what happens if it's wrong?"
- Sepsis prediction, diagnostic support, prior authorization decisions, coding recommendations, clinical triageAnything that substantially influences care, access, safety, or payment.
- Ambient documentation, clinical summarization, operational forecasting, utilization management supportAnything that informs a human decision-maker rather than making the call outright.
- Meeting notes, HR assistance, back-office productivityLower review depth, lower documentation burden, less frequent monitoring.
Pillar 2: uptime isn't monitoring, drift is
"This is where healthcare AI governance is the least mature today," Ciccolone said. Approval isn't the finish line; it's the beginning of a monitoring obligation, because the environment underneath a model keeps changing: workflow changes, population changes, vendor updates, regulatory changes, new data sources.
Uptime is not monitoring. Drift is. A system can be technically available and still be clinically unsafe, biased, degrading, or no longer aligned with its intended use. Cate Ciccolone, Associate Director, Commercial Health IT Advisory, Guidehouse
Organizations should define a baseline before a model ever goes live: what does normal output, accuracy, and latency look like, who receives monitoring reports, what threshold triggers escalation, and who, vendor or health system, owns each monitoring activity. Critically, Ciccolone said, that responsibility belongs in the contract, not just the intake form: "The contract is genuinely where monitoring lives or dies." Governance platforms can automate much of the tracking, but tooling doesn't replace accountability. Clinical judgment, compliance oversight, and executive accountability stay human responsibilities no matter how good the dashboard is.
Pillar 3: a disclaimer is not an override
Ciccolone named three gaps she sees most often missing from healthcare AI governance committees, even the well-staffed ones. Frontline clinicians are the first: "Executives will evaluate AI on capability, but clinicians evaluate it on workflow. I've watched technically excellent tools fail for no reason other than adding five clicks to something a nurse does two hundred times in a shift." Procurement is the second gap, since that's genuinely where AI enters the building; if procurement isn't at the governance table, oversight starts too late by definition. Patient representation is the quieter third gap: patients have a real stake in AI-powered scheduling, chatbots, and digital navigation tools, and they're almost never in the room when it's decided.
A disclaimer is not an override. A human reviewer only reduces risk if they have the right expertise, enough time, enough context, and critically, real authority to stop the workflow. Cate Ciccolone, Associate Director, Commercial Health IT Advisory, Guidehouse
For every high-impact use case, Ciccolone recommends naming the reviewer, writing down who can pause or roll back the model, and logging every override, because regulators will ask, and "we assumed someone would have said something" does not hold up under audit. A mature review process should resemble a clinical review: consistent criteria across clinical safety, privacy, security, compliance, operational impact, monitoring, explainability, and human-in-the-loop accountability, with every review ending in one of three documented outcomes: approved, approved with conditions, or declined.
Pillar 4: if leadership can't see it, it won't get fixed
For organizations starting from zero, Ciccolone's advice is to skip the fifty-page policy and build three operational controls first: a single intake process regardless of whether AI is being built, bought, or piloted; an acceptable use policy, since employees are already using ChatGPT, Claude, Gemini, and Copilot with or without guidance; and a procurement model card requiring vendors to explain intended use, validation methodology, limitations, bias testing, and monitoring capabilities before a contract is signed. "If a vendor can't answer those questions, that in itself is information."
On reporting, Ciccolone recommends tracking five indicators in the same language a board already uses for every other risk: inventory completeness, re-tiering activity, drift incidents, override rate, and escalation time, plus healthcare-specific additions like the number of tools by risk tier, tools with an overdue review, and any AI-related incidents tied to a patient safety event. Set a cadence and hold to it: monthly for high-risk AI, quarterly for medium-risk, periodic attestation for low-risk, and immediate escalation for drift or safety events regardless of tier.
A phased path to maturity
Neither Bailey nor Ciccolone recommend trying to mature all four pillars simultaneously. "Nobody starts at maturity," Ciccolone said, "and trying to build all four pillars at once is usually how organizations end up finishing none of them."
Catalog every AI asset you can find. Assign initial risk tiers. That's the whole phase.
Build drift thresholds and define escalation paths so the monitoring pillar has real teeth.
Connect AI oversight to your existing GRC and vendor risk processes so it stops living as a side project.
Re-tier as things change. Review the metrics. Report to leadership on a real cadence.
Good enough governance for a five-hats compliance team
A question from the audience cut to a real tension: everything discussed assumes a large health system with a GRC platform already in place. What does good enough governance look like for a small or rural hospital with one compliance person wearing five hats?
Bailey's answer: nothing requires an enterprise GRC platform or volumes of documentation. What's required is a reasonable and appropriate set of practices scaled to the organization's size, built on free, trusted guidance that already exists, including the Health Sector Coordinating Council's AI Cyber Governance Framework, the Coalition for Health AI's Responsible AI Guide, and the NIST AI Risk Management Framework. "Nothing says you have to go out and do the enterprise approach if you're not enterprise. Ask a trusted partner. That's what we're here for."
What to take with you
Build both. The framework is where you start, not where you stop.
Not a finish line. The model keeps changing after you approve it.
If no one is accountable, no one is watching.
You don't need every pillar mature on day one. Prioritize.
The organizations that succeed here are not the ones that build the biggest AI governance office. They're the ones making AI oversight part of how they already manage risk. This was never about slowing innovation down. It's about building enough trust, transparency, and operational discipline that healthcare can scale AI safely. Cate Ciccolone, Associate Director, Commercial Health IT Advisory, Guidehouse
Bailey closed with the same practical urgency that opened the summit's earlier sessions: "Artificial intelligence is not something on the horizon. It's here. If you haven't gotten started yet, start Monday. We'll give you the weekend."
Questions this session answers
What's the difference between an AI governance framework and AI governance in practice?
A framework describes what should happen: policies, committees, intended processes. Governance is what actually happens after go-live: who approved a given AI tool, who is watching it, who gets called when it drifts or something goes wrong. Many organizations have a framework on paper without having governance in practice.
What are the four pillars of operational AI governance?
Continuous inventory and risk tiering, monitoring and drift detection, human-in-the-loop controls, and metrics, reporting, and cadence. Each pillar depends on the ones before it, which is why organizations typically build them in sequence rather than trying to mature all four at once.
How should a healthcare organization build its first AI inventory?
Start with three sources: procurement contracts, your EHR vendor's AI roadmap and feature list, and an organizational amnesty survey asking every department what AI they are using, testing, piloting, or considering, with no blame attached. Connect that inventory to systems you already have, like your CMDB and GRC platform, rather than building a new spreadsheet.
How many risk tiers does a healthcare AI governance program need?
Three tiers is enough to start. High risk covers anything that substantially influences care, access, safety, or payment, such as sepsis prediction or prior authorization decisions. Medium risk covers tools that inform a human decision-maker, like ambient documentation or forecasting. Low risk covers back-office productivity use cases like meeting notes.
What does good enough AI governance look like for a small or rural hospital?
It does not require an enterprise GRC platform or volumes of documentation. It requires practices that are reasonable and appropriate for the organization's size, built using free, trusted guidance already available from groups like the Health Sector Coordinating Council, the Coalition for Health AI, and NIST, plus a trusted advisory partner to help translate that guidance into practice.
Is your AI governance real, or just on paper?
Clearwater helps healthcare organizations build the inventory, monitoring, human-in-the-loop controls, and reporting cadence that turn a governance framework into governance that actually works.
Explore AI Risk & Governance ServicesState of AI in Healthcare: Who Defines Trustworthy AI?
Clearwater President Baxter Lee moderates a closing conversation with Brenton Hill, Head of Operations and General Counsel at the Coalition for Health AI (CHAI), and Julie Chua, Applied Cybersecurity Division Director at NIST, on the governance frameworks shaping the future of healthcare AI and what to expect from regulators, boards, clinicians, and patients over the next several years.
Read Session 5 →See the full AI Healthcare Summit series & session replays →
