State of AI in Healthcare: Who Defines Trustworthy AI?

Four weeks earlier, this series opened with a question: who owns AI risk? Since then, Clearwater's AI Healthcare Summit worked through agentic AI's regulatory uncertainty, watched attackers turn AI into a weapon of their own, and got practical about operationalizing governance inside real organizations. The finale closed the loop with the question underneath all of it: as healthcare moves from experimenting with AI to deploying it at scale, who actually defines what trustworthy means, and how does an organization put that into practice?

Clearwater President Baxter Lee put the question to two people writing much of the rulebook the rest of the industry is trying to follow: Brenton Hill, Head of Operations and General Counsel at the Coalition for Health AI (CHAI), previously of Mayo Clinic Platform, and Julie Chua, who leads the Applied Cybersecurity Division at NIST and spent years inside HHS's Office for Information Security co-leading the 405(d) program.

Healthcare is leading on AI, cautiously

Asked whether healthcare is running toward AI or standing back from it, Hill's answer cut against the industry's usual reputation. "Healthcare is usually the group that adopts technology last," he said. "For once, we're seeing that healthcare is actually leading in some of this. Maybe not on everything, but there's a lot of optimism, bundled with an abundance of caution." Chua saw the same pattern from the standards side: organizations are leaning into governance now, in her view, because the sector already lived through the slow, breach-driven path to taking cybersecurity seriously and doesn't want to repeat it with AI.

Inside CHAI's governance playbooks

CHAI's health system members kept telling Hill the same thing: they understood the principles of responsible AI in theory, but needed concrete, lifecycle-grounded steps to actually act on them. That gap produced eight governance playbooks, built with the Joint Commission and more than 100 health systems, covering AI policy, organizational structure and resourcing, AI inventory, risk and impact assessments, responsible data use, third-party vendor management, and staff training and feedback.

Hill was direct about what the playbooks are not: a finished answer. "The goal isn't to give you all the answers, because governance really needs to be tailored to your individual organization's risk position." He described it as a maturity-based framework, one where standing up an AI policy, starting an inventory, and limiting shadow AI on day one counts as a legitimate starting point, not a shortcut.

Even Hill admitted where the playbooks get hardest to implement: AI monitoring. "There's no clear, defined metrics for what we should monitor, and that's a result of the pace the technology is moving. If we spent a year working on metrics for any given use case, you'd have three other use cases that are far more advanced by the time you finished." Training and incentivizing responsible use is the other soft spot, since staff range from AI skeptics to people who want to automate their own job, and organizations don't yet have a proven playbook for moving that whole spectrum toward safe, productive use.

What a good vendor intake actually asks

Much of the current bottleneck, Hill said, sits at the front door: intake, screening, and contracting, with some large health systems reportedly carrying 200 to 500 AI solutions in their procurement queue at once. CHAI studied more than 800 intake questions across 15 institutions and distilled the pattern into roughly 60 core questions spanning eight categories.

The last two categories Hill named were model validation and monitoring, and impact measurement and ROI, both of which loop back to the same open problem: the field still doesn't have settled, widely-adopted metrics for what ongoing AI monitoring should actually look like.

NIST's role: cultivating trust through research, standards, and practice

Chua described NIST's approach to supporting healthcare and other critical infrastructure with a simple shorthand: fundamental and applied research, standards development done in partnership with industry and academia, and tech transfer, meaning research and standards actually validated in practice. That last piece happens largely through NIST's National Cybersecurity Center of Excellence, where NIST builds test beds for its own frameworks, demonstrates that they hold up in real implementations, and takes feedback on where the guidance needs to adjust.

That same logic shaped a recent change to the NIST Cybersecurity Framework. "If anything, the key takeaway from my remarks today is governance and enterprise risk management going hand in hand," Chua said, explaining why CSF 2.0 added a formal Govern function. The NIST AI Risk Management Framework's companion playbook extends that same governance logic to AI-specific trustworthy characteristics, like being secure and resilient, and fair with harmful bias managed, all of which Chua said should get addressed and managed inside an organization's existing governance structure rather than a separate one. The AI RMF itself is currently being updated, with a public comment period expected later this year.

Governance and enterprise risk management go hand in hand. That's fundamental and imperative with any emerging technology an organization chooses to implement, deploy, or integrate into its ecosystem. Julie Chua, Applied Cybersecurity Division Director, NIST

For organizations just getting started, Chua pointed to resources built for exactly that: NIST's Small Business Program, a Quick Start Guide for CSF 2.0, one for enterprise risk management, and a newer Quick Start Guide covering cybersecurity, ERM, and workforce management together, built specifically because organizations kept treating AI workforce readiness as an afterthought rather than a prerequisite. She also flagged NIST's Cyber AI Profile, built to translate between the cybersecurity and AI communities across three lenses: securing AI, defending with AI, and countering AI-specific adversarial threats.

Does governance slow innovation, or enable it?

Chua's answer was structural: organizations that get this right don't build a separate AI governance apparatus. They route AI through the steering committees and risk councils that already exist, treating AI as one more branch feeding into enterprise risk management rather than a parallel bureaucracy competing for the same people's time.

Hill's answer was about precision. "The reason governance does the opposite of slowing things down is you're getting the right people with the right expertise solving the right problem along the governance chain." Organizations succeeding at this apply a genuinely risk-based approach: low-risk tools get a light process, high-risk tools get real scrutiny, and nothing gets the same treatment by default. He pointed to a specific pattern among CHAI's members: business leaders who own an AI use case partnering directly with attorneys on contract language produce far sharper protections than lawyers working alone, because the business owner understands the workflow well enough to ask for the right things: advance notice before a vendor changes a model feature, the right to revalidate before accepting an update, renewal or payment tied to sustained performance benchmarks, and the right to disable a new feature until it has been risk-reviewed.

You can't push responsibility out to a federated model until the guardrails are in place. But once they are, the people closest to the use case can move very quickly within approved applications and approved guardrails. Baxter Lee, President, Clearwater

Who actually owns AI risk

The series opened with this exact question in session one, and the finale circled back to it with a sharper answer. Hill pointed to the Federation of State Medical Boards' 2024 statement: a physician using an AI application in their own practice is responsible, and ultimately liable, for it, regardless of who supplied the tool. Nursing boards haven't issued the equivalent guidance yet, which Hill flagged as a real gap.

At the organizational level, Chua was direct: an enterprise governance mechanism, meaning leadership, should own AI as a category, with individual, discrete risk owners managing specific use cases day to day and escalating when a defined threshold is crossed. "When you elevate a risk, it's not failure," she said. "It means you're managing, and you have a good governance process." Hill's field observation was that organizations routing AI ownership through a dedicated AI officer, in partnership with IT, tend to move fastest; those routing it purely through traditional, often understaffed compliance departments are still figuring out whether that structure can keep pace.

Two pieces of advice, thirty seconds each

Asked for one piece of advice for a leader building an AI governance program this year, Chua's answer was cross-domain collaboration: pull clinicians, compliance, legal, and security into the same conversation, evaluate trustworthiness against the organization's actual mission, and never separate AI risk from the enterprise risk management mindset already in place. Hill's answer was a single word: defensibility. "Know what you're going to track, know your positions, know what you're going to audit, and build your case for the moment something happens, because something will happen. You need to be prepared to defend your use of AI."

On whether CHAI's playbooks and the NIST AI RMF compete or complement each other, Hill was unambiguous: CHAI built its framework directly on top of the NIST AI RMF, adapting it with healthcare-specific specificity. And on the growing problem of vendors quietly adding AI to tools organizations already use, Chua's guidance was equally direct: "You shouldn't find out as a surprise that AI is in your products." That has to be a contract requirement, not an assumption, with organizations retaining the explicit right to say yes, no, or not yet.

Questions this session answers

Watch any session's full replay on the AI Healthcare Summit page →