An AI agent can schedule an appointment, deny a claim, or adjust a medication order without a human ever clicking approve. When it acts wrongly, whose fault is that?
Predictive AI flags a risk. Generative AI drafts a response. A human reviews both before anything happens, and the accountability chain is short and familiar. Agentic AI breaks that chain. It schedules, denies, routes, and logs, executing multi-step workflows with sharply reduced human checkpoints. On paper, that sounds like an efficiency win. In practice, it turns one clean legal question, did the model get it right, into a much messier one: who is accountable when the system acts wrongly?
That question anchored the second session of Clearwater's AI Healthcare Summit, a conversation between Dawn Morgenstern, Clearwater's Chief Privacy Officer, and Adam Greene, a healthcare attorney at Davis Wright Tremaine LLP who spent the week before this session at the American Health Law Association conference hearing the same thing from every direction: every department is being told to implement agentic AI, right now, ready or not.
Why agentic AI breaks the governance models you already have
Morgenstern opened by drawing a line between three categories that most healthcare governance programs were never built to distinguish. Predictive AI flags a risk or predicts an outcome, a human reviews the output, and accountability sits with the person who acted on it. Generative AI drafts content in response to a prompt, a human reviews it before use, and the liability question is evolving but reasonably settled. Agentic AI acts. It schedules, denies, routes, and logs, in multi-step chains with reduced human checkpoints, and accountability for those actions is, in Morgenstern's word, unsettled.
"Human in the loop is going to be a recurring theme throughout today," Morgenstern told attendees early on, and she meant it as a warning as much as a framework. The tools that healthcare organizations already trust, prior authorization support, clinical documentation, scheduling, revenue cycle, triage, are exactly the workflows agentic AI is being layered into first. The shift, as she framed it, is moving governance from asking whether the model answered correctly to asking who is responsible when the agent acts.
Which use cases to move on, and which to slow down on
Greene didn't hedge on this one. "All agentic AI use cases are definitely not equal," he said, walking through a scheduling agent that books July 17th instead of June 17th (inconvenient, not dangerous) against an agent that monitors glucose levels and automatically adjusts medication dosing. "That's what starts keeping me up at night," he said, "because the stakes are really high, and we're now talking about true patient safety concerns that could have fatal results if the agentic AI acts wrongly."
- Prior authorization denialsDirect access-to-care impact; the Lokken v. UnitedHealth precedent applies directly here.
- Clinical documentation & triage supportAgent errors compound into downstream clinical decisions.
- Discharge planning & care coordinationPHI-heavy, multi-system action chains.
- Inpatient monitoring & alertingThe federal government is already here: ARPA-H's ADVOCATE program is building a federally authorized agentic AI that adjusts medications, appointments, and diet for cardiac patients autonomously.
- Revenue cycle & schedulingLower direct patient risk. A more reasonable place to pilot agentic AI first while governance matures.
Before any agentic workflow goes live, Clearwater recommends three questions have documented answers: who approved this agent's access and scope, what does it log at each decision point, and who has the authority to override or shut it down. If any answer is unclear, the workflow is not ready for production. The federal government's own ARPA-H ADVOCATE program builds human override authority directly into its FDA pathway. A commercial deployment should hold itself to the same bar.
The coding risk nobody's asking about yet
Greene named a second concern, one that gets less attention than clinical safety but may be more common: upcoding. "How has your agent been trained?" he asked. Most healthcare organizations run agentic tools on enterprise editions trained on their own data, which means an agent trained on an organization's billing history will learn that history, patterns and all. "If it's trained on your data and you were up-coding before, is your agent going to be up-coding, and whose fault is that? Is that your fault, or is that the vendor's fault, if it was kind of following your footsteps a bit?"
Morgenstern's response cut to the point: an enterprise model only knows what it's shown, and if upcoding was normal before the agent arrived, the agent will treat it as normal too. That's a fraud and abuse exposure that OIG and DOJ can act on today, with no new agentic-specific rule required.
A 50-state patchwork, and the strategy that outlasts it
Multi-state health systems are watching a genuinely fragmented landscape. Colorado replaced its 2024 AI Act with SB 26-189 in May 2026, a narrower automated decision-making technology law taking effect January 2027. Illinois and Texas AI hiring and employment laws are already in effect. California's 2026 wave carries the highest regulatory volume of any state. Meanwhile, at the federal level, Executive Order 14365 directed the Department of Justice to challenge state AI laws directly, a preemption fight with no settled outcome.
Asked where enforcement risk shows up first, Greene didn't hesitate: "My guess is first, class action. Then, second, class action. Then third, class action. And then maybe fourth, state AGs." Plaintiffs' attorneys, he noted, are already shifting from website-disclosure litigation into ambient AI and wiretapping claims, and medical malpractice attorneys are watching agentic AI closely for the same reason.
For an organization operating in a dozen states, the practical answer isn't to chase every statute. It's to build to principles, transparency, human review, documented accountability, that satisfy the substantive intent behind Colorado's law, California's rules, and everything in between at once. Rules get preempted, amended, or struck down in court. Principles survive the churn. Anchoring to a framework like the NIST AI Risk Management Framework gives multi-state organizations one build, not fifty.
What's actually moving at the FDA, OCR, and HHS right now
The regulatory picture is more active than most healthcare compliance teams have bandwidth to track. The FDA has authorized more than 1,250 AI-enabled medical devices, and not one of them is generative or agentic; the boundary for classifying agentic AI as a regulated device remains unsettled, which Morgenstern and Greene both suggested organizations should document as if it were already regulated. The FDA's guidance on AI-enabled medical devices continues to evolve, with new proposals reportedly coming "in short order."
The proposed HIPAA Security Rule update remains in limbo more than a year after publication, with roughly 4,745 public comments and a projected $9 billion first-year compliance cost driving more than 100 organizations to formally request its withdrawal. Meanwhile, ONC's proposed HTI-5 rule would actually remove AI transparency requirements from certified electronic health records, which means organizations should not rely on EHR certification alone as evidence of AI oversight.
Litigation is already ahead of regulation. The Lokken v. UnitedHealth Group case produced a March 2026 federal court order compelling broad discovery into UnitedHealth's nH Predict algorithm, which plaintiffs allege drove post-acute care denials with a roughly 90% appeal reversal rate. OCR's Phase 3 audits are already underway across 50 entities, and the most common gap they're finding is an incomplete risk analysis. None of this required a new agentic-AI-specific rule. Existing HIPAA, malpractice, and non-discrimination law already reach this conduct; as Greene put it, "if one of your people does something wrong, you're going to get sued under something like malpractice or negligence. If that person is now just bits and bytes, I don't necessarily think the cause of action changes dramatically."
Governance that survives the next rule change
Morgenstern's framework for durable governance rests on four habits. First, anchor to the NIST AI Risk Management Framework's Govern-Map-Measure-Manage structure and AI 600-1, not to any single statute; it's built to survive state preemption and federal rule changes, and it lines up with HHS's own OneHHS approach of risk-tiering AI by impact and maintaining a use-case inventory. Second, fix vendor contracts. Most business associate agreements predate agentic AI entirely, and Morgenstern recommends requiring decision logs with access rights, model change notification, bias testing documentation, and explicit audit rights before any agent goes live.
Third, give human-in-the-loop actual structure, not just a policy sentence. That means a named override authority, documented escalation triggers, and pre-action audit trails, not a general assurance that "a human reviews it." And fourth, build feedback loops for incidents and near misses, borrowing the same anomaly-detection instincts security teams already apply to SIEM tools.
A lot of AI governance frameworks look good on paper, but they fall apart when the technology shifts or the incentives change. The ones that hold up over time are risk-based, not rule-based. Dawn Morgenstern, Chief Privacy Officer & Senior Principal Consultant, Clearwater
Greene raised a subtler risk inside "human in the loop" itself: review fatigue. "If someone does a good job reviewing the first time, and the second time, but after twenty times the agent is producing perfect notes every time, do I have any trust that the human is doing as thorough a review on the hundredth time as they did on the first? No." His suggestion: monitor the reviewers the way ambient AI vendors already do, by tracking how much time a reviewer actually spends before clicking accept, and treating a consistently instant approval as a red flag rather than a sign of a well-trained agent.
All agentic AI use cases are definitely not equal. A scheduling mix-up is an inconvenience. An agent monitoring glucose levels and automatically adjusting medication is what starts keeping me up at night. Adam Greene, Partner, Davis Wright Tremaine LLP
Be defensible before you deploy
Asked what documentation a healthcare organization should have in place before deploying agentic AI, Greene pointed to a recognized framework as the foundation, plus proof that vendors can demonstrate compliance rather than simply assert it. "The vendors will tell you what you want to hear," he said. "They're pretty good about that." The test, in his view, is whether a vendor follows a framework like NIST and can show independent auditing, versus a startup whose whole focus is innovate first, ask forgiveness later.
Scope, access level, and a named owner for every agentic system in production.
High-impact AI gets risk controls by design, following HHS's OneHHS model.
What is captured, where it lives, and who can access it.
A named authority and documented triggers for when to intervene.
Audit rights, change notification, and bias testing built into the contract.
Wherever state law requires it, or OCR would expect it.
Greene's closing point on documentation was less about having the records and more about being able to prove you looked at them. He compared it to the inspection checklist taped to the wall in a public restroom: it's one thing to say an organization constantly monitors its agents and nothing has gone wrong; it's another to produce a week-by-week record showing exactly how that monitoring happened. "Think about, from a defense posture, how you can document that you did the good things, and then how you can document that you did the good things."
One guiding principle
Asked to leave healthcare executives with a single guiding principle, Greene offered two, tightly linked. Build strong, dynamic, adaptable governance today, and plan on revisiting it not next year, but possibly next month, because a framework built for a low-risk administrative workflow will need to be revisited when the same organization moves into coding, and revisited again for clinical decision-making. And don't run from agentic AI. "There are going to be people pushing ahead, regardless of what you do," he said. "Embrace those innovations and try to build them into your systems as much as possible."
Morgenstern's closing thought tied the whole session back to the accountability question that opened it: name the owner, log the decision, define the override. Whatever the final regulatory landscape looks like once the FDA, HHS, and the courts finish sorting out where agentic AI fits, that structure holds.
Questions this session answers
What makes agentic AI different from generative AI for governance purposes?
Predictive and generative AI still rely on a human to review the output before anything happens. Agentic AI plans, decides, and executes multi-step actions, such as scheduling, denying a claim, or adjusting a medication order, with reduced human checkpoints. That shifts the governance question from whether the model answered correctly to who is accountable when the system acts wrongly.
Which agentic AI use cases carry the highest liability risk in healthcare?
Prior authorization denials, clinical documentation and triage support, discharge planning, and inpatient monitoring carry the highest risk because they directly affect access to care or compound into downstream clinical decisions. Revenue cycle and scheduling workflows carry lower patient risk and are a safer place to pilot agentic AI first.
How should multi-state health systems handle conflicting state AI laws?
Build to the strictest state standard using durable principles, transparency, human review, and accountability, rather than chasing every individual statute. A single, principles-based governance program can satisfy Colorado's ADMT law, California's rules, and other state requirements simultaneously, and it survives state laws being amended, preempted, or struck down.
What documentation should a healthcare organization have before deploying agentic AI?
An AI inventory entry naming scope, access level, and owner; a risk tier classification; a logging architecture describing what is captured and by whom; a named escalation protocol; a vendor agreement with audit rights and change notification; and a patient-facing disclosure where required by state law or expected by OCR.
Does a HIPAA risk analysis need to look different because AI is involved?
The process itself does not need to change, but organizations should be ready to show regulators exactly where their risk analysis addresses AI as a source of impermissible use or disclosure, and should extend the same scrutiny to vendors whose AI touches their data.
Is your AI governance built to survive the next rule change?
Clearwater helps healthcare organizations build risk-based AI governance that holds up regardless of how the regulatory landscape shifts.
Explore AI Risk & Governance ServicesSession 3: When the Attackers Bring AI: 2026 Healthcare Threats in Plain View
Dave Bailey, VP of Consulting Solutions & Strategy at Clearwater, moderates a panel of security practitioners, Steve Akers, Corporate CISO, Justin Sun, Director of Security Operations, and Philip Burnham, Penetration Tester, breaking down how threat actors are using AI today, from phishing and voice cloning to prompt injection and vulnerabilities inside AI-enabled medical technology.
Read Session 3 →See the full AI Healthcare Summit series & session replays →
