From the thinking hand to the thinking machine

Regulatory Considerations on Clinical Judgement in the Age of Medical AI

← Blog By Chuck Dinerstein, MD, MBA; Matt Medlock; Alex Dagi, MD; Dr. Ileana Pirozzi 7 min read

The Physician is in the Loop. But is the Physician in Control?

Imagine you’re an endovascular surgeon. You are advancing a guidewire through an artery to a distant vessel to seal off a life-threatening aneurysm, relying solely on a two-dimensional image - a live X-ray - to tell you whether or not that additional resistance you felt one millimeter ago was good (distal advancement past a branch point) or bad (subintimal penetration of the vessel wall).

Now imagine an AI-enabled navigation display that can reconstruct the artery in real time, guiding you down the aorta, visualizing the stent being placed, walling off the aneurysm before it bursts. Every movement supports proper placement of a peri-renal aortic graft. And every movement is fully under physician control.

As artificial intelligence enters clinical practice, the FDA is considering how to assess its risks and benefits, and how to regulate its use. Last month, we at Mosaic Intelligence, a San Francisco-based company developing an integrated sensing, navigation, and robotic platform for endovascular interventions, responded to their request for stakeholder comments on how best to regulate the integration of AI into medical devices. Put simply, we believe that regulation should support human oversight of AI systems through design that enhances clinician understanding and promotes human intervention when appropriate.

The Thinking Hand

The earliest depictions of surgery distinguished thinking from doing. Some of the first representations of surgery come from medieval dissections in Europe: a distinguished physician sits high above the room in sumptuous gowns, reading the book of Galen. Below is a surgeon, sleeves rolled up, performing the physical dissection. The hand, and separately, the mind.

Woodcut of a medieval anatomical dissection: a physician reads from a raised chair while a surgeon below dissects the body.
Figure 1: Anatomical dissection in the late Middle Ages. Woodcut from Johannes de Ketham's Fasciculo di medicina (1493).

Surgery has never been merely about technique. In contemporary medicine, the dichotomy between the thinking physician and acting proceduralist has diminished further still: surgeons plan and execute operations, while also managing medical treatment.

As we build intelligent machines, we must further integrate the hand and the mind. Physicians bring medical knowledge, clinical context, and independent judgment to their interaction with AI. The principle guiding us at Mosaic is that, as machines become more capable, they should strengthen surgeons’’ ability to care for the patient, supporting and augmenting their decisions, before, during, and after surgery. Building a machine that can act devoid of that intelligence layer implicitly defines surgery as a purely physical endeavor. Indeed, a technically perfect operation can still be the wrong operation for a patient, leading to clinical deterioration.

In the Loop or in Control?

At Mosaic, we have a straightforward proposition: as machines acquire cognitive capabilities, those systems should extend the clinician’s capabilities and strengthen clinical judgment, not replace it.

The critical regulatory question is not what an AI system says or how autonomous it appears to be. Nor whether a physician remains “in the loop.” Rather, the question is what tasks it performs, the clinical consequences when it is wrong, and whether a physician can recognize and correct those errors.

Fundamentally, machines should not shape the thinking of physicians without also providing tools that buttress their clinical judgment, and they should be designed with four key pillars in mind: (i) independence (ii) influence (iii) uncertainty (iv) observability and (v) reversibility. Only with such thoughtfully designed AI systems can clinicians move from “in the loop” to “in control.”

I. Independence: What Oversight Exists When AI is Wrong?

We agree with the FDA’s proposal of a two-axis risk framework that provides a focused, interpretable assessment of AI-enabled functions.

Chart: consequences (limited, moderate, severe) against activity (informational non-directive, informational action-directing, action-taking HCP-supervised, action-taking fully autonomous), with risk increasing toward the top right.
Figure 2: The FDA's proposed two-axis risk framework for AI-enabled medical device software. Adapted from FDA’s “Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback”

The first axis concerns device activity: What does the software actually do? Does it provide information, interpret data, recommend next steps, or independently perform actions?

The second concerns potential harm: What consequences could result if the software's output or action is wrong?

Conceptually, the framework draws on approaches used in other safety-critical domains, including autonomous vehicles and medical devices. While automotive standards are not directly transferable to medicine, they demonstrate the value of delineating the roles of the human operator and the automated system, and of defining factors that modify the likelihood or severity of harm.

For medical applications, the critical question is less about "autonomy" than about which functions the clinician performs and which the software performs. Additional critical questions include who monitors performance, and who is expected to respond when system performance is inadequate. In that way, whether a machine performs a function autonomously does not indicate whether a machine performs a function safely or unsafely.

Moreover, responsibility must be shared between the clinician and their partner AI system. The allocation of responsibility is central to understanding the potential consequences of failure, as reflected in surgical Morbidity and Mortality conferences. When a complication occurs, the objective is to identify where the decision process or system failed, and determine what might prevent a recurrence. AI systems must be observable in a way that permits root cause analysis. And, when responsibility rests with the system, accountability must be ensured, just as it would with human error.

II. Influence: Tone Matters More Than Words

One concern raised by the FDA was the language used by AI systems to express recommendations. Mosaic argues for a distinction between linguistic tone - whether the AI says "consider," "recommend," or "do," -- and functional influence, where the output actually changes clinician behavior.

We believe the FDA should give greater weight to influence.

Contemporary research demonstrates that professional expertise may reduce the risk of misunderstanding an AI recommendation–without necessarily reducing the risk of being influenced by it: clinicians repeatedly alter their decisions based on AI recommendations–sometimes beneficially, and sometimes detrimentally. This occurs in part due to automation bias: accepting a recommendation because it is readily available, plausible, and authoritative. Sycophantic AI can reinforce confirmation bias rather than challenge it.

Healthcare professionals routinely exchange recommendations and reconsider decisions without treating each communication as an authoritative command. An AI system should support this same iterative process, and be designed to facilitate–rather than influence–physician understanding.

If regulators disproportionately penalize decisive language, manufacturers may be incentivized to produce vague outputs that are linguistically defensible but less clinically useless. In contrast, if regulators deprioritize oversight of functional influence, even apparently indecisive language may bear significant implications for patients.

We believe the regulatory objective should be to preserve independent professional judgment, not to reduce the language's declarativeness. Our goal is to achieve appropriate reliance: accepting useful information while recognizing when a recommendation should be questioned or disregarded.

III. Uncertainty, and the Danger of False Certainty

One of the more consequential distinctions in clinical decision-making is between uncertainty itself and awareness of uncertainty.

A clinician who recognizes that their–or the system’s–assessment is uncertain may seek additional information, change course, or defer an action altogether.

The failure to recognize uncertainty–either because the system communicates unwarranted certainty or because the user places in it unwarranted confidence–is potentially more hazardous. Just as human clinicians communicate doubt and uncertainty, so too should AI systems be expected to articulate levels of confidence. Or, lack thereof.

IV. Observability: Designing for Detectability

We believe meaningful assessment by the FDA involves evaluating observable modifiers of underlying clinical risk, in two key ways: (1) whether the recommendation or action is sufficiently traceable to reconstruct how the decision was reached and (2) whether there are explicit detection mechanisms, opportunities for intervention, and other safeguards that can interrupt the progression from error to harm.

These questions recognize that a physician's ability to challenge AI output cannot be assumed from professional credentials; expertise cannot eliminate automation bias, confirmation bias, or inappropriate reliance on apparently authoritative information. AI systems must be designed for transparency and observability to ensure informed clinician decision-making is preserved.

V. Reversibility: Timely Recognition is Key

Put yourself, again, in the shoes of a vascular surgeon, entering the wrong vessel. The outcome forks significantly, depending on when recognition of the error occurs. Recognize early, and you can re-select correctly. Recognize late, and a misplaced stent could prompt greater harm. In this way, processes that support reversibility are essential.

However, reversibility alone does not equate to safety: a momentary decision may produce consequences that persist long after the intervention ends.

For example, an error in endovascular device apposition may produce no immediate clinical effect yet contribute to early and mid-term restenosis, thrombosis, or occlusion.

The relevant risk assessment must therefore consider not only whether an error is detectable, but the delay in recognition, and the resulting harm. A technically reversible action may have substantial consequences if the need for reversal is not recognized within the relevant therapeutic window, or when reversibility is difficult to attain.

Accordingly, a technically successful procedure is not necessarily evidence that every decision made during that procedure was correct. Nor, that the outcomes will be optimal.

Physician-Centric AI Should Strengthen Judgment

Mosaic's launch comes as the medical community and its regulators are defining how clinicians and intelligent systems should (and already do) collaborate.

Mosaic’s work in intelligent endovascular navigation reflects a broader conviction: that better, multi-dimensional, easily-interpretable information can strengthen physician judgment beyond technique alone.

Ultimately, the measure of medical AI’s risk should not hinge upon how autonomously a machine makes recommendations. It should be whether the interaction produces better-informed clinical decisions while preserving the physician's ability to question, intervene, and remain accountable.

For Mosaic, the future of medical AI is not a machine making decisions for a physician. It is a physician making better decisions with a machine.