How reassuring is it to hear “I asked Gemini” from someone defending their position amid a political furor? If you’re the president of South Korea, apparently not very.

President Lee Jae Myung kicked off an outcry after posting on X that he had checked with Alphabet Inc.’s Google AI tool to rebut critics who said inviting officials from two investigative bodies to a cabinet meeting could undermine their independence. The blowback revealed an almost comedic stage of the AI boom, fueled in equal measure by suspicion and hype.

That’s a known weakness of large language models. While rates of wildly inaccurate responses, or hallucinations, have fallen as models have improved, research shows that outputs can still vary substantially depending on how a prompt is framed. That’s worth keeping in mind before asking a chatbot to mediate a political argument.

Yet one of the loudest lines of attack in this saga, echoed by many online commentators and a former Reform Party leader, was simple: If the president is going to govern by asking Gemini, why not just replace him with AI? Lee said he was merely checking facts, not outsourcing decisions. But the jab struck a nerve. The dueling Gemini screenshots that have consumed Korean social media may look absurd, but they also expose the limits of public trust in a technological revolution racing ahead of the rules governing it.

And beneath the noise lies a harder question: How much of a leader’s judgment are we willing to outsource to machines? A common refrain from the tech industry is that as AI takes over tedious work, human judgment will become more valuable. What we might not realize is just how easily that decision-making can be shaped by machines, too.

Even experts can be heavily influenced by AI, a new Anthropic PBC research found. In an experiment, professional analysts working on a realistic statistical task went along with an AI agent’s randomly assigned “lean” on key analytical choices 86% of the time. When asked to explain their preferences, 77% of analysts used the agents’ own arguments, sometimes verbatim, while only 15% acknowledged its influence.

That’s harder to catch than an outright factual error, like a tool getting a number wrong. Judgment calls often have no single correct answer, so the work can look perfectly sound whichever option is chosen, the paper argues. “Asking experts afterward may not help either, because they may not notice the agent’s influence,” according to the Anthropic report. From the White House to the boardroom, we should worry as much about AI coming for our decision-making as our jobs.

There are several ways to push back. The researchers suggest designing agents to present competing options, asking users for their own views before offering advice, and having AI sound less certain. But none is an easy fix, and some might defeat the point of turning to AI in the first place.

There are myriad ways that AI can make governing more efficient. Using a chatbot to research how to interpret the law isn’t the same as handing over the executive branch. And, at this point, I’d be more suspicious of someone in office denying their use of these tools. Lee isn’t the first world leader to talk about using AI and he certainly won’t be the last. To his credit, he disclosed what he was doing, which is more than can be said for others. But in the future, disclosure will only go so far if users themselves can’t recognize how much a machine is swaying their judgment.

Ultimately, where assistance ends and influence begins is becoming harder to distinguish. As AI moves deeper into governments far beyond Seoul, the line may disappear before we decide where to draw it.