State Farm is making its agents pay for its broken artificial intelligence. The first thing the new “instant summary” tool did, when an Illinois agent tried it on her own account, was report that she had been a State Farm policyholder for almost twenty years fewer than she actually had been, and mark her as late on a payment she was not late on. The agent, who had been doing the job long enough to know the difference, did not need the tool to tell her that the tool was wrong. The general counsel of the National Association of State Farm Agents told the Wall Street Journal that “over the last 10 years, everything that’s been rolled out has had issues. There have been a lot of outages.” A separate ChatGPT-style tool that agents are encouraged to use for everything else has a “knowledge cutoff” of last year, which means its most confident answer about who won the most recent Super Bowl it remembers is the Kansas City Chiefs, in 2024. The Journal collected responses from more than a thousand customers and agents. The dominant adjectives were “terrible,” “infuriating,” and “it sucks.” The dominant noun, if you read closely, was “representative” — the word one shouts into a phone at a chatbot that will not transfer.

It is true that State Farm will tell you — at length, and in the way companies do — that it is not trying to replace its agents with AI. The strategy is “human + digital.” Ninety-three percent of agents have voluntarily opted in to the household-summary tool, which is, on the documented record, a higher adoption rate than most working adults would give a piece of enterprise software issued by a head office, including the office coffee maker. The State Farm position is the position the entire industry is taking: the technology is in service of the human, not a replacement for the human, and the future is human + digital, not digital alone. The trouble is that the policy being implemented is not, on the agent testimony, “human + digital.” It is “digital, with humans retained only to fix what the digital breaks.” An agent in Alabama told the Journal that State Farm is “a great insurance company … a horrible tech company.” A recently retired agent in Mesa, Arizona said the more State Farm tries with technology, the more work it creates for the agent’s office. “AI will most likely not be any different.” Several agents said they expected the smaller offices to consolidate or close, which is the consolidation the company says it is not, formally, planning, and which the company’s own statements to investors imply is, in fact, the operating plan.

The “instant summary” tool is, technically speaking, a retrieval-augmented generation system — a language model that pulls from a customer-data store and produces a natural-language summary of what it found. The architecture is not exotic. It is the same pattern deployed in enterprise copilots, customer-service assistants, and roughly two-thirds of the “AI transformation” press releases the consulting industry has issued since 2023. What the architecture does, when it works, is compress retrieval and presentation into a single call. What it does, when it does not work, is the thing the Illinois agent watched it do: it hallucinates. A retrieval-augmented system can fabricate a date that is not in its retrieval set, or misclassify a payment status, and the only signal to the agent reading the screen is the absence of a fact that should have been there. The error is silent. The agent has to know the customer well enough to catch it. The whole point of the tool, as marketed, was that the agent would not have to know the customer that well.

This is the structural pattern, and the pattern is older than the tools. Cory Doctorow has been writing since 2022 about enshittification — the four-stage sequence by which a platform is good to its end users, then good to its business customers, then claws value back from both for shareholders, then dies. Insurance is not, in the original Doctorovian sense, a platform. The mechanism is. The mechanism is good to the policyholder while the local agent is the human face, then good to the company while the local agent absorbs the cost of the AI’s errors, then extract the surplus of the relationship by reducing the number of agents paid and the offices maintained, then — at the end of the sequence, which the industry is not at yet — collapse the personal relationship that was the actual product. That is, in plain language, what is happening at State Farm. The “human” half of “human + digital” is being treated as overhead to be minimized. The “digital” half is being treated as the product. The tool that hallucinates a payment status is not, on the company’s own incentive structure, a bug. It is a transitional cost. The transition it enables is the cost of the agent.

When a customer’s household summary generated by the model lists a policy start date that is wrong by two decades and a late payment that never occurred, the machine has not solved a problem. It has invented a new one, and it has assigned the problem to the agent, who must now spend fifteen minutes reconciling the fabricated data with the actual general ledger before the customer’s policy can be renewed. This is not efficiency. This is cost-shifting. The company saves the latency of a corrected database query; the agent absorbs the latency of a manual override. The financial sector is already grappling with this exact AI hiring dilemma as tools reshape banking jobs, watching the same pattern of institutions deploying systems that require a human to supervise the machine’s mistakes. The new bipartisan coalition aiming to prepare U.S. workers for AI upheaval will find in State Farm’s rollout a pristine case study of what that upheaval actually looks like on the ground: it looks like a worker doing the machine’s job, plus the machine’s job, for the same salary.

The mechanics of this cost-shifting are precise and deliberate. The insurer introduces a system that promises to cut labor costs, and then degrades the worker’s environment by making the worker the exception-handler for the system’s inevitable failures. The mechanism is the continuous, computer-mediated adjustment of the worker’s environment, where the machine constantly tweaks the parameters of the agent’s daily workload, inserting new friction that did not exist a quarter ago. The politics of the machine are never about the machine itself, but about who owns it and to what purpose. The agent is told the tool will free up time to sell more policies. What actually happens is that the tool creates more work for the agency office to sort out. The labor discipline is the point.

It is also true that State Farm is doing what its competitors are doing, which is to say it is doing what every large insurer is being told by its consultants to do. Allstate’s CEO told investors in April that Allstate is piloting direct AI sales in three states — a spokesman declined to name the states or what the AI tool is selling, which is its own small piece of evidence about the marketing-versus-reality gap. Allstate has also given its agents an AI “customer engagement sidekick” to listen in on their calls. The same labor-displacement pattern is playing out across Wall Street, where banks are pushing AI into hiring decisions at the same speed and with the same disclosure record. The entire industry is running the same playbook at the same time. The frame inside the industry is “future-proofing.” The frame outside the industry is the one the Mesa agent used: the more the company tries, the more work it creates for the people who actually have to deliver the service.

A mutual insurance company is, structurally, owned by its policyholders. That is not a small legal detail. State Farm is not a public company; it has no shareholders to deliver value to in the quarterly sense. It has members, who are the people who buy its policies. The company’s own statement emphasizes the mutual structure: “technology investments based on what’s right for our customers over the long term, not simply how quickly we can bring new capabilities to market.” It is a real structural commitment, and it is one of the reasons the agent model survived this long. The mutual structure, in the best version of itself, is the institution that is supposed to make the trade-off between cost-cutting and customer service differently than a public company would. The trade-off State Farm is making right now is the trade-off a public company would make, and the fact that the legal wrapper says mutual does not change the fact that the operating decision is replace the agent. The policyholders who technically own the company are the population the company is making the decision against. The 104-year-old company is acting, in this respect, exactly as if it had impatient external shareholders to please.

The 19,000 agent offices are not, strictly, State Farm’s. They are independent contractor offices, many of them family businesses, several of them second- or third-generation. The customers are not, strictly, captive in the sense that a Facebook user is captive. They can switch insurers, and some of the Journal’s respondents said they would. But insurance is a high-switching-cost product — the renewal pricing, the claims history, the deductible reset, the multi-line bundling, the local-agent relationship that one does not lightly abandon. The switching cost is high enough that the company can absorb a non-trivial rate of customer loss and still benefit from the consolidation. The agents cannot switch. The agents whose contracts were just amended to terminate a retirement benefit — the company walked that back, but only through 2031, and only “subject to a baseline performance expectation” — are the population with no exit at all. They will, in the tradework sense, get the broken tool and the time to fix what it breaks. That is the deal.

Doctorow’s analytical frame, in The Internet Con and the longer Enshittification book, identifies four forces that historically kept platforms honest: competition, regulation, self-help (the right of users to make the platform work for them, against the platform’s wishes), and labor (the worker who refuses, on grounds of craft or conscience, to ship the broken thing). Each of the four is being weakened in the insurance case, by design or by drift. Competition: the industry is consolidating around a small number of carriers using the same AI vendors, the same consulting playbooks, and the same “future-proof” framing. Regulation: state insurance commissioners are the relevant regulators, and they are well-meaning, but the standards they enforce are about solvency and rate adequacy, not about whether the chatbot hallucinates. Self-help: the customer cannot audit the retrieval-augmented system. The agent can, in principle, but the agent’s contract has just been amended. Labor: the agent who refuses to use the tool, on the grounds that the tool gives wrong answers about her own policy, is the agent the company is looking for an excuse to retire.

There is a tradesman’s phrase for what happens when a manufacturer insists a device is working but the end user cannot get it to perform its basic function: you do not rewrite the manual, you take the thing apart and look at the wiring. State Farm’s own general counsel at the agents’ association admits that every technology rollout over the last decade has carried issues and outages. The diagnosis is not a temporary bug. It is an institutional failure to maintain the underlying infrastructure, now masked by the vocabulary of artificial intelligence. You cannot patch a legacy billing system with a chatbot, any more than you can fix a blown bearing in a rolling mill by putting a digital readout on the control panel. The bearing is still blown; the operator just has a new screen to watch it fail.

The competitive landscape, as the executive memo to agents put it, is changing, and standing still is not an option. It is always worth asking who is standing still and who is being asked to run. The policyholders who just want to call their agent and not spend ten minutes shouting “representative” into an automated phone tree are the ones who will pay the price, either in rate increases to fund the AI pivot or in service degradation when the agent’s office inevitably closes because the margins no longer support the manual reconciliation of machine hallucinations. A thousand readers wrote in to say they would switch insurers if their local office shut. They will not switch to a company that does not use AI; they will switch to one that uses a different one. The underlying extraction, however, remains the same. You can pay extra for the privilege of being sold to by a chatbot that does not know what year it is.

The remedy on the regulatory side is not mysterious. Insurance regulators could require disclosure of the training data, the retrieval sources, and the documented error rates of customer-facing AI tools — the same way financial regulators require disclosure of model risk. Insurance regulators could require a right to a human at the point of service, the same way some jurisdictions have begun to require a right to a human for essential government services. Insurance regulators could, if they had the appetite, treat the practice of marketing a tool as “human + digital” when the operating decision is “digital, with humans to fix it” as the kind of misrepresentation mutual companies are supposed to be above. The affirmative position is structural: the workers who are forced to reconcile the machine’s errors must have the legal standing to audit the systems that govern their labor, and the mutual policyholders must have the right to a human resolution when the stochastic engine fails. The state insurance commissioner in Illinois, where the hallucinating agent is based, has a comment line that accepts complaints. The form is online. Deadlines are, again, the only part of regulatory processes that the regulated actually respect.

There is a Canadian parallel. The Canadian life-insurance industry went through its own version of the AI-customer-tools rollout in 2023 and 2024, and the relevant federal insurance regulator — the Office of the Superintendent of Financial Institutions — has been moving, in published guidance and consultation, toward disclosure requirements for model risk in customer-facing automated decision systems. The U.S. state insurance commissioners, who have a more politically constrained relationship with the carriers they regulate, have not produced an analogue. The institutional gap is the story.

The architecture choice State Farm is making is not, on the record, irreversible. The mutual structure means the policyholders are, in principle, the constituency the company answers to. The agents are, in principle, the people the customers trusted to deliver the service. The “human + digital” framing is not, in principle, a synonym for “replace the human and absorb the digital errors.” It is a marketing line. It can be unmarketed. The training data the AI tool was built on is two years out of date. The tool that hallucinates a payment status can be pulled, debugged, and redeployed with the agent’s experience folded into the design — which would be the engineering-substance approach, and which would also be the approach that produces a tool an agent is willing to defend. None of this requires a new law. It requires the company’s leadership to read the customer mail, take the agent mail seriously, and decide which constituency they are actually answering to. The Journal received more than a thousand responses. The general counsel of the agents’ association is on the record. The work, as ever, is to be done.