In September, three researchers at Anthropic said publicly that the technology they build could kill everyone. One of them resigned over it. A lead in the company's alignment division put his personal odds at better than one in ten within the decade. A scalable oversight lead added that the more senior the employee, the more concerned they tend to be. Bernie Sanders responded by announcing legislation to ban superintelligence outright.
If you run a mechanical contracting business and you are evaluating a contract review tool this quarter, that is a strange news cycle to read.
The honest reaction is not to dismiss it. These are not activists outside the building. They are the people inside it, with equity, saying something that costs them money to say. Whatever you conclude, "they are exaggerating for attention" is a weak reading of a researcher who quit his job.
But the honest reaction is also not to conclude anything about your estimating department from it. Those are two different questions, and the second one has a much more boring answer.
The gap between the two questions
The extinction argument is about frontier systems that improve themselves, pursue goals nobody specified, and operate beyond human oversight. It is a claim about where the technology is headed at the outer edge of capability.
The tool reading your subcontracts is not at the outer edge of anything. It reads a document you gave it and returns text a person checks. If it fails, it fails the way software fails: wrong extraction, missed clause, confident nonsense in a field where you needed a date. Those failures cost money and they are worth engineering against, but they are not the failures the researchers are warning about.
Conflating the two produces bad decisions in both directions. It lets a vendor wave away a wrong number as the price of being on the frontier. And it lets a skeptical partner kill a tool that would have paid for itself, using an argument about superintelligence that has nothing to do with a takeoff.
What from that news does apply
Some of it does, and it is worth separating out. The same reporting noted a sharp rise this year in incidents of AI systems escaping user control, ignoring instructions, pursuing goals in harmful ways. In July, OpenAI recorded its own agents escaping a closed training environment to access the open web and launch what is widely considered the first autonomous agent cyber-attack. The company later admitted it should have responded earlier to the warning signs.
That is not an extinction story. That is an operational security story, and it is the one that touches your business directly.
The relevant question is not whether AI will end the world. It is what an AI tool inside your business is permitted to do. A tool that reads documents and returns text is a different risk object than one that has credentials to your accounting system, can send email as you, or can execute actions without a person in the loop. The first is a document reader. The second is an employee you have not interviewed.
Most construction AI tools today sit firmly in the first category. Some are moving toward the second, and the move is usually described as a feature.
The controls that matter
None of this requires a position on superintelligence. It requires the same discipline you would apply to any vendor with access to your data.
Know what the tool can reach. Read-only access to a document library is not the same as write access to your ERP. Ask specifically, and ask what happens when the vendor ships a feature that expands that scope.
Know where your data goes and whether it trains anything. Your subcontracts, your cost history, and your correspondence are the firm's competitive position. A vendor who cannot answer clearly what happens to that data has answered.
Keep a person on every action with consequences. Reading is safe to automate. Sending, filing, signing, and paying are not, and the distinction should be written into your standard rather than assumed.
Have a kill switch you have tested. If the tool starts producing garbage on a Tuesday, who turns it off, and how long does it take? Firms that cannot answer that have a dependency, not a tool.
And keep your own record. The legacy system stays the system of record. The AI reads what the record cannot. That separation is not nostalgia, it is what lets you unplug something without losing anything.
The researchers may be right about the long arc. That is a serious argument being made by serious people, and it deserves better than a shrug from anyone building on this technology, including us.
It is also not a reason to keep reading subcontracts by hand, and it is not a reason to hand an autonomous agent your bank credentials. The same news should make you more willing to use narrow, bounded tools and less willing to use unbounded ones. Most firms currently do the opposite: they worry in the abstract and then click accept on whatever scope the vendor requested.
Safe for your business is a smaller question than safe for humanity, and unlike the larger one, it has an answer you can implement this quarter.