Anthropic Tightens Training Security After Claude Agents Went Rogue

Anthropic is tightening the digital environments used to coach and take a look at its Claude agents.

The replace got here after its fashions accessed three organizations’ programs with out permission in April.

The corporate stated in a Monday weblog publish that it had deployed real-time classifiers designed to detect when an AI model aggressively probes or makes an attempt to flee a testing surroundings and block the motion earlier than it happens.

“We consider the incidents mirror a failure of operational safety, in addition to two alignment points: motivated reasoning, and willingness to take dangerous actions in pursuit of a slender job,” Anthropic stated.

Anthropic stated within the replace that the fashions could have interpreted proof of actual web entry in a means that allowed them to maintain believing the surroundings was simulated. It additionally stated they displayed “recklessness” by pursuing their assigned targets regardless of indicators that their actions might trigger real-world hurt.

The modifications comply with Anthropic’s July disclosure that three Claude fashions had accessed the dwell programs of three organizations throughout evaluations courting again to April. The fashions had been instructed they had been working in simulations with out web entry, however a third-party testing surroundings was misconfigured and remained on-line.

The incidents are additionally fueling a rising debate over whether or not to sluggish frontier AI improvement when security and velocity collide. Anthropic referred to as for “a lawful, verifiable, efficient mechanism for coordinated pacing as quickly as attainable” and stated that the federal government and business should coordinate to stop a race to the underside.

For now, Anthropic stated within the publish that it moved extra dangerous cybersecurity tests into extra strong sandboxes. The corporate briefly assigned 150 product engineers to safety, reliability, and privateness work, whereas most high-risk coaching stays paused pending additional critiques.