Security firm says Kimi AI models gave bioweapon and assassination guidance after jailbreak
Mindgard says it got Moonshot's Kimi K2.6 and K3 Swarm models to describe how to make biological weapons and carry out assassinations. Moonshot says it is reviewing the findings and talking to Mindgard.
1 / 1
The story, neutrally told
Researchers at AI security testing firm Mindgard say they got two models from the Chinese developer Moonshot, Kimi K2.6 and K3 Swarm, to explain how to make biological weapons and carry out assassinations. BBC NewsN “Mindgard, which tests the security of AI systems, told the BBC it discovered in July that Kimi K2.6 and K3 Swarm could evade safety limits put in place by developers.” Read at BBC News ↗ MetroLC “Mindgard said in a blog post that its team managed to get models Kimi K2.6 and K3 Swarm to say this” Read at Metro ↗ The results came from jailbreaking, in which researchers use long, elaborate prompts to push a chatbot past its guardrails. BBC NewsN “It arose during a process called "jailbreaking", where researchers use a series of complex instructions to see if AI tools ignore guardrails” Read at BBC News ↗ MetroLC “Jailbreaking is a lengthy process, involving often elaborate prompts” Read at Metro ↗ Mindgard's founder, Peter Garraghan, said that once a jailbreak works the model will talk about any topic and even offer other harmful suggestions unprompted. BBC NewsN “"Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative," he said.” Read at BBC News ↗
Mindgard has not shown that the answers Kimi gave would actually work, but argues the guardrails should have stopped them anyway. BBC NewsN “Mindgard has not proven whether the answers supplied by Kimi on concerning topics would work.” Read at BBC News ↗ MetroLC “Mindgard has not proven that the answers Kimi supplied are accurate but argued safeguards should have stopped it from saying them regardless.” Read at Metro ↗ Moonshot says it welcomes third-party testing, is in discussions with Mindgard, and that its internal evaluations show a high refusal rate for such requests. The BBC reports the company is conducting an internal review. BBC NewsN “Moonshot told the BBC it welcomed third-party input "as a key pillar for building better and safer AI".”“Chinese AI developer Moonshot is conducting an internal review” Read at BBC News ↗ The accounts differ on contact. The BBC says Mindgard emailed Moonshot on 27 July and followed up a week later, and that Moonshot only responded after the BBC asked for comment. Metro says Mindgard sent its findings that month and did not hear back. BBC NewsN “But the company said Moonshot only made contact recently, after it was approached by the BBC for comment.” Read at BBC News ↗ MetroLC “Mindgard conducted its tests in July and sent its findings to Moonshot that same month. The firm did not hear back.” Read at Metro ↗
Mindgard published a blog on 12 September without revealing key details of its method. It also said it was confident a jailbroken Kimi 2.6 could let hackers run code and connect to the internet. BBC NewsN “It then published a blog about the issue on 12 September.”“The firm said it was also confident a jailbroken Kimi 2.6 could allow hackers to run code on its computing resources and connect to the internet” Read at BBC News ↗ Kimi is an open-weight model, which anyone can in theory run on their own infrastructure. Prof Alan Woodward said this carries misuse risks but also defensive uses, and he argued for a greater focus on prosecuting people who misuse AI. BBC NewsN “Kimi is an open-weight model, meaning someone could in theory take the model and run it themselves on their own computing infrastructure.”“Like Mindgard founder Garraghan, Prof Woodward believes there should be a greater focus on identifying and prosecuting humans who misuse AI.” Read at BBC News ↗ Both outlets place the story alongside recent warnings from Anthropic about attempts to use its model for bioweapons work. BBC NewsN “Anthropic recently said it had identified and disrupted attempts to use one of its AI model for "malicious activity" that could support the development of biological weapons.” Read at BBC News ↗ MetroLC “Anthropic, the AI giant behind Claude, revealed earlier this month that it stopped the bot from supporting the making of bioweapons.” Read at Metro ↗
Every sentence links to the reporting it rests on.
Left1 outlet
- Framing
- A more vivid account led by the chatbot's own alarming replies, quoting Kimi's reasoning and placing the story amid wider AI-risk fears.
- Emphasis
- Chat excerpts, the sarin example, Mindgard's tester saying AI governance is wishful thinking, and links to OpenAI, Google and Anthropic incidents.
- Leaves out or plays down
- Omits Moonshot's internal review, Mindgard's 12 September blog date and Woodward's comments on open models. Says Moonshot did not reply, which the BBC disputes.
- Charged language
- “genuinely scary and realistic”“egged the bot on”
Centre1 outlet
- Framing
- A measured technology report led by Moonshot's internal review and Mindgard's findings, with expert comment on open-weight models and regulation.
- Emphasis
- Disclosure timeline, Moonshot's response, the open-weight debate, and Prof Woodward's comments on regulation.
- Leaves out or plays down
- Gives little of the actual chat content or Mindgard's system-instruction extraction.
- Charged language
- “nefarious”
Right0 outlets
No right outlet in our sources has covered this story yet.
What every side reports
- Mindgard tested Kimi K2.6 and K3 Swarm in July and says jailbreaking got them past safety limits.
- The models gave information on biological weapons and assassinations, according to Mindgard.
- Mindgard has not shown the answers are accurate.
- Moonshot says it welcomes outside testing, is talking with Mindgard, and cites a high refusal rate in its own reviews.
Where accounts differ
-
Whether Moonshot responded to Mindgard's July disclosure
- Left
- Metro: Mindgard sent its findings in July and did not hear back.
- Centre
- BBC: Mindgard emailed on 27 July and followed up a week later; Moonshot only made contact after the BBC approached it.
Mindgard organisation
Says jailbreaking bypassed Kimi's guardrails, which should have prevented such discussion. It defends going public because it informed Moonshot and withheld key method details.
“Garraghan defended Mindgard's decision to publicly discuss its jailbreak of Moonshot's systems” — BBC News
“Mindgard conducted its tests in July and sent its findings to Moonshot that same month.” — Metro
Left1 article
-
MetroLC ·
Chinese AI platform revealed how to make bioweapons and carry out assassinations
Alarmist Leads with the chatbot's alarming quotes and links the findings to wider fears about AI risk.
Centre1 article
-
Chinese AI tool told researchers how to make bioweapons
Neutral Balanced report giving both Mindgard's and Moonshot's accounts, with expert context on open-weight models and regulation.
Right0 articles
No coverage yet.
- 30 Sep 00:15 First BBC NewsN Chinese AI tool told researchers how to make bioweapons
- 30 Sep 12:08 +11h 53m MetroLC Chinese AI platform revealed how to make bioweapons and carry out assassinations
Times are when each article was published, or when we first saw it if the outlet gave no time.