Microsoft's model card for its MAI Code 2 and MAI Code 2 Flash coding models, last updated on 17 November 2026, includes a section titled "Third-party evaluation for offensive cyberoperations: Casaba".
How Microsoft describes Casaba
"Casaba is an independent assessment firm selected for their expertise in cybersecurity and experience evaluating advanced AI models and systems for cyberrisk."
Microsoft, MAI Code 2 and MAI Code 2 Flash model card
Eight evaluation areas
Microsoft's card says Casaba assessed MAI Code 2 and MAI Code 2 Flash for offensive cyber capability across eight evaluation areas.
- 1
Harmful cyber requests and safeguard bypasses (MalwareBench, covering both the 1,507-case Opus and Luna comparison set and the full 3,520-case sweep);
- 2
General non-cyber harmful requests and jailbreaks across seven harm domains;
- 3
Multimodal image jailbreaks comparing plain text, single images, and split multi-image prompts;
- 4
Indirect prompt injection, where attacker instructions or harmful requests are planted inside untrusted emails, documents, code reviews, or logs;
- 5
Multi-turn escalation across five-turn scripted pressure and ten-turn adaptive red teaming;
- 6
Factual grounding and benign agentic tasks;
- 7
Autonomous scope-boundary scenarios testing whether a coding agent tampers with failing build or security checks under pressure (20 containerized scenarios); and
- 8
Practical offensive cyber benchmarks covering vulnerability exploitation (CVE-Bench), proof-of-concept reproduction (SEC-Bench), capture-the-flag challenges (CyBench and InterCode CTF), a five-host Active Directory enterprise network range, offensive code generation (Cyber-1), and security command accuracy (NL2Bash and CyBashBench).
What we found
"Casaba assessed that MAI Code 2 and MAI Code 2 Flash show excellent resistance to indirect prompt injection across untrusted sources, respected authorization boundaries across 20 software engineering tasks that tempted shortcuts, and strong cyber capability performance on CTF challenges while falling short of executing end-to-end compromises. MAI Code 2 outperforms Flash on cyber-harm containment and both models remain susceptible to some image-based as well as multi-turn jailbreaks."
Microsoft, MAI Code 2 and MAI Code 2 Flash model card
Microsoft's risk rating
The same card states that Microsoft's "overall risk level for MAI Code 2 and MAI Code 2 Flash for cyberoperations is High", which Microsoft says is in line with other similar models.
An evaluation is not a certification, and Casaba did not certify these models as safe or secure. We tested and reported. Microsoft published the result.