Microsoft's model card for MAI-Cyber-1.1-Flash describes Casaba's evaluation under "Safety evaluation and red-teaming".
The model
The card's summary calls MAI-Cyber-1.1-Flash "an efficient 5B parameter cybersecurity-specialized in-house model" and "a cyber permissive model that has been designed to deepen cybersecurity capability to support vulnerability discovery, red teaming, and pen testing." The card says access is "restricted to select Microsoft codename MDASH customers".
How Microsoft describes the evaluation
"Microsoft engaged Casaba Security, an independent third-party firm, to evaluate the cyber capability of MAI-Cyber-1.1-Flash across harmful request containment, adversarial jailbreak techniques, XPIA, secure code generation, and public cyber benchmarks."
Microsoft, MAI-Cyber-1.1-Flash model card
Five areas
Microsoft's card names five areas the evaluation covered.
- 1
Harmful request containment
- 2
Adversarial jailbreak techniques
- 3
XPIA
- 4
Secure code generation
- 5
Public cyber benchmarks
What we found
"This testing identified that the model contained 90.2% of reviewed generally harmful requests; the model had the most failures with benign requests and in low resource languages."
Microsoft, MAI-Cyber-1.1-Flash model card
An evaluation is not a certification, and Casaba did not certify this model as safe or secure. We tested and reported. Microsoft published the result.