Evaluating Microsoft's MAI-Cyber-1.1-Flash for cyber capability

Microsoft engaged Casaba Security to evaluate the cyber capability of MAI-Cyber-1.1-Flash. Here is what Microsoft's model card says.

Microsoft's model card for MAI-Cyber-1.1-Flash describes Casaba's evaluation under "Safety evaluation and red-teaming".

The model

The card's summary calls MAI-Cyber-1.1-Flash "an efficient 5B parameter cybersecurity-specialized in-house model" and "a cyber permissive model that has been designed to deepen cybersecurity capability to support vulnerability discovery, red teaming, and pen testing." The card says access is "restricted to select Microsoft codename MDASH customers".

How Microsoft describes the evaluation

"Microsoft engaged Casaba Security, an independent third-party firm, to evaluate the cyber capability of MAI-Cyber-1.1-Flash across harmful request containment, adversarial jailbreak techniques, XPIA, secure code generation, and public cyber benchmarks."

Microsoft, MAI-Cyber-1.1-Flash model card

Five areas

Microsoft's card names five areas the evaluation covered.

  1. 1

    Harmful request containment

  2. 2

    Adversarial jailbreak techniques

  3. 3

    XPIA

  4. 4

    Secure code generation

  5. 5

    Public cyber benchmarks

What we found

"This testing identified that the model contained 90.2% of reviewed generally harmful requests; the model had the most failures with benign requests and in low resource languages."

Microsoft, MAI-Cyber-1.1-Flash model card

An evaluation is not a certification, and Casaba did not certify this model as safe or secure. We tested and reported. Microsoft published the result.

Need an independent evaluation of an AI model?

We evaluate AI models and systems for cyber risk, and we report what we find.

Get in touch