SecurityScorecard has introduced a benchmark for evaluating AI on third-party risk management: the work of assessing security risks associated with suppliers. Its undated benchmark page describes risk-analysis tasks and adversarial tests.

The distinction matters for a security team considering an AI assistant. A model can identify an answer in a controlled investigation without showing that it will reliably make the right decision with an organization’s changing data, permissions and risk appetite. The published material describes a research instrument; it does not establish a safe level of autonomy for a particular deployment.

The big change

  • The tasks turn risk decisions into questions with verifiable answers.
  • It separates task performance from resistance to manipulation. That matters when an AI system reads supplier-supplied material: the content can be relevant evidence while also containing instructions or claims that should not determine the answer.
  • A score belongs to its task set and run conditions. Before using it to select or oversee a tool, a team still needs evidence that the result holds for its own data, workflow and controls. A leaderboard alone cannot supply that assurance.

What the tasks ask an AI to do

One task asks for the known-exploited vulnerability with the highest EPSS score in frozen findings, and grades the exact CVE identifier.

A correct answer follows the test rule, but real decisions require local context.

That distinction follows from the measures themselves. CISA’s known-exploited vulnerabilities guidance identifies exploitation evidence as a reason to prioritize remediation. FIRST’s EPSS documentation explains that EPSS estimates the probability of observed exploitation activity over the next 30 days, not the impact or exposure in a particular environment.

Why the attack tests are separate

Separate tests probe prompt injection and other attacks. NIST’s adversarial machine learning taxonomy explains that instructions embedded in untrusted material can interfere with an application’s intended behavior. Testing a model’s response to such pressure can reveal a failure that a clean task does not exercise. Equally, a model that resists a particular test has only demonstrated resistance under that test’s conditions.

The research site describes a frozen synthetic portfolio, while repeatability and external validation remain open research. A shared dataset enables comparison, but relevance to a particular organization remains unproven.

The main page describes 120-plus tasks per model; the research site describes 818 in release 2026.14. Their counts have different scopes. BIG CHANGE reviewed the published descriptions but has not run the benchmark or independently verified its results.