# SecurityScorecard’s benchmark tests AI on vendor-risk decisions

> A vendor-risk benchmark evaluates AI task performance and resistance to manipulation.

By BIGCHANGE.AI Editorial

Published: 2026-10-29T15:16:38.398Z
Updated: 2026-10-29T15:16:38.398Z
Canonical: https://bigchange.ai/blog/securityscorecard-tprm-benchmark-ai-vendor-risk

![Conceptual illustration of an analyst using a mouse while viewing generic access rows on a monitor, with a security key on the desk.](https://bigchange.ai/api/media/file/tprm-v2.png)
AI-generated conceptual illustration by BIG CHANGE.

SecurityScorecard has introduced a benchmark for evaluating AI on third-party risk management: the work of assessing security risks associated with suppliers. Its undated [benchmark page](https://securityscorecard.com/tprm-benchmark/) describes risk-analysis tasks and adversarial tests.

The distinction matters for a security team considering an AI assistant. A model can identify an answer in a controlled investigation without showing that it will reliably make the right decision with an organization’s changing data, permissions and risk appetite. The published material describes a research instrument; it does not establish a safe level of autonomy for a particular deployment.

## The big change

- The tasks turn risk decisions into questions with verifiable answers.
- It separates task performance from resistance to manipulation. That matters when an AI system reads supplier-supplied material: the content can be relevant evidence while also containing instructions or claims that should not determine the answer.
- A score belongs to its task set and run conditions. Before using it to select or oversee a tool, a team still needs evidence that the result holds for its own data, workflow and controls. A leaderboard alone cannot supply that assurance.

## What the tasks ask an AI to do

One task asks for the known-exploited vulnerability with the highest EPSS score in frozen findings, and grades the exact CVE identifier. 

A correct answer follows the test rule, but real decisions require local context.

That distinction follows from the measures themselves. [CISA’s known-exploited vulnerabilities guidance](https://www.cisa.gov/sites/default/files/publications/Reducing_the_Significant_Risk_of_Known_Exploited_Vulnerabilities_20211103.pdf) identifies exploitation evidence as a reason to prioritize remediation. [FIRST’s EPSS documentation](https://www.first.org/epss/faq) explains that EPSS estimates the probability of observed exploitation activity over the next 30 days, not the impact or exposure in a particular environment. 

## Why the attack tests are separate

Separate tests probe prompt injection and other attacks. [NIST’s adversarial machine learning taxonomy](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf) explains that instructions embedded in untrusted material can interfere with an application’s intended behavior. Testing a model’s response to such pressure can reveal a failure that a clean task does not exercise. Equally, a model that resists a particular test has only demonstrated resistance under that test’s conditions.

The [research site](https://tprm-bench.driftnet.io/) describes a frozen synthetic portfolio, while repeatability and external validation remain open research. A shared dataset enables comparison, but relevance to a particular organization remains unproven.

The main page describes 120-plus tasks per model; the research site describes 818 in release 2026.14. Their counts have different scopes. BIG CHANGE reviewed the published descriptions but has not run the benchmark or independently verified its results.

## Sources

- [SecurityScorecard: TPRM Benchmark](https://securityscorecard.com/tprm-benchmark/) — The operator describes tasks and grading; results lack independent validation.
- [TPRM Benchmark research site](https://tprm-bench.driftnet.io/) — The research site describes release 2026.14, a synthetic portfolio and open validation questions.
- [CISA: Reducing the Significant Risk of Known Exploited Vulnerabilities](https://www.cisa.gov/sites/default/files/publications/Reducing_the_Significant_Risk_of_Known_Exploited_Vulnerabilities_20211103.pdf) — Explains CISA’s rationale for prioritizing known exploited vulnerabilities; it does not evaluate this benchmark.
- [FIRST: EPSS Frequently Asked Questions](https://www.first.org/epss/faq) — Explains the 30-day exploitation estimate and why it is not a complete environment-specific risk score; it does not evaluate this benchmark.
- [NIST: Adversarial Machine Learning taxonomy](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf) — Describes prompt injection and untrusted input; it does not evaluate the benchmark.