THE WORLD IS NOT STANDING STILL.RSS
BIG CHANGE.
BIG CHANGE, one tap away
BIG CHANGE, one tap away

Make room for the big picture.

Add BIG CHANGE to your home screen or desktop. It opens in its own window, with the latest stories when you’re online.

iPhone & iPad
Open bigchange.ai in Safari. Tap Share, then Add to Home Screen. Turn on Open as Web App if shown, then tap Add.
Android
Open bigchange.ai in Chrome. In the browser menu, choose Install app or Add to Home screen.
Desktop
In Chrome or Edge, use the install icon in the address bar or the browser’s app menu. In Safari on Mac, choose File → Add to Dock.

Markdown edition

# SecurityScorecard’s benchmark tests AI on vendor-risk decisions

> A vendor-risk benchmark evaluates AI task performance and resistance to manipulation.

By BIGCHANGE.AI Editorial

Published: 2026-10-29T15:16:38.398Z
Updated: 2026-10-29T15:16:38.398Z
Canonical: https://bigchange.ai/blog/securityscorecard-tprm-benchmark-ai-vendor-risk

![Conceptual illustration of an analyst using a mouse while viewing generic access rows on a monitor, with a security key on the desk.](https://bigchange.ai/api/media/file/tprm-v2.png)
AI-generated conceptual illustration by BIG CHANGE.

SecurityScorecard has introduced a benchmark for evaluating AI on third-party risk management: the work of assessing security risks associated with suppliers. Its undated [benchmark page](https://securityscorecard.com/tprm-benchmark/) describes risk-analysis tasks and adversarial tests.

The distinction matters for a security team considering an AI assistant. A model can identify an answer in a controlled investigation without showing that it will reliably make the right decision with an organization’s changing data, permissions and risk appetite. The published material describes a research instrument; it does not establish a safe level of autonomy for a particular deployment.

## The big change

- The tasks turn risk decisions into questions with verifiable answers.
- It separates task performance from resistance to manipulation. That matters when an AI system reads supplier-supplied material: the content can be relevant evidence while also containing instructions or claims that should not determine the answer.
- A score belongs to its task set and run conditions. Before using it to select or oversee a tool, a team still needs evidence that the result holds for its own data, workflow and controls. A leaderboard alone cannot supply that assurance.

## What the tasks ask an AI to do

One task asks for the known-exploited vulnerability with the highest EPSS score in frozen findings, and grades the exact CVE identifier. 

A correct answer follows the test rule, but real decisions require local context.

That distinction follows from the measures themselves. [CISA’s known-exploited vulnerabilities guidance](https://www.cisa.gov/sites/default/files/publications/Reducing_the_Significant_Risk_of_Known_Exploited_Vulnerabilities_20211103.pdf) identifies exploitation evidence as a reason to prioritize remediation. [FIRST’s EPSS documentation](https://www.first.org/epss/faq) explains that EPSS estimates the probability of observed exploitation activity over the next 30 days, not the impact or exposure in a particular environment. 

## Why the attack tests are separate

Separate tests probe prompt injection and other attacks. [NIST’s adversarial machine learning taxonomy](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf) explains that instructions embedded in untrusted material can interfere with an application’s intended behavior. Testing a model’s response to such pressure can reveal a failure that a clean task does not exercise. Equally, a model that resists a particular test has only demonstrated resistance under that test’s conditions.

The [research site](https://tprm-bench.driftnet.io/) describes a frozen synthetic portfolio, while repeatability and external validation remain open research. A shared dataset enables comparison, but relevance to a particular organization remains unproven.

The main page describes 120-plus tasks per model; the research site describes 818 in release 2026.14. Their counts have different scopes. BIG CHANGE reviewed the published descriptions but has not run the benchmark or independently verified its results.

## Sources

- [SecurityScorecard: TPRM Benchmark](https://securityscorecard.com/tprm-benchmark/) — The operator describes tasks and grading; results lack independent validation.
- [TPRM Benchmark research site](https://tprm-bench.driftnet.io/) — The research site describes release 2026.14, a synthetic portfolio and open validation questions.
- [CISA: Reducing the Significant Risk of Known Exploited Vulnerabilities](https://www.cisa.gov/sites/default/files/publications/Reducing_the_Significant_Risk_of_Known_Exploited_Vulnerabilities_20211103.pdf) — Explains CISA’s rationale for prioritizing known exploited vulnerabilities; it does not evaluate this benchmark.
- [FIRST: EPSS Frequently Asked Questions](https://www.first.org/epss/faq) — Explains the 30-day exploitation estimate and why it is not a complete environment-specific risk score; it does not evaluate this benchmark.
- [NIST: Adversarial Machine Learning taxonomy](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf) — Describes prompt injection and untrusted input; it does not evaluate the benchmark.