Three new preprints test Jev as an evaluator in different ways. One finds it close to a strong language-model judge on answer preference and evidence-based fact checks, then uses confidence to route uncertain cases. A second finds little benefit from that routing on graded rubrics because the fallback models often repeat Jev’s confident errors. A third medical benchmark reports similar accuracy on research-abstract questions and sizable gaps on harder diagnosis cases.

The practical answer is narrower than “AI can judge AI.” Jev may be a fast first-pass judge for a well-defined task, but the evidence does not establish one dependable judge for every kind of answer. Confidence can help route work only after a team has checked what it predicts on its own examples.

What it means for a model to judge

An evaluator takes one or more model outputs and returns a score or verdict under some rule. A judge might choose which of two responses better follows a prompt, decide whether a claim is supported by supplied documents, score an answer against a rubric, or select a correct option from a medical multiple-choice case. These tasks place different demands on the evaluator.

Jev is TypeSafe’s hosted decision model. Its API accepts structured input and an allowed response type, then returns a choice or score with probabilities over the permitted labels. The interface can fit a task into software that expects a bounded result. A probability is a model estimate, however; it does not independently verify the underlying answer. TypeSafe’s documentation describes the interface, while the research comparisons below assess performance on specific test sets.

The JEV-as-a-Judge study, revised September 27, compares Jev 1.13 with 16 generative and reward-model judges. The rubric-judge study, posted September 24, compares Jev with three lower-cost language models. The medical benchmark, posted September 27, compares Jev 1.13 with GPT-6 Sol under two reasoning settings. All three are preprints. None is a clinical deployment study, and none has been peer reviewed.

Close on some verdicts, weaker on reasoning

The first paper evaluates 5,172 judgments across preference pairs, evidence-grounded factuality, and final-answer checks, then adds follow-up tests and two held-out routing studies. Jev’s main public benchmark scores were 92.5% on a sampled 1,500 RewardBench preference pairs, 78.6% on 350 JudgeBench pairs, and 87.3% on 3,000 HaluEval answers assessed against evidence. On those tasks, the strongest GPT-6 Astra comparator scored 92.5%, 93.1% and 88.4%, respectively. The paper reports Jev at $0.044 per 1,000 base judgments and 0.15 seconds median latency on its timing panel; GPT-6 Astra was $12.182 and 1.89 seconds under the study’s conditions.

These averages conceal the boundary the authors found. Jev was within roughly two points of GPT-6 on chat quality, safety refusals and evidence-grounded factuality. It fell behind on tasks that required deriving or checking a result: 14.3 points on math and 12.9 on code, with a 27.6-point gap on logic puzzles. On JudgeBench’s objective correctness set, Jev scored 78.6% against 93.1% for GPT-6. Blinded adjudication of disputed cases in a 990-item subset sided with GPT-6 on 57 of 69 disputed JudgeBench items and with Jev on one. The adjudication was selective and limited, but it suggests that this gap was not only a problem with the benchmark’s labels.

Here, “judge” meant choosing between two generated answers or deciding whether a single answer was supported or correct given a stated reference. The authors deliberately gave the generative judges the same constrained verdict-and-probability output format as Jev, without rationales. The comparison is therefore about the verdicts under that contract, not about which judge can explain its reasoning to a human.

Confidence routing works when errors separate

A cascade uses a cheaper first judge and sends some cases to a more expensive fallback. In the first study, the rule accepted Jev’s verdict when its maximum label probability was at least 0.9 and sent lower-confidence cases onward. On 1,610 held-out preference pairs, the two-order cascade escalated 31.5% of cases, scored 93.4% against GPT-6’s 92.5%, and cost 41% of GPT-6’s fee. In a pre-specified live test of 570 held-out pairs from two new correctness workloads, Jev alone trailed GPT-6 by 14 points; the cascade matched GPT-6’s accuracy while escalating 74% and saving about a quarter of its fee.

That result has a specific condition: the cases Jev is unsure about must contain errors that the fallback can correct. The authors found that confidence routing weakened on style-adversarial pairs and failed on reference-free prose, where all tested judges were near chance yet often confident. Their study also found that averaging decisions across both answer orders reduced position effects. Thresholds were selected using local labeled examples, with the paper recommending a lower-confidence-bound approach and a held-out check.

The separate rubric study tests per-criterion scoring on nine panels drawn from seven benchmarks, 5,003 item–criterion pairs in total. Two panels were binary; seven used ordered rating scales. All four judges saw the same criterion text, and the authors fixed the rubrics before evaluation. Eight of 27 paired comparisons had a 95% confidence interval excluding no accuracy difference; four remained after correction for multiple comparisons. Some other comparisons met the paper’s equivalence margin, while many were too imprecise to establish either a difference or equivalence. Jev did best relative to the others on binary criteria. On graded panels it was never the top matched judge, and all judges tended to give lower scores than human raters.

The price and speed savings were large in that setup. Jev cost about six cents for all nine panels; the three language-model judges cost 29 to 325 times more and took 30 to 220 times as long. Yet Jev’s confidence did not make a strong cascade. Confidence ranked its errors on most panels, but the fallback models repeated nearly all of Jev’s most confident mistakes. With cross-fitted thresholds, a cascade improved by at most 1.5 percentage points over the best single judge on average; it performed worse than that judge on six of nine panels. This replay used recorded verdicts rather than a prospective live deployment.

The contrast between the two papers is instructive. In pairwise answer comparisons, the first study found a useful division between Jev’s low-confidence errors and a stronger fallback’s capabilities. In criterion scoring, the fallback often shared the same errors, so escalation added cost without much correction. The model’s confidence signal alone does not tell a team whether another model will disagree productively.

Medical multiple-choice results depend on task difficulty

The medical paper evaluates Jev on four existing datasets: 1,373 MetaMedQA exam questions, 500 PubMedQA questions answered from research abstracts, 915 DiagnosisArena multiple-choice cases, and 34 NEJM Case Challenges with a published final diagnosis. It compared Jev with GPT-6 Sol both with medium reasoning and without reasoning. There were 8,469 model requests, and every request returned a valid structured answer.

On PubMedQA, Jev and GPT-6 Sol with medium reasoning scored 78.4% and 78.2%. On MetaMedQA, Jev scored 74.8% versus 82.7%; on DiagnosisArena, 59.8% versus 82.4%; and on the 34 NEJM cases, 61.8% versus 82.4%. GPT-6 without reasoning also had higher point estimates on the two harder case sets. The 34-case NEJM estimate is imprecise, and its main comparison did not remain statistically significant after correction for multiple comparisons. The much larger DiagnosisArena gap persisted without explicit reasoning.

This is a test of selecting among supplied options, not producing a differential diagnosis or treating patients. The paper reports no clinician adjudication of the benchmark answers. It notes that the NEJM images were provided to models as captions, and that the challenging DiagnosisArena cases were filtered against other language models. Its authors call the findings preliminary and say independent replication, clinical adjudication of reference answers and run-to-run stability checks remain pending. The paper explicitly says not to use the results to guide clinical care.

Probability thresholds had uneven value. On MetaMedQA, 52.9% of Jev’s choices had confidence of at least 0.9, and those were 93.4% accurate. At a similar coverage, GPT-6 was just as accurate or more accurate. On DiagnosisArena, Jev’s probability ranking was weak: at the same 0.9 threshold, only 17.3% of items were accepted, and one in four accepted answers was still wrong. In every benchmark, the models’ average selected-option probability exceeded their accuracy. In this sample, a high score did not amount to certainty.

What teams can take from the studies

Together, the papers support testing Jev as a task-specific evaluator, especially where a verdict can be read from supplied text or checked against evidence. They do not support treating its confidence field as a universal safety switch. Different workloads produced different outcomes, and the rubric study shows why a fallback must be evaluated for error independence as well as raw accuracy.

A team considering this pattern needs a representative, labeled sample from its own task. It should define what counts as a correct decision, include difficult and misleading examples, compare against human review or another defensible reference, then measure both error rates and how well confidence separates correct from incorrect decisions. For a cascade, it should also record whether the fallback actually fixes the first-stage errors, how many cases are escalated, and the resulting cost and delay. A reserved test set can reveal whether the threshold transfers beyond the examples used to choose it.

These are practical implications of the studies, not a tested BIG CHANGE procedure. We did not call the Jev API or run a local benchmark. TypeSafe’s API documentation documents structured requests, allowed answer types and model discovery; it does not independently establish accuracy on a team’s workload. Product access, prices and models can change, so teams should check the current documentation when planning a trial.

For now, Jev looks useful as a candidate first judge when the task is narrow, labels are clear and local evaluation shows that confidence routes errors effectively. When judging requires deeper reasoning, grading depends on hidden rater conventions, or several models make the same mistakes, the studies give teams reason to keep human review or another validated check in the loop.

The big change

Jev’s new evaluations move the question from whether a decision model can return a cheap verdict to when that verdict is useful enough to keep. The answer depends on the task: confidence routing improved a held-out pairwise-evaluation cascade, while a separate rubric study found little gain because errors overlapped. Medical multiple-choice results also varied sharply with task difficulty. For AI teams, the next step is a local validation set that checks correctness, confidence and fallback behavior together.

Sources & further reading