# Rerank a real source shortlist with Jev and check the evidence

> Score each passage from your existing retriever with one Jev question, keep candidate and source identities intact, then inspect the top original documents before using the ranking.

By BIG CHANGE Editorial

Published: 2026-10-21T21:55:38.220Z
Updated: 2026-10-21T21:55:38.220Z
Canonical: https://bigchange.ai/blog/jev-rerank-real-source-shortlist-evidence-check

![Conceptual charcoal illustration of a filing drawer with blank passage slips, one orange sleeve, and an open source document.](https://bigchange.ai/api/media/file/jev-source-shortlist-archive-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE.

Your retriever has already found a set of possible source passages. Jev can score each passage against the same yes/no question, then your application can reorder that shortlist. You get a traceable ranking for a real query and a short list of original sources to inspect. A high score alone does not establish that a passage is correct.

This guide is for developers who already have a permitted corpus and a working keyword, vector or hybrid retriever. It uses TypeSafe's documented System One API. BIG CHANGE checked the documentation on 21 October 2026; we did not run a retriever, call Jev, observe scores or measure an improvement. [TypeSafe's reranking cookbook](https://docs.typesafe.ai/cookbooks/rerank_typesafe) demonstrates the two-stage approach on its own legal dataset. Our [earlier Jev coverage](https://bigchange.ai/blog/jev-demos-ai-decisions-everyday-software) described a passage-ranking demo. The procedure below applies the documented API to a reader's actual shortlist without borrowing either demo's results.

## The big change

- **What changed:** TypeSafe documents a second step after fast retrieval: Jev judges each shortlisted passage against the same written yes/no criterion. A developer can reorder the passages already found without replacing the retriever. This guide has not measured whether the new order is better.
- **Why it matters:** A developer can show which source passage moved and under which written criterion it was scored, then open that source before passing evidence into a separate answer system. The review matters because a high Noul score is the model's estimate under the chosen question, not proof that the source supports the answer.
- **What to watch:** The reader's held-out results, source revisions and literal response model determine whether the ranking is useful in that application. An omitted relevant passage remains a retrieval problem; moving aliases, context limits and account rate limits affect repeatability and cost.

## 1. Freeze the inputs before spending anything

Save one real query and the ordered passages your existing retriever returned. Each passage needs its exact text, a unique candidate ID, its original rank, a pointer to the original document, and a document revision or retrieval snapshot. Record the retriever name and configuration, query time, access scope and source permissions separately. The passage pointer must let a reviewer open the underlying source, not merely a search snippet.

Stop if the shortlist is empty, IDs repeat, text or source links are missing, permissions do not allow submission to TypeSafe, or source versions have changed since retrieval. Do not create a plausible-looking query or passage to fill the gap. A reranker can only reorder what retrieval supplied; it cannot find an omitted source.

You also need your own TypeSafe account, API key, available credits and an approved spend limit. Keep the key in an environment variable or secret store, and keep it out of the shortlist, logs and shared output. Check the account's current prices and its token-per-second (TPS) and request-per-second (RPS) limits in the [developer console](https://console.typesafe.ai/). Start sequentially and choose a delay that fits **both** limits for your expected pair size; a high-concurrency example in a cookbook is not a safe default for your organization.

## 2. Write one question and check each pair's size

Use one fixed Noul question for the run. Noul is TypeSafe's yes/no output: the returned number from 0 to 1 is its estimate that the answer to **your stated question** is yes. The following criterion is a starting point for a source-backed factual query; edit it to reflect the actual task **before** seeing scores, then use identical wording for every passage:

```json
{
  "type": "noul",
  "instructions": "Does this passage directly support an answer to the query?",
  "criteria": {
    "true": "The passage explicitly supplies the requested fact in its applicable source and version context.",
    "false": "The passage merely shares words or topic, is superseded, or does not establish the requested fact."
  }
}
```

The criterion does not grant a document authority or access rights. Check those separately. A new criterion is a new run; for a comparison, hold the query and candidate set fixed.

For each request, TypeSafe receives `state` containing the query and **one** passage, plus that Noul question. The [current models reference](https://docs.typesafe.ai/models) gives default limits of 64,000 tokens for `state` plus all questions in one request and 32,000 for `state` plus any one question. With one question, the 32,000-token limit is the tighter check. The official API and model references checked here do not document a public token-count endpoint or tokenizer for an exact preflight. Before paying, assemble the final request for each pair and keep its combined query, passage and criterion comfortably below the stated limit; text length is only a conservative screen, never a verified token count. If a pair looks close to the limit or its size is uncertain, revise or segment the *passage or query* with a new candidate ID linked to its source, inspect what context was lost, and freeze a new shortlist. After a successful call, `usage.input_tokens` records that pair's actual usage; it does not certify the remaining pairs. An oversize rejection stops that pair for revision. Do not enable the experimental long-context header for this recipe. Removing other candidates reduces calls but does not shrink an oversized pair.

## 3. Score one passage at a time

The [System One reference](https://docs.typesafe.ai/api/system-one) documents `POST https://api.typesafe.ai/v1/systemone`, bearer authorization, a named question, `answers`, `model` and `usage`. Put the actual query and passage in this request shape. The angle-bracketed fields below are **unfilled inputs**, not an example result:

```json
{
  "model": "jev-latest",
  "state": {
    "query": "<actual permitted query>",
    "candidate_passage": "<exact text of one actual shortlisted passage>"
  },
  "questions": {
    "supports_query": {
      "type": "noul",
      "instructions": "Does this passage directly support an answer to the query?",
      "criteria": {
        "true": "The passage explicitly supplies the requested fact in its applicable source and version context.",
        "false": "The passage merely shares words or topic, is superseded, or does not establish the requested fact."
      }
    }
  }
}
```

In POSIX Bash or zsh (including WSL or Git Bash on Windows), save one filled request file per candidate, such as `pair-ACTUAL_ID.json`, and never overwrite it during the run. Keep the candidate ID, original rank and source pointer in your own run record; they are **not** inferred from the model's answer. Set `TYPESAFE_API_KEY` from your secret store, and set positive numeric `CONNECT_TIMEOUT_SECONDS` and `PAIR_TIMEOUT_SECONDS` values before the call. The total timeout must fit your preset per-pair latency budget and be at least the connect timeout. The following Bash/zsh command sends one paid request and saves its response. Substitute the same actual ID in all per-candidate filenames. Avoid printing the key or body into shared logs.

```shell
curl --silent --show-error --fail-with-body --request POST "https://api.typesafe.ai/v1/systemone" \
  --connect-timeout "$CONNECT_TIMEOUT_SECONDS" \
  --max-time "$PAIR_TIMEOUT_SECONDS" \
  --header "Authorization: Bearer $TYPESAFE_API_KEY" \
  --header "Content-Type: application/json" \
  --data-binary @pair-ACTUAL_ID.json \
  --dump-header headers-ACTUAL_ID.txt \
  --output response-ACTUAL_ID.json \
  --write-out "%{http_code}\n" > status-ACTUAL_ID.txt
printf '%s\n' "$?" > exit-ACTUAL_ID.txt
```

This command relies on Bash/zsh variable expansion and backslash line continuation. It saves the response headers, body, HTTP status and cURL exit in separate local files tied to the actual candidate ID. Inspect the header file for `Retry-After` on a confirmed 429. An HTTP status of `000` or a missing response after a transport failure is unknown, not a rejection receipt. If your cURL version does not support `--fail-with-body`, inspect the HTTP status and body explicitly. A connect or total timeout is a stop condition, **not** proof the paid request was rejected. The [Python SDK](https://docs.typesafe.ai/sdk/python/api/retries) is another route, but its retries must be configured explicitly for a paid run: a connection failure or timeout can leave billing uncertain.

For a confirmed `429 Too Many Requests`, honor `Retry-After` when supplied, or wait with bounded backoff, then retry only within the call, time and spend budgets you set beforehand. A timeout, broken connection or uncertain server error might have been processed; mark that pair **unknown**, check request and billing evidence, and do not blindly resend it. Stop on authentication, invalid-input or depleted-credit errors. Keep the raw HTTP status and failure beside the candidate ID. Any missing or unknown pair makes the run incomplete.

TypeSafe lists `jev-latest` as a moving alias. Save the **literal** `response.model` on every successful call. If it reports only an alias, the exact served version is unknown, even if a documentation page currently maps that alias to a version. For a version-controlled comparison, first verify a documented versioned ID is available to your account, request that ID and require the same ID in the response; otherwise stop the comparison. The official cookbook embeds an older `jev-1.12` setting, which does not establish the version of a new run.

## 4. Validate the responses, then sort the same IDs

Match each response to the candidate ID held outside the response. Require exactly the named `supports_query` answer with `type: "noul"`, a finite numeric `noul` from 0 through 1, a nonempty literal `model`, and numeric `usage.input_tokens` and `usage.output_tokens`. Reject malformed values rather than replacing them with zero. Keep the candidate's original text, source pointer and revision intact. Sort by descending noul; when scores tie, preserve original retrieval rank. Verify that the final IDs are an exact permutation of the frozen IDs, with no duplicates, omissions or substitutions.

Save a before/after table with candidate ID, old and new rank, raw noul, response model, token usage, source revision and link. Sum actual input and output tokens and compare observed cost with the current rate in your account. As checked on 21 October, TypeSafe's [models page](https://docs.typesafe.ai/models) listed Jev at **$0.042 per million input tokens, with output tokens free**. Prices and credits can change; retrieval, storage, retries and any later answer generation are separate costs. Plan a maximum number of calls, worst-case pair sizes and a spend/latency ceiling before the first request, then stop when any ceiling is reached.

For a local check, save the frozen input in `shortlist.json` with top-level `query`, `request_model`, `question` and `candidates`. Set `question` to exactly the inner object at `request.questions.supports_query` above: its `type`, `instructions` and `criteria`, without the outer `supports_query` wrapper. Each candidate needs `id`, `rank` (1 through the shortlist length), `text`, `source`, `source_revision`, `request_file`, `response_file`, `status_file` and `exit_file`. Point the last four fields to the saved files for that exact candidate. Use your real values only. The following standard-library script refuses an incomplete or malformed run, checks each saved request, HTTP status and cURL exit against the frozen input, prints the ranking, and shows a SHA-256 fingerprint of each original passage. Run it only after all response files exist; the source link and revision still need human verification. Save the initial `shortlist.json` securely before making calls and compare it byte for byte afterward, so the script is working from the same frozen shortlist.

```python
import hashlib
import json
import math
from pathlib import Path

data = json.loads(Path("shortlist.json").read_text(encoding="utf-8"))
query = data.get("query")
request_model = data.get("request_model")
question = data.get("question")
rows = data.get("candidates")
if (not isinstance(query, str) or not query.strip()
        or not isinstance(request_model, str) or not request_model.strip()
        or not isinstance(question, dict) or not isinstance(rows, list) or not rows):
    raise ValueError("Query, model, question and shortlist are required")
if (question.get("type") != "noul"
        or not isinstance(question.get("instructions"), str)
        or not isinstance(question.get("criteria"), dict)
        or any(not isinstance(question["criteria"].get(k), str)
               or not question["criteria"][k].strip() for k in ("true", "false"))):
    raise ValueError("One complete Noul question is required")

ids, ranks, files, scored = set(), set(), set(), []
for item in rows:
    if not isinstance(item, dict):
        raise ValueError("Malformed candidate")
    cid, rank = item.get("id"), item.get("rank")
    if not isinstance(cid, str) or not cid or cid in ids:
        raise ValueError("Missing or repeated candidate ID")
    if type(rank) is not int or rank < 1 or rank in ranks:
        raise ValueError("Missing or repeated original rank")
    for field in ("text", "source", "source_revision", "request_file", "response_file", "status_file", "exit_file"):
        if not isinstance(item.get(field), str) or not item[field].strip():
            raise ValueError(f"{cid}: missing {field}")
    for field in ("request_file", "response_file", "status_file", "exit_file"):
        if item[field] in files:
            raise ValueError(f"{cid}: request/response file reused")
        files.add(item[field])
    ids.add(cid)
    ranks.add(rank)
if ranks != set(range(1, len(rows) + 1)):
    raise ValueError("Original ranks must be 1 through shortlist length")

for item in rows:
    cid = item["id"]
    status = Path(item["status_file"]).read_text(encoding="utf-8").strip()
    exit_code = Path(item["exit_file"]).read_text(encoding="utf-8").strip()
    if status != "200" or exit_code != "0":
        raise ValueError(f"{cid}: HTTP {status or 'missing'}, cURL exit {exit_code or 'missing'}")
    request = json.loads(Path(item["request_file"]).read_text(encoding="utf-8"))
    if (not isinstance(request, dict) or request.get("model") != request_model
            or request.get("state") != {"query": query, "candidate_passage": item["text"]}
            or request.get("questions") != {"supports_query": question}):
        raise ValueError(f"{cid}: saved request differs from frozen input")
    path = Path(item["response_file"])
    response = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(response, dict):
        raise ValueError(f"{cid}: malformed response")
    answers = response.get("answers")
    answer = answers.get("supports_query") if isinstance(answers, dict) else None
    if not isinstance(answer, dict) or set(answers) != {"supports_query"} or answer.get("type") != "noul":
        raise ValueError(f"{cid}: missing named Noul answer")
    score = answer.get("noul")
    if type(score) not in (int, float) or not math.isfinite(score) or not 0 <= score <= 1:
        raise ValueError(f"{cid}: invalid Noul value")
    model = response.get("model")
    usage = response.get("usage")
    if not isinstance(model, str) or not model.strip() or not isinstance(usage, dict):
        raise ValueError(f"{cid}: missing model or usage")
    if request_model.startswith("jev-") and request_model not in ("jev-latest", "jev-preview") and model != request_model:
        raise ValueError(f"{cid}: response model differs from requested version")
    for field in ("input_tokens", "output_tokens"):
        if type(usage.get(field)) is not int or usage[field] < 0:
            raise ValueError(f"{cid}: invalid {field}")
    digest = hashlib.sha256(item["text"].encode("utf-8")).hexdigest()
    scored.append((item, score, model, usage, digest))

versioned = {row[2] for row in scored if row[2] not in ("jev-latest", "jev-preview")}
if len(versioned) > 1:
    raise ValueError("Different literal model versions in one ranking")
scored.sort(key=lambda row: (-row[1], row[0]["rank"]))
if len(scored) != len(rows) or {row[0]["id"] for row in scored} != ids:
    raise ValueError("Candidate ID permutation failed")
for new_rank, (item, score, model, usage, digest) in enumerate(scored, 1):
    print(json.dumps({"id": item["id"], "old_rank": item["rank"],
        "new_rank": new_rank, "noul": score, "response_model": model,
        "input_tokens": usage["input_tokens"], "output_tokens": usage["output_tokens"],
        "source": item["source"], "source_revision": item["source_revision"],
        "passage_sha256": digest}, ensure_ascii=False))
print("total_input_tokens", sum(row[3]["input_tokens"] for row in scored))
print("total_output_tokens", sum(row[3]["output_tokens"] for row in scored))
```

Save the block as `check.py`, then run `python check.py > ranking.txt` in the directory containing the files. Record the saved shortlist's original checksum, the request and response filenames, the HTTP status for each call, and the human source checks with `ranking.txt`. If any check fails, keep the original shortlist and the failure record. Do not display the output as a completed reranking until the response and source issue is resolved. The script prints no passage text, but source pointers and query material may still be sensitive; protect the local files and output accordingly.

## 5. Open the evidence, not just the ranking

Open the original documents for the highest-ranked passages. Check the exact cited text, document revision, access rights and whether each passage really supports the query under the written criterion. Look at a high-scoring false match and a known relevant passage that failed to rise if either exists. Record what you found rather than treating Noul as calibrated factual truth or as verification of a generated answer.

For a meaningful local evaluation, repeat the frozen process on several **real held-out queries** with known relevant passage IDs. Report initial retrieval coverage, top-1 and top-K placement before and after reranking, failures, elapsed time and total token usage. If a relevant document never entered the original shortlist, that is a retrieval gap. Any threshold, changed criterion or cross-provider comparison is a future evaluation plan; no gain, cutoff or latency advantage is implied by this documentation-led guide.

TypeSafe [says](https://help.typesafe.ai/articles/5974925716-how-is-submitted-data-used-or-retained) it does not train Jev on customer requests or responses, while submitted data may be processed for service, billing, abuse prevention and telemetry; retention depends on agreements and any zero-retention arrangement. Review the current terms and your disclosure authority before sending private material. This guide ends at source-linked reranking and evidence inspection. It does not implement retrieval, generate an answer or validate citations in one.

## Sources

- [TypeSafe reranking cookbook](https://docs.typesafe.ai/cookbooks/rerank_typesafe) — Documents shortlist then query/passage Noul reranking; its CLERC result and older embedded version are not reader results.
- [System One API reference](https://docs.typesafe.ai/api/system-one) — POST endpoint, bearer auth, state, questions and response model/answer/usage fields.
- [Noul documentation](https://docs.typesafe.ai/primitives/noul) — Yes/no 0–1 output and true/false criteria; no independent factual verification implied.
- [Models, context and rate limits](https://docs.typesafe.ai/models) — Versioned IDs, moving aliases, 64k/32k default context, account TPS/RPS and date-stamped listed input price.
- [Python SDK retry policy](https://docs.typesafe.ai/sdk/python/api/retries) — Retry behavior and configurable timeout/connection/status handling; article uses a manual HTTP call.
- [TypeSafe data-handling statement](https://help.typesafe.ai/articles/5974925716-how-is-submitted-data-used-or-retained) — Vendor account of training, service processing and retention dependencies; reader must check terms.