Markdown edition

# OmicsBench tests AI agents across dependent bioinformatics steps

> An author-reported benchmark follows AI agents through dependent bioinformatics tasks. Scores fall on later steps, while feedback improves results under limits.

By BIG CHANGE Editorial

Published: 2026-10-17T16:20:34.838Z
Updated: 2026-10-17T16:20:34.838Z
Canonical: https://bigchange.ai/blog/omicsbench-bioinformatics-agent-workflows

![Conceptual charcoal illustration of an unbranded workstation with abstract dots on its monitor and an external drive beside it.](https://bigchange.ai/api/media/file/omicsbench-bioinformatics-workstation-concept-v1.png)
AI-generated conceptual illustration by BIG CHANGE; the screen is not an OmicsBench interface, research result or measured performance.

OmicsBench asks an AI agent to carry a bioinformatics analysis across successive, dependent requests. In an [October 15 preprint](https://arxiv.org/html/2610.21074v1), Gideon Shaked and Siddharth Nagisetty report that the agents they evaluated lost accuracy as the session progressed and when a step relied on earlier answers. The result puts a more specific question before computational biologists than whether an agent can solve one isolated task: can it keep an analysis faithful to the data and instructions over the course of a study?

## The big change

- **What changed:** OmicsBench grades a sequence of dependent analysis steps in one continuing agent session. It exposes errors that a single final answer can hide, although the evidence here comes from the benchmark authors' own preprint.
- **Why it matters:** A research team judging an agent for multi-step analysis can examine where its results drift, including whether it follows the specified selection criteria. The paper reports lower scores on dependent steps, so a good answer to one prompt is weak evidence for the whole workflow.
- **What to watch:** The public runner and three released tasks let researchers inspect the method. The remaining 94 tasks are private; independent evaluations on held-out work, with analysis quality checked beyond answer matching, would give stronger evidence about practical use.

## How the benchmark follows a study

OmicsBench contains 97 tasks built from published biological studies and their datasets, divided into 662 subtasks. Each task is a graph of steps: an edge means a later answer depends on an earlier one. The median task has seven subtasks, and the median longest dependency chain is three steps. A subtask becomes a user turn in a continuous session. The agent can use the data and its previous files, but it must write a specified answer file for each turn. The authors say human experts curated studies and reviewed the resulting tasks after an agent drafted them. [The paper's methods](https://arxiv.org/html/2610.21074v1) describe that construction and the evaluation procedure.

This design differs from a short evaluation in which each question ends after one response. It also differs from a study-scale prompt that gives an agent the entire objective at once and grades only its final product. Here the sequence makes it possible to observe whether an earlier result is carried correctly into later analysis. One [public example](https://arxiv.org/html/2610.21074v1) asks the agent to identify a kidney cell-state signature and then use that result in subsequent analysis of regulators and biological programs. Those links make the dependencies concrete without establishing that the complete workflow produces a sound new scientific finding.

The score is an average. One or more deterministic graders compare fields in each answer file with an answer key; their mean is the subtask score, and the task score averages its subtasks. The authors built 58 grader types for outputs such as gene sets, enrichment results and numerical values. Some resolve alternative biological identifiers through reference databases, so a valid gene synonym need not fail a literal string comparison. A score therefore measures agreement with the benchmark's defined answers and tolerances, not a full expert judgment of the analysis or its scientific interpretation.

## The results depend on feedback

The authors tested 12 models across three agent harnesses. Each task was run three times per mode in a network-isolated container; the reported mean first averages those runs at the task level. In *single-pass* mode, the agent gets no answer feedback before the next subtask. Mean task scores across model and harness combinations ranged from 0.43 to 0.57 on a zero-to-one scale. In *gated* mode, a score below 0.7 triggers a failure notice and up to two further attempts. Mean scores ranged from 0.54 to 0.67. The best reported combination, Opus 5 on OpenCode, scored 0.57 without feedback and 0.67 with it. These are author-reported benchmark results, not a BIG CHANGE run or an independent replication.

The improvement with retries needs careful interpretation. The authors observed agents guessing through options on some closed-choice subtasks, so they excluded 177 such subtasks from their turn and retry analyses. They say other forms of gaming the feedback are possible and that the share of the gated improvement attributable to this behavior is unquantified. The gated scores describe performance under this particular signal and attempt limit. They should not be read as proof that an agent corrected the underlying analysis.

Across the combinations, second-turn subtask scores were 0.06 to 0.12 below first-turn scores in the paper's single-pass analysis. Subtasks depending on earlier answers scored below independent ones. With gated retries, only 22% to 34% of the subtasks included in that retry analysis reached the 0.7 threshold within three attempts, versus 15% to 24% on the first attempt. Those patterns address the benchmark's main question more directly than a model leaderboard: keeping the chain of work intact remains difficult in these tests.

## What the failure analysis shows

The authors examined failed first attempts on a sample of 150 subtasks. Their automated analysis found cases where agents returned too many results, replaced a requested selection rule with their own, used a different procedure, or answered from prior knowledge instead of the staged data. For a biologist, a returned list may look plausible even when it was selected with the wrong statistical filter. The paper's examples make that failure more useful to inspect than a single aggregate score.

Those failure labels are exploratory. A language-model judge generated descriptions and categories; the authors checked repeated annotation for agreement but did not validate the labels against human annotators. Nor can the limited cross-harness runs isolate how much of a score difference comes from the model and how much from the surrounding agent software. The task collection spans multiple biological domains, but the authors say each domain has too few tasks for reliable within-domain model comparisons.

Researchers can inspect the [author-linked repository](https://huggingface.co/datasets/gideonshaked/OmicsBench), which contains the runner and three public task files with answer keys. The paper says the other 94 tasks are retained as a private test set. That public subset supports examination of how OmicsBench is built; it cannot reproduce the paper's full 97-task ranking on its own. The preprint measures agents against specified analysis answers. It does not establish that they can make reliable scientific discoveries, run unattended in a research lab, or replace computational biologists.

## Sources & further reading

- [Shaked and Nagisetty, “OmicsBench,” arXiv v1, October 15, 2026](https://arxiv.org/html/2610.21074v1). The methods, results, limitations and appendices support the figures and scoring description here. This is an author-reported preprint.
- [OmicsBench public repository](https://huggingface.co/datasets/gideonshaked/OmicsBench). The author-linked release contains the runner and three public task files. The paper's Appendix E explains the held-out set and licenses.

## Sources

- [Shaked and Nagisetty: OmicsBench, arXiv v1](https://arxiv.org/html/2610.21074v1) — Author-reported preprint. Methods, results, limitations and Appendix E describe the 97-task benchmark and its three-task public release; BIG CHANGE has not run it.
- [OmicsBench public repository](https://huggingface.co/datasets/gideonshaked/OmicsBench) — Author-linked runner and three public question files confirmed from direct repository tree listings. The paper says the other 94 tasks remain private.