# OpenAI maps how pilot scientists use ChatGPT
> Logs describe research use, while survey answers record expectations about discovery.
By BIGCHANGE.AI Editorial
Published: 2026-11-20T01:52:38.001Z
Updated: 2026-11-20T01:52:38.001Z
Canonical: https://bigchange.ai/blog/openai-scientists-chatgpt-pilot-study

AI-generated conceptual illustration for BIG CHANGE.
OpenAI released Science at the Frontier on November 16. The report combines usage records with a survey about AI and scientific progress. [OpenAI report](https://openai.com/index/science-at-the-frontier/)
## The big change
- Reading and writing account for over half of the average participant's scientific messages. The categories describe requests for help. They do not establish the scientific correctness of answers or whether discoveries followed.
- A separate experiment asked people to implement either human or AI-generated ideas. Reviewers then assessed the executed projects, exposing a gap between promising proposals and evidence from the work needed to carry them out.
- A research team still needs an evidence trail from an idea to a result. National Academies guidance distinguishes rerunning an analysis from collecting new evidence, which answer different questions about a finding.
## What the usage records capture
The logs cover 1,819 summer-pilot participants from August 20 to September 22 and include ChatGPT and Codex. Workflow shares average each scientist's distribution of scientific messages, equally weighted. Privacy-preserving classifiers labelled requests without researchers reading underlying messages or responses. Classification used the request and, where needed, the preceding exchange. [OpenAI report](https://openai.com/index/science-at-the-frontier/)
Of the survey's 251 scientists, 250 answered a question about a hypothetical discovery taking ten years without AI. Of those respondents, 98% expected AI to shorten it. These are expectations, rather than observed discovery times. Respondents had applied for expanded AI access and chosen to answer, limiting generalization. [OpenAI report](https://openai.com/index/science-at-the-frontier/)
## Following an idea through execution
Chenglei Si and colleagues examined what happened when proposed research was implemented. In their June 2025 arXiv paper, 43 completed projects covered 19 human ideas and 24 from a Claude 3.5 Sonnet-based agent. Participants had three months to implement their assigned idea. Fifty-eight experts reviewed the resulting papers and code, blinded to the idea's source. [Execution study](https://arxiv.org/abs/2506.20803)
The AI ideas had initially received higher proposal ratings. After execution, their ratings dropped more than the human ideas' ratings on novelty, excitement, effectiveness and overall quality. But when reviews were averaged per project, the final score differences between human and AI projects were not statistically significant. [Execution study](https://arxiv.org/abs/2506.20803)
The experiment was limited to prompting techniques in natural language processing, with a modest sample of completed projects. That setting limits how far the findings extend across models, research methods and scientific fields. [Execution study](https://arxiv.org/abs/2506.20803)
## Make the result inspectable
The National Academies' 2019 consensus report Reproducibility and Replicability in Science offers useful distinctions for a team evaluating research work. Computational reproducibility asks whether the same data, code and analysis conditions yield consistent results. Replicability asks whether studies collecting their own data address the same question consistently. They are different checks on the evidence supporting a finding. [National Academies report](https://www.nationalacademies.org/read/25303/chapter/7)
Consider a hypothetical physics project analyzing pendulum timings. Rerunning its saved analysis on the same recorded timings would examine computational reproducibility. Repeating measurements and assessing whether they support the same conclusion would address replication. The distinction makes clear which evidence a later researcher needs to perform each check. [Report definitions](https://nap.nationalacademies.org/resource/25303/R%26R%20for%20SBS3.pdf)
The report also explains why reproducibility alone does not guarantee correctness. A second person can reproduce an error in shared code. Making the underlying materials inspectable gives others a chance to examine the analysis and identify that mistake. [National Academies report](https://www.nationalacademies.org/read/25303/chapter/7)
BIG CHANGE reviewed the public papers. We did not inspect the private pilot messages or reproduce the experiments.
## Sources
- [OpenAI, Science at the Frontier](https://openai.com/index/science-at-the-frontier/) — Pilot logs and survey.
- [Si et al., The Ideation–Execution Gap](https://arxiv.org/abs/2506.20803) — June 2025 version and execution limits.
- [National Academies, Reproducibility and Replicability in Science](https://www.nationalacademies.org/read/25303/chapter/7) — 2019 definitions and evidence requirements.
The BIG CHANGE newsletter
The big picture. At your pace.
Recent stories on AI and robotics, the shifts worth watching and practical ideas to use. Choose a daily briefing, weekly digest or monthly perspective.
Next scheduled send (UTC): . Your first edition arrives at the next scheduled send after you confirm.
Your privacy, your choice.
Necessary storage supports site security and remembers your choices. Optional Google Analytics stays off until you allow it. You can read every story with necessary storage only. Privacy details