NORTHWESTERN UNIVERSITY

MedicalHarness

A Controlled Evaluation of LLMs and
Agent Harnesses on Medical Tasks

Ziqing Wang · Lili Zhao · Kaize Ding

Northwestern University

Read the paper Code & data Explore results

The model is one part of an agent. We study how the harness changes its performance across four kinds of medical work—and which mechanisms make a difference.

107medical tasks
4capability domains
5 × 5models × harnesses
8,025evaluation episodes
Four medical task environments feed into a model-by-harness comparison and MH-Lab, a controlled study of context management, planning, and tool exposure.
One benchmark. Two levels of study: compare complete agent systems, then isolate individual mechanisms.

THE BENCHMARK

Four environments.
Four different demands.

MedicalHarnessBench brings together 107 tasks adapted from four public benchmarks. Each environment tests a different capability needed to turn a medical request into a completed task.

32 tasks

MedTool

Medical tool use

Read patient data, choose clinical calculators, and return the correct numeric results.

Patient recordsCalculations
23 tasks

MedMemory

Long-context retrieval

Find the evidence needed to answer a question within a long history of patient consultations.

Patient historyEvidence & answer
28 tasks

MedPlanning

Medical data workflows

Work through clinical records, trial eligibility, and pathology tasks to produce structured outputs.

Clinical inputsOutput files
24 tasks

MedWeb

Browser interaction

Complete administrative workflows across an EMR, a payer portal, and a fax portal.

Browser actionsUpdated state

THE STUDY DESIGN

From systems to mechanisms.

We ask how much outcomes change with the harness, where those differences arise, and when an individual mechanism helps.

01

Compare model–harness pairs

Five open-weight models run under five product harnesses on the same tasks. Within each model, comparisons share the model checkpoint, serving stack, context window, and episode budget, with three seeds.

Gemma 4Qwen 3.6GPT-OSSNemotron 3.5GLM-4.7-Flash
OpenClawHermesZeroClawCodexClaude Code
02

Switch one mechanism at a time

MH-Lab keeps the model and execution loop fixed while changing context management, planning, or tool exposure. Traces show whether a mechanism was used and how behavior changed.

Context managementKeep history usable
PlanningTrack unfinished work
Tool exposureAccess the right tools
Inside MH-Lab

EXPLORE THE RESULTS

Same model.
Different harness.

Browse all 25 model–harness pairs in each environment. Each value summarizes three seeds on the same task set.

MedMemory · Outcome

Loading the released study results…

0100%
Mean outcome and standard deviation across three seeds
ModelOpenClawHermesZeroClawCodexClaude Code
Loading results…

Mean ± standard deviation across three seeds.

Download data

RQ1 · HOW MUCH?

≈24%

of outcome variation is associated with the harness and its interaction with the model.

Equal-weight average across the four environments. No single harness is best across models and tasks.

RQ2 · WHERE?

Getting to an answer is part of the gap.

Execution traces separate answer availability from answer quality, then locate the different failure modes in each environment.

RQ3 · WHY?

A useful mechanism depends on the model and the task.

Controlled switches test context management, planning, and tool exposure within the same agent loop.

Outcome variation broken down by model, harness, their interaction, and run variability, alongside harness spreads in answer rate and score.
From the paper: outcome variation across configurations, and differences in answer rate and answer score.

CONTROLLED MECHANISM STUDY

Meet MH-Lab.

A shared agent loop with explicit switches.

To examine one mechanism at a time, MH-Lab keeps task tools, model, and execution conditions fixed within each comparison.

Context management
Summarize older history as context fills, or retain the accumulated history.
Planning
Expose a persistent plan and update tool, or remove explicit planning support.
Tool exposure
Expose tool schemas directly, or make tools available through search and call.
Explore the implementation
MH-Lab executes a repeated model, tool, and observation loop with switchable context management, planning, and tool exposure.

BUILD ON THIS WORK

Start with one task.

Install the code, check the execution loop locally, then connect your own model endpoint. The repository documents the task data and harness configuration.

A local smoke test

The smoke test uses a local test endpoint and runs without a GPU. Bootstrap first installs the pinned runner and project patches.

Full setup instructions
Terminal
git clone https://github.com/REAL-Lab-NU/MedicalHarness.git
cd MedicalHarness
python -m pip install -e '.[dev]'
medicalharness bootstrap
medicalharness smoke
Run a MedTool task with your own model

Real episodes run on Linux with bubblewrap. After installation and bootstrap, start an OpenAI-compatible model server and replace MODEL and the endpoint below with your deployment. The MedTool export uses data included in the repository.

Single-task example
python scripts/export_calc_to_harnessbench.py --output tasks
medicalharness run-one \
  --tasks tasks --task medcalc-001 --harness mhlab \
  --model MODEL --upstream http://127.0.0.1:8000/v1 \
  --output runs/example

CITATION

Cite this work.

If you use MedicalHarnessBench, MH-Lab, or these evaluation results, please cite MedicalHarness.

BibTeX
@misc{wang2026medicalharness,
  title  = {MedicalHarness: A Controlled Evaluation of LLMs and
            Agent Harnesses on Medical Tasks},
  author = {Wang, Ziqing and Zhao, Lili and Ding, Kaize},
  year   = {2026},
  url    = {https://github.com/REAL-Lab-NU/MedicalHarness}
}