Skip to main content

๐Ÿง  nnInteractive Agent

An agentic 3D annotation workflow that gives general-purpose AI agents nnInteractive as a tool, so they can interpret requests, place spatial prompts, inspect the resulting volume masks and refine them iteratively.

Official codePaperBase model weights

โ† Model atlas

At a glanceโ€‹

FieldDetails
Model typeAgentic annotation framework built around nnInteractive; it is not a standalone segmentation backbone.
Research team / organizationPaper affiliations include the Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ); National Center for Tumor Diseases (NCT) Heidelberg; Helmholtz Imaging; Heidelberg University; University Hospital Bonn; and Heidelberg University Hospital.
Input / modalityNIfTI, NRRD, MHA, TIFF stacks, PNG/JPG and NPY. The paper evaluates CT, MRI, PET and micro-CT; the project also documents microscopy and photographs.
Prompt / interactionNatural-language task request; the agent finds targets, sends spatial scribbles/boxes/points to nnInteractive, inspects rendered 2D/3D views and iterates corrections.
Code and model termsAgent software: Apache-2.0. The nnInteractive v1 checkpoint uses CC BY-NC-SA 4.0. The external LLM/provider has separate terms.

๐Ÿ“š Paper and publicationโ€‹

nnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators โ€” technical report, arXiv:2610.31080v1 (2026; submitted 22 October 2026). The arXiv record identifies it as a technical report and does not list a journal or conference venue.

๐Ÿ“ฆ Checkpoints and accessโ€‹

nnInteractive Agent has no separate trained segmentation checkpoint. Its setup installs the nnInteractive backend and downloads the base model weights (about 400 MB) from the official model registry; the paper's benchmark uses nnInteractive v1.0. The general-purpose language model is configured separately through Claude Code, Codex or another compatible agent runtime.

The base weights are linked above. Their license is separate from the Apache-2.0 agent code.

โš™๏ธ Official setup and inferenceโ€‹

Requires Python 3.10 or newer. Local inference strongly benefits from an NVIDIA GPU with about 10 GB of VRAM; CPU inference is possible but slow. Log into Codex or Claude Code before setup.

python -m venv .venv
source .venv/bin/activate
pip install "nninteractive-agent[local] @ git+https://github.com/MIC-DKFZ/nninteractive-agent"

# Register the MCP server and install its segmentation skill:
nninteractive-agent setup codex
# Or: nninteractive-agent setup claude

# Start an agent session and describe the task:
codex "Segment the liver in /data/ct.nii.gz and save the result."
# Or use: claude "Segment the liver in /data/ct.nii.gz and save the result."

For inference on a separate GPU server, configure nninteractive-agent setup codex --server <URL>. The agent writes an instance map and objects.json; supported inputs include medical volumes, TIFF stacks and common 2D images.

๐Ÿ“Š Reported evaluationโ€‹

The technical report evaluates 14 public images and 51 targets. From the target request alone, it reports mean Dice of 0.81 for Claude Code and 0.84 for Codex; when each target's location is also provided, both reach 0.85. These are results for the specific agent, LLM and benchmark protocol in the paper.

โš ๏ธ Practical notesโ€‹

  • The agent decides what to segment and where to place prompts; nnInteractive produces the voxel masks. Record both the LLM/runtime and the nnInteractive checkpoint.
  • Agent-selected prompts and interaction rounds are not equivalent to a fixed manual-click budget.
  • The code, segmentation weights, paper and LLM service have separate licenses or usage terms; review each source before use.

๐Ÿ“œ Licenseโ€‹

The nninteractive-agent repository is Apache-2.0. The official nnInteractive v1 weights are CC BY-NC-SA 4.0. The arXiv paper is CC BY-SA 4.0. Follow the separate terms for any selected LLM provider.

Citation: Fabian Isensee et al., โ€œnnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators,โ€ arXiv:2610.31080, 2026. See the paper and official project. This repository links to these sources; it does not redistribute code, weights or paper files.

Upstream sources checked: 27 October 2026. Requirements, model weights and service terms can change; follow the official repository and model registry.