Skip to main content

๐Ÿค– MedSAM-Agent

An agentic interactive-segmentation framework that autonomously plans multiple rounds of prompts and segmentation calls, rather than asking a person to place every correction.

Official codePaperWeights

โ† Model atlas

At a glanceโ€‹

FieldDetails
Model typeAgentic multi-turn segmentation framework
Research team / organizationThe Chinese University of Hong Kong, Tencent Hunyuan and collaborators
Input / modality2D medical images from six modalities and 21 datasets in the authors' evaluation.
Prompt / interactionNatural-language target description plus agent-generated clicks; selectable Qwen or GPT grounding model and a separate segmentation backbone.
Code licenseThe GitHub repository snapshot does not include a top-level LICENSE file. Check the model card and each backbone's terms before reuse; do not assume the Qwen agent and segmentation weights share one license.

๐Ÿ“š Paper and publicationโ€‹

MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning โ€” arXiv:2602.03320 (2026 preprint).

๐Ÿ“ฆ Checkpoints and accessโ€‹

The official Hugging Face model is Saint-lsy/MedSAM-Agent-Qwen3-VL-8B-MedSAM2. Inference also requires a segmentation checkpoint (e.g. MedSAM2) and its configuration. Dataset/trajectory releases are marked pending in the upstream README.

The code, model checkpoint and any required foundation-model checkpoint can have different terms. Review each linked source before use or redistribution.

โš™๏ธ Official setup and inferenceโ€‹

conda create -n msagent python=3.11
conda activate msagent
pip install -r requirements.txt
# Download the Qwen agent model and a MedSAM2 checkpoint, then:
python infer/run_single_inference.py \
--img-path infer/demo/BTCV-0-106_CT_abdomen.png \
--target-description "right kidney in abdomen CT" \
--model-path /path/to/mllm_model \
--seg-checkpoint /path/to/MedSAM2_latest.pt \
--seg-model medsam

Upstream reports its development environment as Python 3.11, CUDA 12.9 and PyTorch 2.8.0. The paper's training used eight H20 GPUs; that is a training setup, not a guaranteed minimum inference requirement.

โš ๏ธ Version and practical notesโ€‹

This is an autonomous multi-turn method. Its action policy chooses prompts, so its interaction count is not directly comparable with a fixed manual-click budget. Record the grounding model and segmentation backbone separately.

๐Ÿ“œ Licenseโ€‹

The GitHub repository snapshot does not include a top-level LICENSE file. Check the model card and each backbone's terms before reuse; do not assume the Qwen agent and segmentation weights share one license.

Citation: Cite the paper above and the upstream project as directed by its authors. This repository links to the original paper; it does not redistribute the code or weights.

Source pages checked: 27 October 2026. Upstream environments, weights and access policies may change; use the linked project as the source of truth.