AI and the World's Hard Problems · Theoretical computer science · Nature (computational biology) · 2026-10-23
Foundation model-driven framework for human–object interaction prediction with segmentation mask integration
Human–object interaction detection has mostly relied on detection-based methods. Seg2HOI integrates a segmentation-based vision foundation model, kept frozen, and predicts quadruplets that add segmentation masks for human–object pairs to standard triplets. It achieves performance comparable to state-of-the-art methods on two public benchmarks, even in zero-shot settings, and can generate quadruplets and interactive segmentations from text and visual prompts not used in training.