SAM 2: Segment Anything in Images and Videos

Title: SAM 2: Segment Anything in Images and Videos

Authors: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

Published: Aug 1 2024

Link: https://arxiv.org/abs/2408.00714

Summary (Generated by Microsoft Copilot):

Introduction:

• SAM 2 (Segment Anything Model 2) is a foundation model for visual segmentation in images and videos, designed to handle promptable segmentation tasks.

Challenges:

• Video segmentation faces unique challenges such as motion, deformation, occlusion, lighting changes, and efficient processing of numerous frames.

Methods:

• SAM 2 uses a transformer architecture with streaming memory for real-time video processing and a data engine to collect a large video segmentation dataset.

Novelties:

• SAM 2 is 6× faster and more accurate than its predecessor, SAM, and requires 3× fewer interactions for video segmentation.

Results:

• SAM 2 achieves better segmentation accuracy and outperforms previous models in both video and image segmentation benchmarks.

Performances:

• SAM 2 shows strong performance across various tasks, including zero-shot video and image segmentation, and demonstrates minimal performance discrepancy across demographic groups.

Limitations:

• SAM 2 struggles with segmenting objects across shot changes, crowded scenes, long occlusions, and fast-moving objects with fine details.

Discussion:

• SAM 2 represents a significant milestone in video segmentation, offering improvements in speed, accuracy, and interactive experience. Future work could focus on enhancing motion modeling and inter-object communication.