论文精读：《SAM 2: Segment Anything in Images and Videos》

更好阅读体验：
【我的博客原文】

概览

SAM2提出了PVS(Promptable Visual Segmentation)任务。在Section 3与Appendix B中,文章给出了他的具体定义。在Section 3中,文章具体描述了这个任务为:

Our PVS taskallows providing prompts to the model on any frame of a video. Prompts can bepositive/negative clicks, boxes, or masks, either to define an object to segment or to refine a model-predicted one. To provide an interactive experience, upon receiving a prompt on a specific frame, the modelshould **immediately respond** with a valid segmentation mask of the object on this frame. After receiving initial prompts (either on the same frame or different frames), themodel should **propagate** these prompts to obtain the masklet of the object across the entire video, localizing the segmentation mask of the target on every video frame. Additional prompts can be provided to the model on any frame to refine the segment throughout the video (example in Fig. 2).

简要来讲,他将Segment Anything的Task在“视频与交互”的领域做了延伸扩展。文章着重点出了propagate这一概念。具体来说,这意味着模型应该将帧间稀疏的标注传递到每一帧。

相较SAM,SAM2更聚焦于视频中物体的连续分割。在SAM2的实现中,它设计了一个Memory Bank与Memory Attention用以保留并propagate帧间的Mask结果与prompt。

与SAM类似,它也采用了和SAM一致的Image Encoder、一致的Prompt Encoder与结构类似的Mask Decoder。在工作进行过程中,SAM2也着重聚焦于Data Engine的构建——本质上,他也是一个数据工程。

我认为贡献如下:

• 定义了PVS这一任务
• 设计了Memory Bank与Memory Attention的架构用于帧间prompts传递与帧间分割一致性
• 设计了渐进式数据引擎
• 发布了SA-V数据集

Related Work

• Image segmentation：在本文中主要聚焦于SAM以及SAM的下游工作。
• Interactive Video Object Segmentation (iVOS)：对于视频，可交互的标注希望连续追踪的目标物体，模型及时的响应、分割。
• Video Object Segmentation (VOS)：对于视频，在第一帧标注出目标物体，模型应该自动在整个视频中标注出目标。
• Video segmentation datasets：常见的VOS数据集如：Yotube-VOS、DAVIS

Task: promptable visual segmentation

对PVS的具体定义如下：

Our PVS task allows providing prompts to the model on any frame of a video. Prompts can be positive/negative clicks, boxes, or masks, either to define an object to segment or to refine a model-predicted one. To provide an interactive experience, upon receiving a prompt on a specific frame, the model should immediately respond with a valid segmentation mask of the object on this frame. After receiving initial prompts (either on the same frame or different frames), the model should propagate these prompts to obtain the masklet of the object across the entire video, localizing the segmentation mask of the target on every video frame. Additional prompts can be provided to the model on any frame to refine the segment throughout the video (example in Fig. 2).

Model & Appendix D

模型架构非常值得一读。简要来说，他在SAM的基础上，增加了Memory Bank与Memory Attention，用以实现在视频帧间进行propagate。架构如下：

• Image encoder
• Memory attention
• Prompt encoder and mask decoder
• Memory encoder
• Memory bank

相较SAM，SAM2多了很多Memory的架构。

在这一章中有提到：

The frame embedding used by the SAM 2 decoder is not directly from an image encoder and is instead conditioned on memories of past predictions and prompted frames.It is possible for prompted frames to also come “from the future” relative to the current frame.Memories of frames are created by the memory encoder based on the current prediction and placed in a memory bank for use in subsequent frames. The memory attention operation takes the per-frame embedding from the image encoder and conditions it on the memory bank, before the mask decoder ingests it to form a prediction.

这里说到”prompted frames to also come “from the future” relative to the current frame”。我对其理解是这样的：Memory Bank保存分两部分（见下文）——过去预测帧与提示帧。在其中，过去预测帧都是 “from past” 的，而提示帧则可能来自当前图片后面。我想我之后需要深度挖掘一下代码来确定下。

Image Encoder

SAM2的Image Encoder采用了Hiera Image Encoder。Hiera Image Encoder是Meta提出的一个高性能Image Encoder，它主张通过强大的预训练任务（如 MAE）来替代分层视觉 Transformer 中专用模块（例如卷积）提供的空间偏差。

这是什么意思呢？原始的的ViT也没有“专用模块”啊？确实，原始的的ViT没有。

但是”分层视觉 Transformer”比如大名鼎鼎的Swin Transformer就有：

好，Hiera 确实没有繁杂的Patch merging操作，但是他却有“Q Pooling”，这又是什么？下图是MViT给出的Q Pooling/Pooling attention架构：

简单来说，原始的Attention是：

Q = X W Q ， K = X W k , V = X W v Attn ( X ) = softmax ( Q K ⊤ d ) V Q=XW_Q，K = XW_k, V=XW_v\\ \text{Attn}(X)=\text{softmax}(\frac{QK^\top}{\sqrt{d}})VQ=XWQ​