Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning

Table of Contents

Overview

Paper: Ye et al., Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning (cvpr2024 open access or arxiv).

img

(Figures and tables in this post are from the original paper)

Novelties of the Paper

  • They proposed Medical Continual Self-Supervised (MedCoSS) paradigm to prevent modal conflicts and catastrophic forgetting.
  • MedCoSS assingns each modality data to separate training stage in continual learning.
  • Rehearsal buffers are introduced to keep previous modal data.
  • Modalities: Report, X-ray, CT, MRI and Pathological images.

Performance Evaluation Methods

  • They compared MedCoSS to single-modal pre-training and, multi-modal pre-traingin incluing Joint SSL, EWC, ER, PackNet, CaSSLe (see table).

img

Discussions

  • MedCoSS performs best in some modalities, but not others.
  • The size of the rehearsal buffers is a trade-off between performance and multi-modal data collision and computational costs.