Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities

Title: Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities

Authors: Nhat Le, Ciyue Shen, Chintan Shah, Blake Martin, Daniel Shenker, Harshith Padigela, Jennifer Hipp, Sean Grullon, John Abel, Harsha Vardhan Pokkalla, Dinkar Juyal

Published: Jul 15 2024

Link: https://arxiv.org/abs/2407.10785

Summary (Generated by Microsoft Copilot):

Introduction:

• The paper explores mechanistic interpretability for medical imaging using a pathology foundation model.

Challenges:

• Batch effects in pathology images.
• Polysemantic neurons storing multiple concepts.

Methods:

• Analysis of features from a ViT-Small encoder on pathology images and spatial transcriptomics datasets.

Novelties:

• Interpretable representation of cell and tissue morphology and gene expression.

Results:

• Embedding dimensions capture complex pathology-related concepts.
• Linear regression models predict nuclear characteristics.

Performances:

• Pearson correlation of 0.51 to 0.91 for nuclear characteristics prediction.

Limitations:

• Preliminary investigation; further decomposition of concepts needed.

Discussion:

• Generalizability of embeddings across different datasets and modalities.