Résumé
British Machine Vision Conference, Nov 2024, Glasgow, United
Kingdom Cross-modal knowledge distillation (CMKD) refers to the scenario in which a
learning framework must handle training and test data that exhibit a modality
mismatch, more precisely, training and test data do not cover the same set of
data modalities. Traditional approaches for CMKD are based on a teacher/student
paradigm where a teacher is trained on multi-modal data with the aim to
successively distill knowledge from a multi-modal teacher to a single-modal
student. Despite the widespread adoption of such paradigm, recent research has
highlighted its inherent limitations in the context of cross-modal knowledge
transfer.Taking a step beyond the teacher/student paradigm, here we introduce a
new framework for cross-modal knowledge distillation, named DisCoM-KD
(Disentanglement-learning based Cross-Modal Knowledge Distillation), that
explicitly models different types of per-modality information with the aim to
transfer knowledge from multi-modal data to a single-modal classifier. To this
end, DisCoM-KD effectively combines disentanglement representation learning
with adversarial domain adaptation to simultaneously extract, foreach modality,
domain-invariant, domain-informative and domain-irrelevant features according
to a specific downstream task. Unlike the traditional teacher/student paradigm,
our framework simultaneously learns all single-modal classifiers, eliminating
the need to learn each student model separately as well as the teacher
classifier. We evaluated DisCoM-KD on three standard multi-modal benchmarks and
compared its behaviourwith recent SOTA knowledge distillation frameworks. The
findings clearly demonstrate the effectiveness of DisCoM-KD over competitors
considering mismatch scenarios involving both overlapping and non-overlapping
modalities. These results offer insights to reconsider the traditional paradigm
for distilling information from multi-modal data to single-modal neural
networks.