Transferred Discrepancy: Quantifying the Difference Between Representations

Download paper here

Understanding what information neural networks capture is an essential problem in deep learning, and studying whether different models capture similar features is an initial step to achieve this goal. Previous works sought to define metrics over the feature matrices to measure the difference between two models. In this work, we propose a novel metric that goes beyond previous approaches. We argue that we should design the metric based on a similar principle. For that, we introduce the transferred discrepancy (TD), a new metric that defines the difference between two representations based on their downstream-task performance. We also find that TD may be used to evaluate the effectiveness of different training strategies. This suggests a training strategy that leads to more robust representation also trains models that generalize better.

Yunzhen Feng , Runtian Zhai, Di He, Liwei Wang, Bin Dong