PROJECT TITLE :

Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Data

ABSTRACT:

Deep models that have been through training retain their acquired knowledge in the form of model parameters. These parameters serve as "memory" for the trained models, which enables them to generalize well on data that they have not previously encountered. The utility of a trained model, on the other hand, is severely limited in the absence of training data, and can only be used for either inference or better initialization towards a target task. In this paper, we take it a step further and extract synthetic data by making use of the parameters that were learned about the model. These representations of the training data, which we refer to as Data Impressions, can be implemented in a variety of contexts and serve as proxies for the original data. These are helpful in situations in which only the pretrained models are available, and the training data is not being distributed (e.g., due to privacy or sensitivity concerns). We demonstrate how data impressions can be used to solve a variety of problems in computer vision, including unsupervised domain adaptation, continuous learning, and knowledge distillation. In addition to this, we investigate the robustness of lightweight models to adversarial inputs after being trained through knowledge distillation with these data impressions. In addition, we show that data impressions are effective in the generation of data-free Universal Adversarial Perturbations (UAPs) that have higher fooling rates. Extensive experiments carried out on benchmark datasets demonstrate that competitive performance can be achieved using data impressions even in the absence of the original training data.