This project aims at investigating multimodal learning through the lens of imperfect data, where imperfect is intended as unlabeled, partially labelled or noisy labelled data, few-shot settings, imbalanced data, biased data, or a combination of these issues.
In particular, this project, named TALIM, Transfer and Adaptive Learning in Imperfect Multimodal data scenarios, proposes to investigate Computer Vision and Deep Learning approaches and methods in presence of imperfect multimodal data along three research lines. First, TALIM aims at exploring self-supervised methods to cope with the absence of labels while leveraging on the multimodality nature of the data to build robust pre-trained models. We also address these data flaws in multimodal settings in the context domain adaptation (DA) scenarios, where different modalities can support each other to increase model generalization, robustness to noise, and adaptation to a different modality. To this end, data augmentation and even generation of new synthetic samples can be leveraged to cope with the data scarcity. Third, distillation methods based on teacher-student architectures can be designed, in order to extract relevant information from either stream and managing situations in which a modality may be missing or unreliable.
Multimodal learning with imperfect data posits challenging open questions to actually cope with application domains effectively working in the wild, on how computational systems can adapt to new environments, scenarios and tasks when no, little or corrupted information is available a priori. While these topics are already addressed in the literature, just specific tailored solutions have been proposed for single-modal use cases, and a systematic study of these scenarios over multimodal data has not received sufficient attention.
The core technical idea to tackle these issues is based on the development of data augmentation and generation methods. The intuition is that the problems derived from data with missing, partially missing or wrong annotations, as well as under-represented or mis-represented classes, can be faced by finding adequate data transformation of the available data, or even by generating suitable synthetic data able to correct or re-balance the flaws of the available samples.
In the context of these non-ideal scenarios, TALIM aims at:
- Introducing a framework that unifies data augmentation and data generation for domain adaptation in imperfect multimodal data settings, so that generated/augmented data could be exploited in learning tasks of interest while
improving model performance.
- Along the same strategic line, developing self-supervised training methods leveraging the different multimodal data and the distillation paradigm in imperfect multimodal settings, especially when labels are fully unavailable.
- Apply such algorithms to several actual use cases of interest in the areas of industrial process control, behavior analysis, biomedical and health, and possibly climate and environment monitoring.
