Deep learning with limited data

Deep neural networks is a machine learning technique based on a model of neuron-like elements. The \"deep\" refers here to networks with many layers that are capable of representing complex functions. Such models have been proven to be extremely valuable as they can learn hierarchical data representations from examples and predict previously unseen data. Such techniques enabled much of the recent progress in computer vision and pattern recognition, and the advancements of deep neural networks was made possible by the availability of large data sets (Big Data), some refinement of algorithms, and the availability of increased computational power with specialized processors such as graphical processing units (GPUs). Companies like Google, Microsoft, Amazon and Facebook, and also numerous smaller engineering firms and market analysis firms are embracing this new technology.

However, many applications do not have the luxury of having a large set of examples available to train complex models. In the research program proposed here I am trying to develop methods for applications with limited data, which includes a variety of situations such as small number of measurements, biased examples, or examples from peripheral data. Applications with limited data are common. For example, in the ocean science community it is common to have a fairly small coverage of measurements compared to the vastness of the oceans. My lab has started to investigate how artificial data can augment pre-training and facilitate the narrowing of the search space. For example, there exist a variety of physical models of ocean dynamics such as the exchange of CO2 with the atmosphere. While these models are generally considered to be insufficient for predictions on the required scale, it is possible that these simulations can provide data for the pre-training of deep networks.

Another area that promises a whole new level of techniques to training deep networks is the guidance of the learning process by domain experts, also called transfer learning. Domain experts include hereby either deep neural networks that have been trained on complementary or similar domains, or even human experts. While transfer learning has been considered for some time, the use of deep networks to learn appropriate communication channels for learning opened exciting new possibilities for much progress in this area. Such expert training would provide network training that parallels human learning. Such techniques could further drastically reduce the need of large training sets.

The long-term goal of this research program is to develop specific methods and tools to both evaluate a machine-learning problem in terms of its complexity and data need, and to provide a comprehensive toolbox to apply pre-learning and expert transfer learning strategies to problem domains that suffer from limited data.