Unai Garciarena Hualde: ”Contributions to neuronal architecture search in generative and multitask modeling”

Zuzendariak_Directores: Alexander Mendiburu Alberro / Roberto Santana Hermida.

2021_07_22, 14:00  Sala Ada Lovelace aretoa.

Abstract:

In the last decade, deep neural networks have risen as powerful computing systems, which, when properly trained, have shown impressive performances in vastly different problems and domains: from classification to optimization, from health to logistics. However, the performance of these models is largely dependent on its structure (the architecture, training hyperparameters, etc.). Currently, in most cases, the process for improving the performance of a neural model in a given task involves the development of another, more complex deep neural network. Multiple repetitions of this process have established why manually devising the structure of a neural model is becoming an increasingly complex assignment.   
In this context, researchers and practitioners have recently shown interest in methods which automatically search for structures of neural models. Due to the costly nature of deep neural network training, enormous amounts of computation time are spent on automatic development of neural structures in order to obtain model configurations which are capable of outperforming the previous best proposal. In a framework in which the evaluation of every structure counts, developing efficient methods for automated neural structure design would bring two benefits: i) save considerable amounts of computational time and resources (greener computing), and ii) provide better architectures.   
That is precisely the final aim of this dissertation. To that end, we embrace two different problems with very different characteristics. First, we focus on the generative adversarial network, which has produced impressive results in, among others, the area of realistic image generation. Despite this,  the automated optimization of its structure has not gathered as much interest as other popular deep neural models, although the results reported in this dissertation suggest that the performance of these models is also strongly dependent on their structure. Secondly, we focus on the heterogeneous multi-task learning problem, a framework with an ample potential, but an equally high complexity level. In this case, not only the efficiency of structural search methods is investigated, but a new model with the capacity to answer to heterogeneous problems and an optimizable structure is also proposed.   
In both cases, different types of approaches to the automated search of structures are undertaken, to cover as many types of optimization algorithms as possible, always with the same mindset: efficiently finding strong performing structures. Extensive analyses have been carried out in all cases in order to understand the results provided by each approach in each search type. These studies have shown that efficiency gains in structural searches can be achieved by investigating the characteristics of the model and the mechanics of the search algorithms; and that strong relationships between the components of neural structures exists, in the sense that for one component of a network to work properly, it needs to be surrounded by the right components.