Iterative robust search (iRoSe) : a framework for coevolutionary hyperparameter optimisation

Abstract

Finding an optimal hyperparameter configuration for machine learning algorithms is challenging due to hyperparameter effects that could vary with algorithms, dataset and distribution, as also due to the large combinatorial search space of hyperparameter values requiring expensive trials. Furthermore, extant optimisation procedures that search out optima randomly and in a manner non-specific to the optimisation problem, when viewed through the "No Free Lunches" theorem, could be considered a priori unjustifiable. In seeking a coevolutionary, adaptive strategy that robustifies the search for optimal hyperparameter values, we investigate specifics of the optimisation problem through 'macro-modelling' that abstracts out the complexity of the algorithm in terms of signal, control factors, noise factors and response. We design and run a budgeted number of 'proportionally balanced' trials using a predetermined mix of candidate control factors. Based on the responses from these proportional trials, we conduct 'main effects analysis' of individual hyperparameters of the algorithm, in terms of the signal to noise ratio, to derive hyperparameter configurations that enhance targeted performance characteristics through additivity. We formulate an iterative Robust Search (iRoSe) hyperparameter optimisation framework that leverages these problem-specific insights. Initialised with a valid hyperparameter configuration, iRoSe evidences ability to adaptively converge to a configuration that produces effective gain in performance characteristic, through designed search trials that are justifiable through extant theory. We demonstrate the iRoSe optimisation framework on a Deep Neural Network and CIFAR-10 dataset, comparing it to Bayesian optimisation procedure, to highlight the transformation achieved.

Date of Award	2020
Original language	English