A selection perturbative hyper-heuristic for neural architecture search

Abstract

Neural architecture search explores the architecture space, referred to as the design spaces, to find an architecture that produces good results. Various approaches, such as genetic algorithms, are usually used to explore this space. This study investigates exploring an alternative space, namely, the heuristic space using a hyper-heuristic to indirectly explore the design space. The study introduces the concept of a NAS operator space (NOS). A single point selection perturbative hyper-heuristic (SPHH-NAS) explores a heuristic space that maps to the NOS which then maps to the design space. A choice function is used for heuristic selection and the Adaptive Improvement Limited Target Acceptance (AILTA) for move acceptance. It is anticipated that indirectly searching the design space will facilitate reaching areas of the search space that could not be reached by searching the space directly. SPHH-NAS was evaluated on three NAS benchmark sets, namely, NAS-101, NAS-201 and NAS-301. In addition to this the approach is evaluated on two real-world datasets. SPHH-NAS was found to outperform majority of the previous approaches used to solve these problems. In addition to this SPHH-NAS resulted in a reduction in computational cost.

HIGHLIGHTS • This is the first study using a selection perturbative hyper-heuristics for neural architecture search. • The selection perturbative hyper-heuristic produces good results for NAS. • The selection perturbative reduces computational cost for NAS.