Fitness landscape analysis of feed-forward neural networks

Abstract

English: Neural network training is a highly non-convex optimisation problem with poorly understood properties. Due to the inherent high dimensionality, neural network search spaces cannot be intuitively visualised, thus other means to establish search space properties have to be employed. Fitness landscape analysis encompasses a selection of techniques designed to estimate the properties of a search landscape associated with an optimisation problem. Applied to neural network training, fitness landscape analysis can be used to establish a link between the properties of the error landscape and various neural network hyperparameters. This study applies fitness landscape analysis to investigate the influence of the search space boundaries, regularisation parameters, loss functions, activation functions, and feed-forward neural network architectures on the properties of the resulting error landscape. A novel gradient-based sampling technique is proposed, together with a novel method to quantify and visualise stationary points and the associated basins of attraction in neural network error landscapes.

isiZulu: Ukuqeqeshwa komphambo womuzwa okwenzeka ngemfanelo okunenkinga engekho mangungu kakhulu ekwenzeni kahle emandleni okuveza aqondakala kabi. Ngenxa yobukhulu obuphezulu ngokwemvelo, izikhala zokufuna umphambo wemizwa azikwazi ukuqondakala ngeso lengqondo, yingakho kudingeka olunye uhlelo lokumisa amandla ezikhala zokufuna amandla okuveza okumele asetshenziswe. Ukuhlaziya indawo yokuqina komzimba kuhlanganisa amasu akhethiwe aklanyelwe ukulinganisela amandla okuveza okufuna ngokubuka ngamehlo okuhlobene nenkinga yokwenza ngemfanelo. Ngokusetshenziswa ekuqeqeshweni komphambo wemizwa, ukuhlaziya indawo yokuqina komzimba kungasetshenziswa ukumisa ukuhlanganisa ngaphakathi kwamandla okuvezwa kwephutha lokubuka ngamehlo kanye nemingcele esebenzayo yemiphambo yemizwa ehlukahlukene. Lolu cwaningo lusebenzisa ukuhlaziya indawo yokuqina komzimba ukuphenya umthelela wemingcele yezikhala zokufuna, imingcele yokujwayela, ukulahleka kwamandla okusetshenziswa, kanye nezakhiwo zomphambo wemizwa yokuphakela phambili ngaphezu kwamandla okuveza okuwumphumela oyiphutha wendawo. Kuphakanyiswa isu elisha eliyisampula exhaswa ubukhulu bokukhuphuka, kanye nendlela entsha yokubala kanye nokubona ngeso lengqondo amaphuzu amile kanye nokukhangwa kobhesini abahlobene ngaphakathi kwezindawo ezingamaphutha zomphambo wemizwa.

Afrikaans: Neurale netwerkleer is 'n hoogs, niekonvekse optimiseringsprobleem met swak begrepe eienskappe. As gevolg van die inherente hoë dimensionaliteit, kan neurale netwerk soekruimtes nie intuïtief gevisualiseer word nie; dus moet ander maniere gebruik word om soekruimte-eienskappe te vestig. Fiksheidlandskapontleding sluit 'n seleksie van tegnieke in, wat ontwerp is om die eienskappe van 'n soeklandskap wat met 'n optimiseringsprobleem geassosieer word, te skat. Wanneer fiksheidlandskapontleding toegepas word op neurale netwerkleer, kan dit gebruik word om 'n verband tussen die eienskappe van die foutlandskap en verskeie neurale netwerk hiperparameters te vestig. Hierdie studie pas fiksheidslandskapontleding toe om die invloed van die soekruimtegrense, regulariseringsparameters, verliesfunksies, aktiveringsfunksies en voerentoevoer- neurale netwerkargitekture op die eienskappe van die gevolglike foutlandskap te ondersoek. n Nuwe gradiënt-gebaseerde steekproeftegniek word voorgestel, tesame met 'n nuwe metode, om stilstaande punte en die gepaardgaande geassosieerde bakke in neurale netwerk foutlandskappe te kwantifiseer en te visualiseer.

Sepedi: Tlwaetšo ya netweke ya nyurale ke bothata bja go dira gore dilo di šome gabotse bjo e sego bja go kobegela ntle kudu bjo bo nago le diponagalo tšeo di sa kwešišwego gabotse. Ka lebaka la tlhago ya tekanyo ya godimo, dikgoba tša go nyaka netweke ya nyurale di ka se bonwe ka leihlo la kgopolo, ke ya mekgwa e mengwe ya go hlama diponagalo tša sekgoba sa go nyaka e swanetše go thwala. Tshekatsheko ya boitekanelo bja sebopego e akaretša kgetho ya dithekniki tšeo di hlamilwego go akanyetša diponagalo tša boitekanelo tša go nyaka tšeo di amanago le bothata bja go dira gore dilo di šome gabotse. Tirišo ya tlwaetšo ya netweke ya neural, tshekatsheko ya ponagalo ya boitekanelo bja sebopego go e ka šomišwa go hloma kgokagano magareng ga diponagalo tša sebopego tšeo di fošagetšego le di-hyperparameter tše di fapafapanego tša netweke ya nyurale. Thuto ye e diriša kudu tshekatsheko boitekanelo go nyakišiša khuetšo ya mellwane ya sekgoba sa go nyaka, "ditekanyetšo" tša go dira gore dilo di tlwaelege, tahlegelo ya tšhomišo, mešomo ya go tsenya tirišong, le ditlhamo tša netweke ya dinyurale tša go fepa pele tšweletšo ya diponagalo tša sebopego se fošagetšego seo se tšweletšwago. Thekniki ye mphsa ya go tšea mehlala šišintšwe, mmogo le mokgwa wo mofsa wa go lekanyetša le go bona ka leihlo la kgopolo dintlha tše di sa emego le dibeisine tša kgogedi tše di amanago le tšona ka diponagalo tša diphošo tša sebopego sa netweke ya "nyurale".