Please be advised that the site will be down for maintenance on Sunday, September 1, 2024, from 08:00 to 18:00, and again on Monday, September 2, 2024, from 08:00 to 09:00. We apologize for any inconvenience this may cause.
dc.contributor.author | Bosman, Anna Sergeevna![]() |
|
dc.contributor.author | Engelbrecht, Andries![]() |
|
dc.contributor.author | Helbig, Marde![]() |
|
dc.date.accessioned | 2021-09-17T05:44:13Z | |
dc.date.issued | 2020-08 | |
dc.description.abstract | Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points via gradient-based stochastic sampling. The proposed technique is used to perform an empirical study of the loss surfaces generated by two different error metrics: quadratic loss and entropic loss. The empirical observations confirm the theoretical hypothesis regarding the nature of neural network attraction basins. Entropic loss is shown to exhibit stronger gradients and fewer stationary points than quadratic loss, indicating that entropic loss has a more searchable landscape. Quadratic loss is shown to be more resilient to overfitting than entropic loss. Both losses are shown to exhibit local minima, but the number of local minima is shown to decrease with an increase in dimensionality. Thus, the proposed visualisation technique successfully captures the local minima properties exhibited by the neural network loss surfaces, and can be used for the purpose of fitness landscape analysis of neural networks. | en_ZA |
dc.description.department | Computer Science | en_ZA |
dc.description.embargo | 2022-03-12 | |
dc.description.librarian | hj2021 | en_ZA |
dc.description.uri | http://www.elsevier.com/locate/neucom | en_ZA |
dc.identifier.citation | Bosman, A.S., Engelbrecht, A. & Helbig, M. 2020, 'Visualising basins of attraction for the cross-entropy and the squared error neural network loss functions', Neurocomputing, vol. 400, pp. 113-136. | en_ZA |
dc.identifier.issn | 0925-2312 (print) | |
dc.identifier.issn | 1872-8286 (online) | |
dc.identifier.other | 10.1016/j.neucom.2020.02.113 | |
dc.identifier.uri | http://hdl.handle.net/2263/81891 | |
dc.language.iso | en | en_ZA |
dc.publisher | Elsevier | en_ZA |
dc.rights | © 2020 Elsevier GmbH. All rights reserved. Notice : this is the author’s version of a work that was accepted for publication in Neurocomputing. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. A definitive version was subsequently published in Neurocomputing, vol. 400, pp. 113-136 , 2020. doi : 10.1016/j.neucom.2020.02.113. | en_ZA |
dc.subject | Fitness landscape analysis | en_ZA |
dc.subject | Neural networks | en_ZA |
dc.subject | Cross-entropy | en_ZA |
dc.subject | Squared error | en_ZA |
dc.subject | Local minima | en_ZA |
dc.subject | Loss functions | en_ZA |
dc.title | Visualising basins of attraction for the cross-entropy and the squared error neural network loss functions | en_ZA |
dc.type | Postprint Article | en_ZA |