In this work, the “effective dimension” of the output of the hidden layer of a one-hidden-layer neural network with random inner weights of its computational units is investigated. To do this, a polynomial approximation of the sigmoidal activation function of each computational unit is used, whose degree is chosen based both on a desired upper bound on the approximation error and on an estimate of the range of the input to that computational unit. This estimate of the range is parameterized by the number of inputs to the network and by an upper bound both on the size of the random inner weights of the network and on the size of its inputs. The results show that the Root Mean Square Error (RMSE) on the training set is influenced by the effective dimension and by the quality of the features associated with the output of the hidden layer.

Deeper insights into neural nets with random weights

Giorgio Gnecco;
2022

Abstract

In this work, the “effective dimension” of the output of the hidden layer of a one-hidden-layer neural network with random inner weights of its computational units is investigated. To do this, a polynomial approximation of the sigmoidal activation function of each computational unit is used, whose degree is chosen based both on a desired upper bound on the approximation error and on an estimate of the range of the input to that computational unit. This estimate of the range is parameterized by the number of inputs to the network and by an upper bound both on the size of the random inner weights of the network and on the size of its inputs. The results show that the Root Mean Square Error (RMSE) on the training set is influenced by the effective dimension and by the quality of the features associated with the output of the hidden layer.
978-3-030-97546-3
Neural networks with random weights, Hyperbolic tangent, Polynomial approximation, Effective dimension, Approximate rank
File in questo prodotto:
File Dimensione Formato  
AJCAI2021Paper21CameraReadyVersion.pdf

embargo fino al 18/03/2023

Tipologia: Documento in Post-print
Licenza: Creative commons
Dimensione 1.62 MB
Formato Adobe PDF
1.62 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
Li2022_Chapter_DeeperInsightsIntoNeuralNetsWi.pdf

non disponibili

Tipologia: Versione Editoriale (PDF)
Licenza: Nessuna licenza
Dimensione 687.14 kB
Formato Adobe PDF
687.14 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/20.500.11771/21084
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
social impact