P-Laplacian: Difference between revisions
Jump to navigation
Jump to search
en>RayAYang →Energy formulation: more specific |
en>RayAYang →External links: this counts as further reading instead |
||
| Line 1: | Line 1: | ||
In the [[mathematics|mathematical]] theory of [[neural networks]], the '''universal approximation theorem''' states<ref>Balázs Csanád Csáji. Approximation with Artificial Neural Networks; Faculty of Sciences; Eötvös Loránd University, Hungary</ref> that a [[feedforward neural network|feed-forward]] network with a single hidden layer containing a finite number of [[neuron]]s, the simplest form of the [[multilayer perceptron]], is a universal approximator among [[continuous functions]] on [[Compact_space|compact subsets]] of [[Euclidean space|'''R'''<sup>n</sup>]], under mild assumptions on the activation function. | |||
One of the first versions of the [[theorem]] was proved by [[George Cybenko]] in 1989 for [[sigmoid function|sigmoid]] activation functions.<ref name=cyb>Cybenko., G. (1989) [http://actcomm.dartmouth.edu/gvc/papers/approx_by_superposition.pdf "Approximations by superpositions of sigmoidal functions"], ''[[Mathematics of Control, Signals, and Systems]]'', 2 (4), 303-314</ref> | |||
Kurt Hornik showed in 1991<ref name=horn> Kurt Hornik (1991) "Approximation Capabilities of Multilayer Feedforward Networks", ''Neural Networks'', 4(2), 251–257 </ref> that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of being universal approximators. The output units are always assumed to be linear. For notational convenience, only the single output case will be shown. The general case can easily be deduced from the single output case. | |||
== Formal statement == | |||
The theorem<ref name=cyb/><ref name=horn/><ref>Haykin, Simon (1998). ''Neural Networks: A Comprehensive Foundation'', Volume 2, Prentice Hall. ISBN 0-13-273350-1.</ref><ref>Hassoun, M. (1995) ''Fundamentals of Artificial Neural Networks'' MIT Press, p. 48</ref> in mathematical terms: | |||
<blockquote> | |||
Let φ(·) be a nonconstant, [[Bounded function|bounded]], and [[monotonic function|monotonically]]-increasing [[continuous function|continuous]] function. Let ''I''<sub>''m''</sub> denote the ''m''-dimensional [[unit hypercube]] [0,1]<sup>''m''</sup>. The space of continuous functions on ''I''<sub>''m''</sub> is denoted by ''C''(''I''<sub>''m''</sub>). Then, given any function ''f'' ∈ ''C''(''I''<sub>''m''</sub>) and є > 0, there exist an integer ''N'' and real constants ''α''<sub>''i''</sub>, ''b''<sub>''i''</sub> ∈ '''R''', ''w''<sub>''i''</sub> ∈ '''R'''<sup>''m''</sup>, where ''i'' = 1, ..., ''N'' such that we may define: | |||
: <math> | |||
F( x ) = | |||
\sum_{i=1}^{N} \alpha_i \varphi \left( w_i^T x + b_i\right) | |||
</math> | |||
as an approximate realization of the function ''f'' where ''f'' is independent of φ; that is, | |||
: <math> | |||
| F( x ) - f ( x ) | < \varepsilon | |||
</math> | |||
for all ''x'' ∈ ''I''<sub>''m''</sub>. In other words, functions of the form ''F(x)'' are [[Dense set|dense]] in ''C''(''I''<sub>''m''</sub>). | |||
</blockquote> | |||
==References== | |||
{{Reflist}} | |||
{{DEFAULTSORT:Universal Approximation Theorem}} | |||
[[Category:Theorems in discrete mathematics]] | |||
[[Category:Neural networks]] | |||
[[Category:Network architecture]] | |||
[[Category:Networks]] | |||
[[Category:Information, knowledge, and uncertainty]] | |||
{{applied-math-stub}} | |||
Revision as of 19:00, 22 April 2013
In the mathematical theory of neural networks, the universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons, the simplest form of the multilayer perceptron, is a universal approximator among continuous functions on compact subsets of Rn, under mild assumptions on the activation function.
One of the first versions of the theorem was proved by George Cybenko in 1989 for sigmoid activation functions.[2]
Kurt Hornik showed in 1991[3] that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of being universal approximators. The output units are always assumed to be linear. For notational convenience, only the single output case will be shown. The general case can easily be deduced from the single output case.
Formal statement
The theorem[2][3][4][5] in mathematical terms:
Let φ(·) be a nonconstant, bounded, and monotonically-increasing continuous function. Let Im denote the m-dimensional unit hypercube [0,1]m. The space of continuous functions on Im is denoted by C(Im). Then, given any function f ∈ C(Im) and є > 0, there exist an integer N and real constants αi, bi ∈ R, wi ∈ Rm, where i = 1, ..., N such that we may define:
as an approximate realization of the function f where f is independent of φ; that is,
for all x ∈ Im. In other words, functions of the form F(x) are dense in C(Im).
References
43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.
- ↑ Balázs Csanád Csáji. Approximation with Artificial Neural Networks; Faculty of Sciences; Eötvös Loránd University, Hungary
- ↑ 2.0 2.1 Cybenko., G. (1989) "Approximations by superpositions of sigmoidal functions", Mathematics of Control, Signals, and Systems, 2 (4), 303-314
- ↑ 3.0 3.1 Kurt Hornik (1991) "Approximation Capabilities of Multilayer Feedforward Networks", Neural Networks, 4(2), 251–257
- ↑ Haykin, Simon (1998). Neural Networks: A Comprehensive Foundation, Volume 2, Prentice Hall. ISBN 0-13-273350-1.
- ↑ Hassoun, M. (1995) Fundamentals of Artificial Neural Networks MIT Press, p. 48