View source for Star (game theory)

[[File:Different_Views_on_RKHS.png|thumb|right|Figure illustrates related but varying approaches to viewing RKHS]]

In [[functional analysis]] (a branch of [[mathematics]]), a '''reproducing kernel Hilbert space''' is a [[Hilbert space]] of [[function space|functions]] in which pointwise evaluation is a [[bounded operator|continuous linear functional]].  Equivalently, they are spaces that can be defined by '''reproducing kernels'''.  The subject was originally and simultaneously developed by [[Nachman Aronszajn]] (1907&ndash;1980) and [[Stefan Bergman]] (1895&ndash;1977) in 1950.

In this article we assume that Hilbert spaces are [[complex number|complex]].  The main reason for this is that many of the examples of reproducing kernel Hilbert spaces are spaces of [[analytic function]]s, although some [[real number|real]] Hilbert spaces also have reproducing kernels.

An important subset of the reproducing kernel Hilbert spaces are the reproducing kernel Hilbert spaces associated to a continuous kernel. These spaces have wide applications, including [[complex analysis]], [[harmonic analysis]], [[quantum mechanics]], [[statistics]] and [[machine learning]].

==Definition==
Let ''X'' be an arbitrary [[Set (mathematics)|set]] and ''H'' a [[Hilbert space]] of [[complex-valued function]]s on ''X''.  We say that ''H'' is a '''reproducing kernel Hilbert space''' if the [[linear map]] 

:<math> L_{x} : f \mapsto f(x) </math>

from ''H'' to the complex numbers is [[continuous function (topology)|continuous]] for any ''x'' in ''X''; equivalently, if <math> L_x </math> is a [[Bounded operator]] for any ''x'' in ''X''.  By the [[Riesz representation theorem]], this implies that for every ''x'' in ''X'' there exists a unique element ''K''<sub>''x''</sub> of ''H'' with the property that:

:<math>  f(x) = L_{x}(f) = \langle f,\ K_x \rangle \quad \forall f \in H \quad (*).</math>

The function ''K''<sub>''x''</sub> is called the point-evaluation function at the point ''x''.

Since ''H'' is a space of functions, the element ''K''<sub>''x''</sub> is itself a function with [[domain (mathematics)|domain]] ''X'', and may be written as ''K''<sub>''x''</sub>''(y)''. The family of functions ''K''<sub>''x''</sub>''(y)'' may be encoded into a single function ''K'' : ''X'' × ''X'' → '''C''' by defining

:<math> K(x,y) \ \stackrel{\mathrm{def}}{=}\   \overline{K_x(y)}. </math>

This function is called the '''reproducing kernel for the Hilbert space ''H''''' and it is determined entirely by ''H'' because the Riesz representation theorem guarantees, for every ''x'' in ''X'', that the element ''K''<sub>''x''</sub> satisfying (*) is unique.

== Properties ==

=== The Reproducing Property ===
It is clear from the discussion above that 

:<math> K(x,y) \;=\; \overline{K(y,x)} \;=\; \langle K_y,K_x\rangle. </math>

In particular, 

:<math> K(x,x) \;=\; \langle K_x, K_x \rangle \;\geq\; 0, \quad \forall x\in X. </math>

Note that 

:<math> K_x \;=\; 0 \quad \text{ if and only if } \quad f(x) = 0 \quad \forall \; f\in H. </math>

=== Orthonormal Bases ===
If {φ<sub>1</sub>, φ<sub>2</sub>, ...} is an orthonormal basis of ''H'', then

:<math> K(x,y)  =\sum_{k=1}^{\infty}\varphi_k (x) \overline{\varphi_k (y)}. </math>

== Reproducing Kernel==
===Definition ===

Let X be some set. A function <math> K: X \times X \rightarrow \mathbb{R} </math> is a '''reproducing kernel''' if it is symmetric and positive definite:

<math> \sum_{i,j = 1}^n c_ic_jK(t_i,t_j) \geq 0 </math>

We have now defined what a reproducing kernel is in its own right. And we can check that K, the '''reproducing kernel for the Hilbert space ''H''''' above, is in fact a '''reproducing kernel'''. 

-Symmetry follows from the symmetry property of inner products: <math> <K_t, K_x>_H = <K_x,K_t>_H </math>. 

-Positive-definiteness follows because <math> \sum_{i,j = 1}^n c_ic_jK(t_i,t_j) = \sum_{i,j = 1}^n c_ic_j<K_{t_i},K_{t_j}>_H = \|\sum_{j=1}^nc_jK_{t_j}\|_H^2 \geq 0 </math>

== Moore-Aronszajn Theorem ==
We have seen how a reproducing kernel Hilbert space defines reproducing kernel function. It follows from the definition of an inner product that the kernel defined is (conjugate) symmetric and [[positive definite kernel|positive definite]]. The Moore-Aronszajn theorem goes in the other direction; it says that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's ''Theory of Reproducing Kernels'', although he attributes it to [[E. H. Moore]].

<blockquote>'''Theorem'''. Suppose ''K'' is a symmetric, [[positive definite kernel]] on a set ''E''. Then there is a unique Hilbert space of functions on ''E'' for which ''K'' is a reproducing kernel.</blockquote>

'''Proof'''. For all ''x'' in ''E'', define ''K<sub>x</sub>'' = ''K''(''x'', ⋅ ). Let ''H''<sub>0</sub> be the linear span of {''K<sub>x</sub>'' : ''x'' ∈ ''E''}. Define an inner product on ''H''<sub>0</sub> by
:<math>\left\langle \sum_{j=1}^n b_j K_{y_j}, \sum_{i=1}^m a_i K_{x_i} \right \rangle = \sum_{i=1}^m \sum_{j=1}^n \overline{a_i} b_j K(y_j, x_i).
</math>
The symmetry of this inner product follows from the symmetry of ''K'' and the non-degeneracy follows from the fact that ''K'' is positive definite.

Let ''H'' be the [[Completion (metric space)|completion]] of ''H''<sub>0</sub> with respect to this inner product. Then ''H'' consists of functions of the form

:<math> f(x) = \sum_{i=1}^\infty a_i K_{x_i} (x) </math>

where <math>\sum_{i=1}^\infty a_i^2 K (x_i, x_i) < \infty</math>. The fact that the above sum converges for every ''x'' follows from the [[Cauchy-Schwarz inequality]].

Now we can check the RKHS property, (*):

:<math>\langle f, K_x \rangle = \left \langle \sum_{i=1}^\infty a_i K_{x_i}, K_x \right \rangle= \sum_{i=1}^\infty a_i K (x_i, x) = f(x).</math>

To prove uniqueness, let ''G'' be another Hilbert space of functions for which ''K'' is a reproducing kernel. For any ''x'' and ''y'' in ''E'', (*) implies that

:<math>\langle K_x, K_y \rangle_H = K(x, y) = \langle K_x, K_y \rangle_G. \, </math>

By linearity, <math>\langle \cdot, \cdot \rangle_H = \langle \cdot, \cdot \rangle_G</math> on the span of {''K<sub>x</sub>'' : ''x'' ∈ ''E''}. Then ''G'' = ''H'' by the uniqueness of the completion.

==Integral Operators and Mercer Theorem==
We may characterize a symmetric positive semi-definite kernel K via the integral operator, using the [[Mercer's theorem]]. 

Suppose K is a continuous positive semi-definite kernel on a compact set <math> X\subset\mathbb{R}^n, \mu </math> a Borel measure on <math>X</math>, 
and the integral operator <math> T_K: L_2(X) \rightarrow L_2(X) </math> defined by 
<math> [T_K f](\cdot) =\int_X  K(\cdot,t) f(t)\, d\mu(t) </math> is positive semi-definite. 

By the Mercer Theorem, there is an orthonormal basis <math> \{\phi_i\}_i </math> of <math> L_2(X) </math> consisting of eigenfunctions of <math> T_K </math> such that the corresponding sequence of eigenvalues <math> \{\sigma_i\}_i </math> is nonnegative. The eigenfunctions corresponding to non-zero eigenvalues are continuous on X and K has the representation

<math> K(s,t) = \sum_{j=1}^\infty \sigma_j \, \phi_j(s) \, \phi_j(t) </math>

where the convergence is absolute and uniform, that is, <math> \lim_{n \to \infty}\sup_{u,v}|K(u,v) - \sum_{j=1}^n \sigma_j \, \phi_j(u) \, \phi_j(v)| = 0 </math>

If X is a compact set, then <math> T_K </math> is a compact, positive, and self-adjoint operator, which yields
a decreasing sequence <math> (\sigma_i)_i \geq 0 </math> such that <math> \lim_{i \to \infty}\sigma_i = 0 </math> and
<math> T_K\phi(x) = \sigma_i\phi_i(x) </math> , where the <math> \{\phi_i\} </math> form an orthonormal basis of <math> L_2(X) </math>. 

This means that <math> \forall{f}{\in}L_2(X) \ f = \sum_{i=1}^\infty <f,\phi_i>\phi_i </math>, so that the action of <math> T_K </math> can be written as <math> [T_K f]  = \sum_{i=1}^\infty \sigma_i<f,\phi_i>\phi_i </math>

Furthermore, a Reproducing Kernel Hilbert Space <math> H </math> can now be constructed, with <math> K </math> as the reproducing kernel. 

It can be shown to be <math> H = \{f {\in}L_2(X)\mathrel{\Bigg|} \sum_{i=1}^\infty\frac{<f,\phi_i>^2}{\sigma_i} < \infty\} </math>, 

with the inner product of <math> H </math> given by <math> <f,g>_H = \sum_{i=1}^\infty \frac{<f,\phi_i>_{L_2}<g,\phi_i>_{L_2}}{\sigma_i} </math>

==Feature Map==
{{main|Kernel trick}}
A feature map is a map <math> \varphi: X \rightarrow F </math>, where <math> F </math> is a Hilbert space called the Feature Space. 

Every feature map defines a kernel via <math> K(x,s) = <\varphi(x), \varphi(s)> </math> 

For example, for the reproducing kernel in the previous section, we can define <math> \varphi(x) = (\sqrt(\sigma_i)\phi_i(x))_i </math>. We can always associate a feature map to every kernel; in fact, there exist infinitely many feature maps associated with <math> K </math>. 

With feature maps, we can now view functions as hyperplanes in the Feature Space. For example, for any <math> x{\in}X </math>,
<math> \{w {\in}F \mathrel{|} f(x) = <w, \varphi(x)>\}</math> is a hyperplane.

In machine learning, kernel-based procedures allow us to map the data from the input space X into the Feature Space where linear methods may then be used.

== Examples ==
'''Linear Kernel''':
<math> K(x,y) = <x,y> </math>

'''Radial Basis Function Kernels''':

These are kernels which satisfy <math> K(x,y) = K(\|x - y\|) </math>

:-'''Gaussian Kernel''':
:Sometimes referred to as the [[Radial basis function kernel]],
:<math> K(x,y) = e^{-\frac{\|x - y\|^2}{2\sigma^2}}, \sigma > 0 </math>

:-'''Polynomial Kernel''':
:<math> K(x,y) = (\alpha\langle x,y \rangle + 1)^d, \alpha {\in} \mathbb{R}, d {\in} \mathbb{N} </math>

:-'''Exponential Kernel''':
:<math> K(x,y) = e^{-\frac{\|x - y\|}{2\sigma^2}}, \sigma > 0 </math>

:-'''Laplacian Kernel''':
:<math> K(x,y) = e^{-\frac{\|x - y\|}{\sigma}}, \sigma > 0 </math>

In another example, when ''X'' is finite and ''H'' consists of all complex-valued functions on ''X'', then an element of ''H'' can be represented as an array of complex numbers. If the usual [[inner product]] is used, then ''K<sub>x</sub>'' is the function whose value is 1 at ''x'' and 0 everywhere else, and ''K(x,y)'' can be thought of as an identity matrix since ''K(x,y)=1'' when ''x=y'' and ''K(x,y)=0'' otherwise. In this case, ''H'' is isomorphic to '''C'''<sup>''n''</sup>.

The case of ''X'' = '''[[unit disc|D]]''' is more sophisticated, here the [[Bergman space]] [[H square|''H''<sup>2</sup>('''D''')]] is the space of [[square-integrable function|square-integrable]] [[holomorphic]] functions on '''D'''. It can be shown that the reproducing kernel for ''H''<sup>2</sup>('''D''') is

:<math>K(x,y)=\frac{1}{\pi}\frac{1}{(1-x\overline{y})^2}.</math>

This kernel is an example of a [[Bergman kernel]], named for [[Stefan Bergman]].

== See also ==
*[[Positive definite kernel]]
*[[Mercer's theorem]]
*[[Kernel trick]]
*[[Kernel embedding of distributions]]

==Notes==
{{Reflist}}

==References==
* {{cite journal
 |first=Nachman |last=Aronszajn |authorlink=Nachman Aronszajn
 |title=Theory of Reproducing Kernels
 |journal=[[Transactions of the American Mathematical Society]]
 |volume=68 |issue=3 |pages=337&ndash;404 |year=1950
 |jstor=1990404
 |mr=51437
 |doi=10.1090/S0002-9947-1950-0051437-7
}}
* Alain Berlinet and Christine Thomas, ''Reproducing kernel Hilbert spaces in Probability and Statistics'', Kluwer Academic Publishers, 2004.
* {{cite journal
 |first=George |last=Kimeldorf 
 |first2=Grace |last2=Wahba |authorlink2=Grace Wahba
 |url=http://www.stat.wisc.edu/~wahba/ftp1/oldie/kw71.pdf 
 |title=Some results on Tchebycheffian Spline Functions
 |journal=Journal of Mathematical Analysis and Applications
 |volume=33 |issue=1 |year=1971 |pages=82&ndash;95 |doi=10.1016/0022-247X(71)90184-3
 |mr=290013
}}
* [[Grace Wahba]],  ''Spline Models for Observational Data'', [http://www.siam.org/books/ SIAM], 1990.
* {{cite journal
 |first1=Felipe |last1=Cucker
 |first2=Steve |last2=Smale
 |title=On the Mathematical Foundations of Learning
 |journal=[[Bulletin of the American Mathematical Society]]
 |volume=39 |issue=1 |pages=1&ndash;49 |year=2002
 |doi=10.1090/S0273-0979-01-00923-5
 |mr=1864085
}}

[[Category:Hilbert space]]