Data Mining (62 page)

and

where m is the number of inputs for j
^th
neuron. Also, we use the symbol v as a shorthand notation of the previously defined variable
net
. The backpropagation algorithm applies a correction Δw
_ji
(
n
) to the synaptic weight w
_ji
(
n
), which is proportional to the partial derivative δE(
n
)/δw
_ji
(
n
). Using the chain rule for derivation, this partial derivative can be expressed as

The partial derivative δE(
n
)/δwji(
n
) represents a sensitive factor, determining the direction of search in weight space. Knowing that the next relations

are valid, we can express the partial derivative ∂E(
n
)/∂w
_ji
(
n
) in the form

The correction Δw
_ji
(
n
) applied to w
_ji
(
n
) is defined by the delta rule

where η is the learning-rate parameter of the backpropagation algorithm. The use of the minus sign accounts for gradient descent in weight space, that is, a direction for weight change that reduces the value E(
n
). Asking for φ′(v
_j
[
n
]) in the learning process is the best explanation for why we prefer continuous functions such as log-sigmoid and hyperbolic as a standard-activation function at the node level. Using the notation δ
_j
(
n
) = e
_j
(
n
)·
, where δ
_j
(
n
) is the
local gradient
, the final equation for w
_ji
(
n
) corrections is

The local gradient δ
_j
(
n
) points to the required changes in synaptic weights. According to its definition, the local gradient δ
_j
(
n
) for output neuron j is equal to the product of the corresponding error signal e
_j
(
n
) for that neuron and the derivative φ′(v
_j
[
n
]) of the associated activation function.

Derivative φ′(v
_j
[
n
]) can be easily computed for a standard activation function, where differentiation is the only requirement for the function. If the activation function is sigmoid, it means that in the form

the first derivative is