Classical Neural Networks

The first logical neuron was developed by W. S. McCulloch and W.A. Pitts in 1943 [2]. It described the fundamentals functions and structures of a neural cell reporting that a neuron will fire an impulse only if a threshold value is exceeded.

Figure 1: McCulloch-Pitts neuron model.

Figure 1 shows the basic elements of McCulloch-Pitts model: $x$ is the input vector, $w$ is a weights vector, $y$ is output, $R$ is number of elements in input and $f$ is the activation function that determine the output value. A simple choice for $f$ is the signal function $sgn(.)$. In this case, the weights are used to calculate a weighted sum of the inputs. If it exceeds the threshold $b$ the output is $1$ else the value of $y$ is $-1$, that is:

 

$$y=sgn(\sum_{i=1}^{R}w_{i}x_{i}-b).$$ (1)

But the McCulloch-Pitts neuron did not have a mechanisms for learning. Based on biological evidences, D.O. Hebb suggested a rule to adapt the weights, that is, a learning rule [2]. This biological inspired procedure can be expressed in the following manner:

 

$$w_{i}^{new}=w_{i}^{old}+\Delta w_{i};\quad \Delta w_{i}=\eta(y^{desired}-y)x_{i},$$ (2)

where $w^{new}$ and $w^{old}$ are adapted weights and initials weights respectively, $\eta$ is a real parameter to control the rate of learning and $y^{desired}$ is the desired (know) output. This learning rule plus the elements of Figure 1 is called the perceptron model for a neuron. The learning typically occurs for example through training, or exposure to a known set of input/output data. The training algorithm iteratively adjusts the connection weights $\left\{ w_{i}\right\}$ analogous to synapses in biological nervous. These connection weights store the knowledge necessary to solve specific problems.