NEURAL NETWORKS
1.0.0 Introduction
The recent rise of interest in neural networks has its roots in the recognition that the
brain performs computations in a different manner than do conventional digital computers.
Computers are extremely fast and precise at executing sequences of instructions that have
been formulated for them. A human information process sing system is composed of
neurons switching at speeds about a million times slower than computer gates. Yet, humans
are more efficient than computers at computationally complex tasks such as speech
understanding. Moreover, not only humans, but also even animals, can process visual
information better than the fastest computers.
Artificial neural systems, or neural networks (NN), are physical cellular systems,
which can acquire, store, and utilize experiential knowledge. The knowledge is in the form
of stable states or mappings embedded in networks that can be recalled in response to the
presentation cues. Neural network processing typically involves dealing with large-scale
problems in terms of dimensionality, amount of data handled, and the volume of simulation
or neural hardware processing. This large-scale approach is both essential and typical for
real-life applications. By keeping view of all these, the research community has made an
effort in designing and implementing the various neural network models for different
applications. Now let us formally define the basic idea of neural network:
Definition: A neural network is a computing system made up of a number of
simple, highly interconnected nodes or processing elements, which process information
by its dynamic state response to external inputs.
1.1.0 Humans and Computers
Human beings are more intelligent than computers. Computers could only do logical
things well. But in case of solving cross word puzzles, vision problem, controlling an arm to
pick it up or something similar, that requires exceptionally complex techniques. Like these
problems, human beings do better than computers.
Computers are designed to carry out one instruction after another, extremely rapid,
whereas our brains work with many slower units. Whereas a computer can typically carry
out a few million operations every second, the units in the brain respond about ten per
second. However, they work on many different things at once, which computer can’t do.
The computer is a high-speed, serial machine and is used as such, compared to the
slow, highly parallel nature of the brain. Counting is an essentially serial activity, as is
,adding, with the thing done one after another, and so the computer can beat the brain any
time. For vision, or speech recognition, the problem is a highly parallel one, with many
different and conflicting inputs, triggering many different and conflicting ideas and
memories, and it is only the combining of all these different factors that allow us to perform
such feats, but then, our brains are able to operate in parallel easily and so we leave the
computers far behind.
The conclusion that we can reach from all of this is that the problems that we are
trying to solve are immensely parallel ones.
1.2.0 History of artificial neural networks
• The field of neural networks is not new. The first formal definition of a synthetic
neuron model based on the highly simplified considerations of the biological model
proposed by McCulloch and Pitts in 1943. The McCulloch-Pitts (MP) neuron model
resembles what is known as a binary logic device.
• The next major development, after the MP neuron model was proposed, occurred in
1949, when D.O. Hebb proposed a learning mechanism for the brain that become the
starting point for artificial neural networks (ANN) learning (training) algorithms. He
postulated that as the brain learns, it changes its connectivity patterns.
• The idea of learning mechanism was first incorporated in ANN by E. Rosenblatt
1958.
• By introducing the least mean squares (LMS) learning algorithm, Widrow and Hoff
developed in 1960 a model of a neuron that learned quickly and accurately. This
model was called ADALINE for ADAptive LInear NEuron. The applications of
ADALINE and its extension to MADALINE (for Many ADALINES) include pattern
recognition, weather forecasting, and adaptive controls. The monograph on learning
machines by Nils Nilsson (1965) summarized the developments of that time.
• In 1969, research in the field of ANN suffered a serious setback. Minsky and Papert
published a book on perceptrons in which they proved that single layer neural
networks have limitations in their abilities to process data, and are capable of any
mapping that is linearly separable. They pointed out, carefully applying mathematical
techniques, that are logical Exclusive-OR (XOR) function could not be realized by
perceptrons.
• Further, Minsky and Papert argued that research into multi-layer neural networks
would be unproductive. Due to this pessimistic view of Minsky and Papert, the field
, of ANN entered into an almost total eclipse for nearly two decades. Fortunately,
Minsky and Papert’s judgment has been disapproved; multi-layer perceptron networks
can solve all nonlinear separable problems.
• Nevertheless, a few dedicated researchers such as Kohonen, Grossberg, Anderson and
Hopfield continued their efforts.
• The study of learning in networks of threshold elements and of the mathematical
theory of neural networks was pursued by Sun - Ichi – Amari (1972, 1977). Also
Kunihiko Fukushima developed a class of neural network architectures known as
neocognitrons in 1980.
• There have been many impressive demonstrations of ANN capabilities: a network has
been trained to convert text to phonetic representations, which were then converted to
speech by other means (Sejnowsky and Rosenberg 1987); other network can
recognize handwritten characters (Burr 1987); and a neural network based image-
compression system has been devised (Cottrell, Munro, and Zipser 1987). These all
use the backpropagation network, perhaps the most successful of the current
algorithms. Backpropagation, invented independently in three separate research
efforts (Werbos 1974, Parker 1982, and Rumelhart, Hinton and Williams 1986)
provides a systematic means for training multi-layer networks, thereby overcoming
limitations presented by Minsky.
1.3.0 Characteristics of ANN
Artificial neural networks are biologically inspired; that is, they are composed of
elements that perform in a manner that is analogous to the most elementary functions of the
biological neuron. The important characteristics of artificial neural networks are learning
from experience, generalize from previous examples to new ones, and abstract essential
characteristics from inputs containing irrelevant data.
1.3.1 Learning
The NNs learn by examples. Thus, NN architectures can be ‘trained’ with known
examples of a problem before they are tested for their ‘inference’ capability on unknown
instances of the problem. They can, therefore, identify new objects previously untrained.
ANN can modify their behavior in response to their environment. Shown a set of inputs
(perhaps with desired outputs), they self-adjust to produce consistent responses. A wide
variety of training algorithms has been discussed in later units.
, 1.3.2 Parallel operation
The NNs can process information in parallel, at high speed, and in a distributed
manner.
1.3.3 Mapping
The NNs exhibit mapping capabilities, that is, they can map input patterns to their associated
output patterns.
1.3.4 Generalization
The NNs possess the capability to generalize. Thus, they can predict new outcomes
from past trends. Once trained, a network’s response can be to a degree, insensitive to minor
variations in its input. This ability to see through noise and distortion to the pattern that lies
within is vital to pattern recognition in a real-world environment. It is important to note that
the ANN generalizes automatically as a result of its structure, not by using human
intelligence embedded in the form of adhoc computer programs.
1.3.5 Robust
The NNs are robust systems and are fault tolerant. They can, therefore, recall full
patterns from incomplete, partial or noisy patterns
1.3.6 Abstraction
Some ANN’s are capable of abstracting the essence of a set of inputs. i.e. they can
extract features of the given set of data, for example, convolution neural networks are used to
extract different features from images like edges, dark spots, shapes ..etc. Such networks are
trained for feature patterns based on which they can classify or cluster the given input set.
1.3.7 Applicability
ANN’s are not a panacea. They are clearly unsuited to such tasks as calculating the
payroll. They are preferred for a large class of pattern-recognition tasks that conventional
computers do poorly, if at all.
1.4.0 Applications
Neural networks are preferred when the task is related to large-amount data processing. The
following are the potential applications of neural networks:
• Classification
1.0.0 Introduction
The recent rise of interest in neural networks has its roots in the recognition that the
brain performs computations in a different manner than do conventional digital computers.
Computers are extremely fast and precise at executing sequences of instructions that have
been formulated for them. A human information process sing system is composed of
neurons switching at speeds about a million times slower than computer gates. Yet, humans
are more efficient than computers at computationally complex tasks such as speech
understanding. Moreover, not only humans, but also even animals, can process visual
information better than the fastest computers.
Artificial neural systems, or neural networks (NN), are physical cellular systems,
which can acquire, store, and utilize experiential knowledge. The knowledge is in the form
of stable states or mappings embedded in networks that can be recalled in response to the
presentation cues. Neural network processing typically involves dealing with large-scale
problems in terms of dimensionality, amount of data handled, and the volume of simulation
or neural hardware processing. This large-scale approach is both essential and typical for
real-life applications. By keeping view of all these, the research community has made an
effort in designing and implementing the various neural network models for different
applications. Now let us formally define the basic idea of neural network:
Definition: A neural network is a computing system made up of a number of
simple, highly interconnected nodes or processing elements, which process information
by its dynamic state response to external inputs.
1.1.0 Humans and Computers
Human beings are more intelligent than computers. Computers could only do logical
things well. But in case of solving cross word puzzles, vision problem, controlling an arm to
pick it up or something similar, that requires exceptionally complex techniques. Like these
problems, human beings do better than computers.
Computers are designed to carry out one instruction after another, extremely rapid,
whereas our brains work with many slower units. Whereas a computer can typically carry
out a few million operations every second, the units in the brain respond about ten per
second. However, they work on many different things at once, which computer can’t do.
The computer is a high-speed, serial machine and is used as such, compared to the
slow, highly parallel nature of the brain. Counting is an essentially serial activity, as is
,adding, with the thing done one after another, and so the computer can beat the brain any
time. For vision, or speech recognition, the problem is a highly parallel one, with many
different and conflicting inputs, triggering many different and conflicting ideas and
memories, and it is only the combining of all these different factors that allow us to perform
such feats, but then, our brains are able to operate in parallel easily and so we leave the
computers far behind.
The conclusion that we can reach from all of this is that the problems that we are
trying to solve are immensely parallel ones.
1.2.0 History of artificial neural networks
• The field of neural networks is not new. The first formal definition of a synthetic
neuron model based on the highly simplified considerations of the biological model
proposed by McCulloch and Pitts in 1943. The McCulloch-Pitts (MP) neuron model
resembles what is known as a binary logic device.
• The next major development, after the MP neuron model was proposed, occurred in
1949, when D.O. Hebb proposed a learning mechanism for the brain that become the
starting point for artificial neural networks (ANN) learning (training) algorithms. He
postulated that as the brain learns, it changes its connectivity patterns.
• The idea of learning mechanism was first incorporated in ANN by E. Rosenblatt
1958.
• By introducing the least mean squares (LMS) learning algorithm, Widrow and Hoff
developed in 1960 a model of a neuron that learned quickly and accurately. This
model was called ADALINE for ADAptive LInear NEuron. The applications of
ADALINE and its extension to MADALINE (for Many ADALINES) include pattern
recognition, weather forecasting, and adaptive controls. The monograph on learning
machines by Nils Nilsson (1965) summarized the developments of that time.
• In 1969, research in the field of ANN suffered a serious setback. Minsky and Papert
published a book on perceptrons in which they proved that single layer neural
networks have limitations in their abilities to process data, and are capable of any
mapping that is linearly separable. They pointed out, carefully applying mathematical
techniques, that are logical Exclusive-OR (XOR) function could not be realized by
perceptrons.
• Further, Minsky and Papert argued that research into multi-layer neural networks
would be unproductive. Due to this pessimistic view of Minsky and Papert, the field
, of ANN entered into an almost total eclipse for nearly two decades. Fortunately,
Minsky and Papert’s judgment has been disapproved; multi-layer perceptron networks
can solve all nonlinear separable problems.
• Nevertheless, a few dedicated researchers such as Kohonen, Grossberg, Anderson and
Hopfield continued their efforts.
• The study of learning in networks of threshold elements and of the mathematical
theory of neural networks was pursued by Sun - Ichi – Amari (1972, 1977). Also
Kunihiko Fukushima developed a class of neural network architectures known as
neocognitrons in 1980.
• There have been many impressive demonstrations of ANN capabilities: a network has
been trained to convert text to phonetic representations, which were then converted to
speech by other means (Sejnowsky and Rosenberg 1987); other network can
recognize handwritten characters (Burr 1987); and a neural network based image-
compression system has been devised (Cottrell, Munro, and Zipser 1987). These all
use the backpropagation network, perhaps the most successful of the current
algorithms. Backpropagation, invented independently in three separate research
efforts (Werbos 1974, Parker 1982, and Rumelhart, Hinton and Williams 1986)
provides a systematic means for training multi-layer networks, thereby overcoming
limitations presented by Minsky.
1.3.0 Characteristics of ANN
Artificial neural networks are biologically inspired; that is, they are composed of
elements that perform in a manner that is analogous to the most elementary functions of the
biological neuron. The important characteristics of artificial neural networks are learning
from experience, generalize from previous examples to new ones, and abstract essential
characteristics from inputs containing irrelevant data.
1.3.1 Learning
The NNs learn by examples. Thus, NN architectures can be ‘trained’ with known
examples of a problem before they are tested for their ‘inference’ capability on unknown
instances of the problem. They can, therefore, identify new objects previously untrained.
ANN can modify their behavior in response to their environment. Shown a set of inputs
(perhaps with desired outputs), they self-adjust to produce consistent responses. A wide
variety of training algorithms has been discussed in later units.
, 1.3.2 Parallel operation
The NNs can process information in parallel, at high speed, and in a distributed
manner.
1.3.3 Mapping
The NNs exhibit mapping capabilities, that is, they can map input patterns to their associated
output patterns.
1.3.4 Generalization
The NNs possess the capability to generalize. Thus, they can predict new outcomes
from past trends. Once trained, a network’s response can be to a degree, insensitive to minor
variations in its input. This ability to see through noise and distortion to the pattern that lies
within is vital to pattern recognition in a real-world environment. It is important to note that
the ANN generalizes automatically as a result of its structure, not by using human
intelligence embedded in the form of adhoc computer programs.
1.3.5 Robust
The NNs are robust systems and are fault tolerant. They can, therefore, recall full
patterns from incomplete, partial or noisy patterns
1.3.6 Abstraction
Some ANN’s are capable of abstracting the essence of a set of inputs. i.e. they can
extract features of the given set of data, for example, convolution neural networks are used to
extract different features from images like edges, dark spots, shapes ..etc. Such networks are
trained for feature patterns based on which they can classify or cluster the given input set.
1.3.7 Applicability
ANN’s are not a panacea. They are clearly unsuited to such tasks as calculating the
payroll. They are preferred for a large class of pattern-recognition tasks that conventional
computers do poorly, if at all.
1.4.0 Applications
Neural networks are preferred when the task is related to large-amount data processing. The
following are the potential applications of neural networks:
• Classification
