Probability
Probability theory provides a mathematical foundation to concepts such as
“probability”, “information”, “belief”, “uncertainty”, “confidence”,
“randomness”, “variability”, “chance” and “risk”. Probability theory is
important to empirical scientists because it gives them a rational
framework to make inferences and test hypotheses based on uncertain
empirical data. Probability theory is also useful to engineers building
systems that have to operate intelligently in an uncertain world. For
example, some of the most successful approaches in machine perception
(e.g., automatic speech recognition, computer vision) and artificial
intelligence are based on probabilistic models. Moreover probability
theory is also proving very valuable as a theoretical framework for
scientists trying to understand how the brain works. Many computational
neuroscientists think of the brain as a probabilistic computer built with
unreliable components, i.e., neurons, and use probability theory as a
guiding framework to understand the principles of computation used by
the brain. Consider the following examples:
• You need to decide whether a coin is loaded (i.e., whether it tends to
favor one side over the other when tossed). You toss the coin 6 times
and in all cases you get “Tails”. Would you say that the coin is
loaded?
• You are trying to figure out whether newborn babies can distinguish
green from red. To do so you present two colored cards (one green,
one red) to 6 newborn babies. You make sure that the 2 cards have
equal overall luminance so that they are indistinguishable if recorded
by a black and white camera. The 6 babies are randomly divided into
two groups. The first group gets the red card on the left visual field,
and the second group on the right
7
,visual field. You find that all 6 babies look longer to the red card than the green
card. Would you say that babies can distinguish red from green?
• A pregnancy test has a 99 % validity (i.e., 99 of of 100 pregnant womepositive)
and 95 % specificity (i.e., 95 out of 100 non pregnant women test negative). A
woman believes she has a 10 % chance of being pregnant. She takes the test and
tests positive. How should she combine her prior beliefs with the results of the
test?
• You need to design a system that detects a sinusoidal tone of 1000Hz in the
presence of white noise. How should design the system to solve this task
optimally?
• How should the photoreceptors in the human retina be interconnected to
maximize information transmission to the brain?
While these tasks appear different from each other, they all share a common
problem: The need to combine different sources of uncertain information to make
rational decisions. Probability theory provides a very powerful mathematical
framework to do so. Before we go into mathematical aspects of probability theory I
shall tell you that there are deep philosophical issues behind the very notion of
probability. In practice there are three major interpretations of probability,
commonly called the frequentist, the Bayesian or subjectivist, and the axiomatic or
mathematical interpretation.
1. Probability as a relative frequency
This approach interprets the probability of an event as the proportion of times such
an event is expected to happen in the long run. Formally, the probability of an
event E would be the limit of the relative frequency of occurrence of that event as
the number of observations grows large
P(E) = limn→∞nE
where nE is the number of times the event is observed out of a total of n
independent experiments. For example, we say that the probability of “heads”
when tossing a coin is 0.5. By that we mean that if we toss a coin many many
times and compute the relative frequency of “heads” we expect for that relative
frequency to approach 0.5 as we increase the number of tosses.
,This notion of probability is appealing because it seems objective and ties
our work to the observation of physical events. One difficulty with the ap
proach is that in practice we can never perform an experiment an infinite
number of times. Note also that this approach is behaviorist, in the sense
that it defines probability in terms of the observable behavior of physical
systems. The approach fails to capture the idea of probability as internal
knowledge of cognitive systems.
2. Probability as uncertain knowledge.
This notion of probability is at work when we say things like “I will
probably get an A in this class”. By this we mean something like
“Based on what I know about myself and about this class, I would
not be very surprised if I get an A. However, I would not bet my life
on it, since there are a multitude of factors which are difficult to
predict and that could make it impossible for me to get an A”. This
notion of probability is “cognitive” and does not need to be directly
grounded on empirical frequencies. For example, I can say things
like “I will probably die poor” even though I will not be able to
repeat my life many times and count the number of lives in which I
die poor.
This notion of probability is very useful in the field of machine
intelligence. In order for machines to operate in natural
environments they need knowledge systems capable of handling the
uncertainty of the world. Probability theory provides an ideal way to
do so. Probabilists that are willing to represent internal knowledge
using probability theory are called “Bayesian”, since Bayes is
recognized as the first mathematician to do so.
, 3.Probability as a mathematical model.
Modern mathematicians avoid the frequentist vs. Bayesian controversy
by treating probability as a mathematical object. The role of
mathematics here is to make sure probability theory is rigorously
defined and traceable to first principles. From this point of view it is
up to the users of probability theory to apply it to whatever they see
fit. Some may want to apply it to describe limits of relative
frequencies. Some may want to apply it to describe subjective
notions of uncertainty, or to build better computers. This is not
necessarily of concern to the mathematician. The application of
probability theory to those domains will be ultimately judged by its
usefulness.
Intuitive Set Theory
We need a few notions from set theory before we jump into probability theory. In doing so
we will use intuitive or “naive” definitions. This intuitive approach provides good
mnemonics and is sufficient for our purposes but soon runs into problems for more
advanced applications. For a more rigorous definition of set theoretical concepts and an
explanation of the limitations of the intuitive approach you may want to take a look at the
Appendix.
• Set: A set is a collection of elements. Sets are commonly represented using curly
brackets containing a collection of elements separated by commas. For example
A = {1, 2, 3}
tells us that A is a set whose elements are the first 3 natural numbers. Sets can also be
represented using a rule that identifies the elements of the set. The prototypical notation is
as follows
{x : x follows a rule}
Probability theory provides a mathematical foundation to concepts such as
“probability”, “information”, “belief”, “uncertainty”, “confidence”,
“randomness”, “variability”, “chance” and “risk”. Probability theory is
important to empirical scientists because it gives them a rational
framework to make inferences and test hypotheses based on uncertain
empirical data. Probability theory is also useful to engineers building
systems that have to operate intelligently in an uncertain world. For
example, some of the most successful approaches in machine perception
(e.g., automatic speech recognition, computer vision) and artificial
intelligence are based on probabilistic models. Moreover probability
theory is also proving very valuable as a theoretical framework for
scientists trying to understand how the brain works. Many computational
neuroscientists think of the brain as a probabilistic computer built with
unreliable components, i.e., neurons, and use probability theory as a
guiding framework to understand the principles of computation used by
the brain. Consider the following examples:
• You need to decide whether a coin is loaded (i.e., whether it tends to
favor one side over the other when tossed). You toss the coin 6 times
and in all cases you get “Tails”. Would you say that the coin is
loaded?
• You are trying to figure out whether newborn babies can distinguish
green from red. To do so you present two colored cards (one green,
one red) to 6 newborn babies. You make sure that the 2 cards have
equal overall luminance so that they are indistinguishable if recorded
by a black and white camera. The 6 babies are randomly divided into
two groups. The first group gets the red card on the left visual field,
and the second group on the right
7
,visual field. You find that all 6 babies look longer to the red card than the green
card. Would you say that babies can distinguish red from green?
• A pregnancy test has a 99 % validity (i.e., 99 of of 100 pregnant womepositive)
and 95 % specificity (i.e., 95 out of 100 non pregnant women test negative). A
woman believes she has a 10 % chance of being pregnant. She takes the test and
tests positive. How should she combine her prior beliefs with the results of the
test?
• You need to design a system that detects a sinusoidal tone of 1000Hz in the
presence of white noise. How should design the system to solve this task
optimally?
• How should the photoreceptors in the human retina be interconnected to
maximize information transmission to the brain?
While these tasks appear different from each other, they all share a common
problem: The need to combine different sources of uncertain information to make
rational decisions. Probability theory provides a very powerful mathematical
framework to do so. Before we go into mathematical aspects of probability theory I
shall tell you that there are deep philosophical issues behind the very notion of
probability. In practice there are three major interpretations of probability,
commonly called the frequentist, the Bayesian or subjectivist, and the axiomatic or
mathematical interpretation.
1. Probability as a relative frequency
This approach interprets the probability of an event as the proportion of times such
an event is expected to happen in the long run. Formally, the probability of an
event E would be the limit of the relative frequency of occurrence of that event as
the number of observations grows large
P(E) = limn→∞nE
where nE is the number of times the event is observed out of a total of n
independent experiments. For example, we say that the probability of “heads”
when tossing a coin is 0.5. By that we mean that if we toss a coin many many
times and compute the relative frequency of “heads” we expect for that relative
frequency to approach 0.5 as we increase the number of tosses.
,This notion of probability is appealing because it seems objective and ties
our work to the observation of physical events. One difficulty with the ap
proach is that in practice we can never perform an experiment an infinite
number of times. Note also that this approach is behaviorist, in the sense
that it defines probability in terms of the observable behavior of physical
systems. The approach fails to capture the idea of probability as internal
knowledge of cognitive systems.
2. Probability as uncertain knowledge.
This notion of probability is at work when we say things like “I will
probably get an A in this class”. By this we mean something like
“Based on what I know about myself and about this class, I would
not be very surprised if I get an A. However, I would not bet my life
on it, since there are a multitude of factors which are difficult to
predict and that could make it impossible for me to get an A”. This
notion of probability is “cognitive” and does not need to be directly
grounded on empirical frequencies. For example, I can say things
like “I will probably die poor” even though I will not be able to
repeat my life many times and count the number of lives in which I
die poor.
This notion of probability is very useful in the field of machine
intelligence. In order for machines to operate in natural
environments they need knowledge systems capable of handling the
uncertainty of the world. Probability theory provides an ideal way to
do so. Probabilists that are willing to represent internal knowledge
using probability theory are called “Bayesian”, since Bayes is
recognized as the first mathematician to do so.
, 3.Probability as a mathematical model.
Modern mathematicians avoid the frequentist vs. Bayesian controversy
by treating probability as a mathematical object. The role of
mathematics here is to make sure probability theory is rigorously
defined and traceable to first principles. From this point of view it is
up to the users of probability theory to apply it to whatever they see
fit. Some may want to apply it to describe limits of relative
frequencies. Some may want to apply it to describe subjective
notions of uncertainty, or to build better computers. This is not
necessarily of concern to the mathematician. The application of
probability theory to those domains will be ultimately judged by its
usefulness.
Intuitive Set Theory
We need a few notions from set theory before we jump into probability theory. In doing so
we will use intuitive or “naive” definitions. This intuitive approach provides good
mnemonics and is sufficient for our purposes but soon runs into problems for more
advanced applications. For a more rigorous definition of set theoretical concepts and an
explanation of the limitations of the intuitive approach you may want to take a look at the
Appendix.
• Set: A set is a collection of elements. Sets are commonly represented using curly
brackets containing a collection of elements separated by commas. For example
A = {1, 2, 3}
tells us that A is a set whose elements are the first 3 natural numbers. Sets can also be
represented using a rule that identifies the elements of the set. The prototypical notation is
as follows
{x : x follows a rule}
