ENCE 742 Affan Probability ITDS Assignment 1-5 University of Maryland

Information and Data Science Project
Introduction/ Scope

The document serves as a place holder for the ITDS assignment 1 to 5

Content

1. Assignment 1 - - - - - - - - - 2

Entropy
Test Entropy
Joint Entropy
Conditional Entropy Mutual Entropy
Normalized Version of entropy and mutual information

2. Assignment 2 - - -- - - - - - - 6

Difference between entropy of Discrete RV
Differential Entropy
Differential Entropy of Gaussian and Estimated PDF

3. Assignment 3 - - - - - - - - - 10

PMF of Iris features
Entropy of the features of Irish dataset
Mutual information between pairs of features

4. Assignment 4 - - - - - - - - - 12

Bayes Classifier Function: continuous Random variable (RV)
Naïve Bayes Classifier function for continuous and independent RV
Naïve Bayes Classifier function for continuous, independent, and Gaussian
distributed RV
Computation of Comparism of average accuracy

===============================================
==================
Annexes - - - - - - - - - 16

List of the scripts and functions

References




Chapter 1

Assignment 1

1/20

,This assignment request for the following tasks

1. A function called ’entropy" which computes the entropy of a discrete random
variable given its probability mass function [p1; p2; :::; pN].

2. A script called ‘test entropy2’ which computes the entropy for a generic
binary random variable as a function of p0 and plots the entropy function.

3. write a function called joint entropy" which computes the joint entropy of two
generic discrete random variables given their joint p.d.f.

4. write a function called conditional entropy" which computes the conditional
entropy of two generic discrete random variables given their joint and
marginal p.d.f.

5. write a function called mutual information" which computes the mutual
information of two generic discrete random variables given their joint and
marginal p.d.f.

6. write the functions for normalized versions of conditional entropy, joint
entropy and mutual information for the discrete case.

Entropy
A function called ’entropy" computes the entropy of a discrete random variable
given its probability mass function pmf. It accepts an array probability vector and
outputs a single value for the entropy.

The function performs in accordance to the following equation
Nx Nx
1
H ( x )=H NX ( P1 , P 2 , … , PNx ) ≝ ∑ p x∗log =−∑ p x∗log px
k=1 px k=1
Where
p x =probability of a particular outcome
H ( x )=The Entropy of the univeriate pmf

Running the script

fX = [0.2, 0.3, 0.1, 0.1, 0.1, 0.2]

Hx = entropy.entropy(fX)
print('The entropy of the given pmf fX = [0.2, 0.3, 0.1, 0.1, 0.1, 0.2] is : ',
Hx)

>>> output
The entropy of the given pmf fX = [0.2, 0.3, 0.1, 0.1, 0.1, 0.2]is :
2.4464393446710155

Test entropy2

2/20

, The ‘test_entropy2’ is a script that computes the entropy for a generic binary
random variable as function of p0 and plots the entropy function. The script request
for an input of a singular value of p for the range ( 0 < p < 1) and yield an output.

It also plots all possible distribution of the range 0 < p < 1

For the random variable X, X = 0 (HEAD), X = 1 (TAIL)

P ( X=0 )= p , P ( X=1 )=q
N
sumation of a probabilities ∑ Px=1
i=1
→ q=1− p
1 1
We can compute theentropy H ( x )= p∗log 2 + ( 1−p ) log 2 [bits]
p ( 1− p )

Running the script

What is the value of your random probability? P0 = 0.3

>>> output

0.8812908992306926




Joint entropy
This function called “joint_ entropy" computes the joint entropy of two generic
discrete random variables given their joint p.d.f.

The function follows the following equation.
+∞ +∞
1
Joint entropy h ( X , Y )= ∫ ∫ f XY ( x , y ) log 2
−∞ −∞ f XY ( x , y )



3/20