UNIT – 3: INTRODUCTION TO STATISTICAL LEARNING THEORY
FEATURE EXTRACTION
1. Principal Component Analysis: Principal Component Analysis is an unsupervised learning
algorithm that is used for the dimensionality reduction in machine learning. It is a statistical process
that converts the observations of correlated features into a set of linearly uncorrelated features with
the help of orthogonal transformation.
These new transformed features are called the Principal Components. It is a technique to draw strong
patterns from the given dataset by reducing the variances. The PCA algorithm is based on some
mathematical concepts such as:
➢ Variance and Covariance
➢ Eigenvalues and Eigen factors
Some common terms used in PCA algorithm:
(i) Dimensionality: It is the number of features or variables present in the given dataset. More easily,
it is the number of columns present in the dataset.
(ii) Correlation: It signifies that how strongly two variables are related to each other. Such as if one
changes, the other variable also gets changed. The correlation value ranges from -1 to +1. Here, -1
occurs if variables are inversely proportional to each other, and +1 indicates that variables are directly
proportional to each other.
(iii) Orthogonal: It defines that variable are not correlated to each other, and hence the correlation
between the pair of variables is zero.
(iv) Eigenvectors: If there is a square matrix M, and a non-zero vector v is given. Then v will be
eigenvector if Av is the scalar multiple of v.
(v) Covariance Matrix: A matrix containing the covariance between the pair of variables is called
the Covariance Matrix.
Principal Components in PCA: The transformed new features or the output of PCA are the
Principal Components. The number of these PCs are either equal to or less than the original features
present in the dataset. Some properties of these principal components are given below:
➢ The principal component must be the linear combination of the original features.
➢ These components are orthogonal, i.e., the correlation between a pair of variables is zero.
➢ The importance of each component decreases when going to 1 to n, it means the 1 PC has the
most importance, and n PC will have the least importance.
Steps for PCA algorithm
(i) Getting the dataset: Firstly, we need to take the input dataset and divide it into two subparts X
and Y, where X is the training set, and Y is the validation set.
(ii) Representing data into a structure: Now we will represent our dataset into a structure. Such as
we will represent the two-dimensional matrix of independent variable X. Here each row corresponds
to the data items, and the column corresponds to the Features. The number of columns is the
dimensions of the dataset.
, (iii) Standardizing the data: In this step, we will standardize our dataset. Such as in a particular
column, the features with high variance are more important compared to the features with lower
variance. If the importance of features is independent of the variance of the feature, then we will
divide each data item in a column with the standard deviation of the column. Here we will name the
matrix as Z.
(iv) Calculating the Covariance of Z: To calculate the covariance of Z, we will take the matrix Z,
and will transpose it. After transpose, we will multiply it by Z. The output matrix will be the
Covariance matrix of Z.
(v) Calculating the Eigen Values and Eigen Vectors: Now we need to calculate the eigenvalues
and eigenvectors for the resultant covariance matrix Z. Eigenvectors or the covariance matrix are the
directions of the axes with high information. And the coefficients of these eigenvectors are defined as
the eigenvalues.
(vi) Sorting the Eigen Vectors: In this step, we will take all the eigenvalues and will sort them in
decreasing order, which means from largest to smallest. And simultaneously sort the eigenvectors
accordingly in matrix P of eigenvalues. The resultant matrix will be named as P*.
(vii) Calculating the new features Or Principal Components: Here we will calculate the new
features. To do this, we will multiply the P* matrix to the Z. In the resultant matrix Z*, each
observation is the linear combination of original features. Each column of the Z* matrix is
independent of each other.
(viii) Remove less or unimportant features from the new dataset: The new feature set has
occurred, so we will decide here what to keep and what to remove. It means, we will only keep the
relevant or important features in the new dataset, and unimportant features will be removed out.
Example:
Step 1: Data
➢ We consider a dataset having n features or variables denoted by X1; X2; ……. ;Xn
➢ Let there be N examples
➢ Let the values of the ith features Xi be Xi1; Xi2; ………; XiN
Features Example 1 Example 2 ……. Example N
X1 X11 X12 ……. X1N
X2 X21 X22 ……. X2N
Xi Xi1 Xi2 ……. XiN
Xn Xn1 Xn2 ……. XnN
Step 2: Compute the means of the variables
Features Example 1 Example 2 ……. Example N
X1 X11 X12 ……. X1N
X2 X21 X22 ……. X2N
Xi Xi1 Xi2 ……. XiN
Xn Xn1 Xn2 ……. XnN
FEATURE EXTRACTION
1. Principal Component Analysis: Principal Component Analysis is an unsupervised learning
algorithm that is used for the dimensionality reduction in machine learning. It is a statistical process
that converts the observations of correlated features into a set of linearly uncorrelated features with
the help of orthogonal transformation.
These new transformed features are called the Principal Components. It is a technique to draw strong
patterns from the given dataset by reducing the variances. The PCA algorithm is based on some
mathematical concepts such as:
➢ Variance and Covariance
➢ Eigenvalues and Eigen factors
Some common terms used in PCA algorithm:
(i) Dimensionality: It is the number of features or variables present in the given dataset. More easily,
it is the number of columns present in the dataset.
(ii) Correlation: It signifies that how strongly two variables are related to each other. Such as if one
changes, the other variable also gets changed. The correlation value ranges from -1 to +1. Here, -1
occurs if variables are inversely proportional to each other, and +1 indicates that variables are directly
proportional to each other.
(iii) Orthogonal: It defines that variable are not correlated to each other, and hence the correlation
between the pair of variables is zero.
(iv) Eigenvectors: If there is a square matrix M, and a non-zero vector v is given. Then v will be
eigenvector if Av is the scalar multiple of v.
(v) Covariance Matrix: A matrix containing the covariance between the pair of variables is called
the Covariance Matrix.
Principal Components in PCA: The transformed new features or the output of PCA are the
Principal Components. The number of these PCs are either equal to or less than the original features
present in the dataset. Some properties of these principal components are given below:
➢ The principal component must be the linear combination of the original features.
➢ These components are orthogonal, i.e., the correlation between a pair of variables is zero.
➢ The importance of each component decreases when going to 1 to n, it means the 1 PC has the
most importance, and n PC will have the least importance.
Steps for PCA algorithm
(i) Getting the dataset: Firstly, we need to take the input dataset and divide it into two subparts X
and Y, where X is the training set, and Y is the validation set.
(ii) Representing data into a structure: Now we will represent our dataset into a structure. Such as
we will represent the two-dimensional matrix of independent variable X. Here each row corresponds
to the data items, and the column corresponds to the Features. The number of columns is the
dimensions of the dataset.
, (iii) Standardizing the data: In this step, we will standardize our dataset. Such as in a particular
column, the features with high variance are more important compared to the features with lower
variance. If the importance of features is independent of the variance of the feature, then we will
divide each data item in a column with the standard deviation of the column. Here we will name the
matrix as Z.
(iv) Calculating the Covariance of Z: To calculate the covariance of Z, we will take the matrix Z,
and will transpose it. After transpose, we will multiply it by Z. The output matrix will be the
Covariance matrix of Z.
(v) Calculating the Eigen Values and Eigen Vectors: Now we need to calculate the eigenvalues
and eigenvectors for the resultant covariance matrix Z. Eigenvectors or the covariance matrix are the
directions of the axes with high information. And the coefficients of these eigenvectors are defined as
the eigenvalues.
(vi) Sorting the Eigen Vectors: In this step, we will take all the eigenvalues and will sort them in
decreasing order, which means from largest to smallest. And simultaneously sort the eigenvectors
accordingly in matrix P of eigenvalues. The resultant matrix will be named as P*.
(vii) Calculating the new features Or Principal Components: Here we will calculate the new
features. To do this, we will multiply the P* matrix to the Z. In the resultant matrix Z*, each
observation is the linear combination of original features. Each column of the Z* matrix is
independent of each other.
(viii) Remove less or unimportant features from the new dataset: The new feature set has
occurred, so we will decide here what to keep and what to remove. It means, we will only keep the
relevant or important features in the new dataset, and unimportant features will be removed out.
Example:
Step 1: Data
➢ We consider a dataset having n features or variables denoted by X1; X2; ……. ;Xn
➢ Let there be N examples
➢ Let the values of the ith features Xi be Xi1; Xi2; ………; XiN
Features Example 1 Example 2 ……. Example N
X1 X11 X12 ……. X1N
X2 X21 X22 ……. X2N
Xi Xi1 Xi2 ……. XiN
Xn Xn1 Xn2 ……. XnN
Step 2: Compute the means of the variables
Features Example 1 Example 2 ……. Example N
X1 X11 X12 ……. X1N
X2 X21 X22 ……. X2N
Xi Xi1 Xi2 ……. XiN
Xn Xn1 Xn2 ……. XnN
