What is Linear Discriminant Analysis (LDA)

Generally we can say that Linear Discriminant Analysis is a dimensionality reduction technique like PCA( principle component analysis) but LDA is a supervised machine learning method while PCA is not. In this article I will add some points related to LDA and the steps to find LDA of a data set.

PCA VS LDA

Objective of LDA

1. LDA focuses on maximizing the separability among the know categories in the data
2. LDA try to create a new axis (lower dimension )and project the points into it using following criteria

a. Maximizing the distance between means of two classes

b. Minimizing the variation within classes

LD1 — The first new axis created using LDA which accounts for the most separability/variation between the classes

Like regression method there are some assumptions are there to perform the LDA they are

Assumptions of LDA

1. Each feature in the data set should be normally distributed
2. Each feature has the same variance, the value of each feature varies around the mean with the same amount on average
3. Each variable should be randomly sampled
4. There should be less/no multicollinearity between the features

Steps to find LDA

1. Compute the class means of dependent variable
2. Derive the covariance matrix of the class variable
3. Compute the within class — scatter matrix (S1+S2)
4. Compute the between class scatter matrix
5. Compute the Eigen values and eigen vectors from the within class and between class scatter matrix
6. Sort the values of eigen values and select the top k values
7. Select the eigen vectors corresponds to the top k eigen vectors
8. Obtain the LDA by taking the dot product of eigen vectors and original data

LDA is considered as a linear classification method like Logistic regression but there are some advantages over it,

Capabilities of LDA

1. LDA can be used when dependent variable having more than two classes
2. LDA works better when classes are well separated to each other
3. LDA work well with lower samples of data as well

I hope this article will give some ideas about LDA and its working

can check more detailed explanation here : What is LDA (Linear Discriminant Analysis) in Python (mygreatlearning.com)

Thanks for reading . Keep Learning !