LDA calaulate - HackMD

# LDA calaulate ## How to make LDA Classifier If X∈R^d^ is a test data, the decsion function of LDA: ![](https://i.imgur.com/mJnUeBE.png) **Defined:** $$ H = \vec W^t \vec X + b $$ $$ \vec W \in R^d \vec X \in R^d $$ $\vec W$ : weight vector of the plane $\vec X$ : factor vector b : bias of the plane $\vec X$ is classified as positive if $D(X)=H = \vec W^t \vec X + b >0$ $\vec X$ is classified as negitive if $D(X)=H = \vec W^t \vec X + b <0$ **Note:** $\vec W, \vec X$ are unknown. (by data-driven way to ditermine) **LDA W & b are gain by:** $$\vec W =(\bar u_1 - \bar u_2)^t ∑^{-1}$$ $$\vec b =-\cfrac{1}{2}(\vec u_1-\vec u_2)^t∑^{-1}(\vec u_1-\vec u_2)-\ln\cfrac {c_{12} π_2}{c_{21} π_1} $$ **covariance is calculated:**$$∑=\cfrac{1}{N-1}\sum^N_{i=1}(\bar u_1 - \bar u_2)(\bar u_1 - \bar u_2)^t$$ **$π_1,π_2$ is calculated:**$$π_1=\cfrac{N_2}{N_1+N_2},π_2=\cfrac{N_1}{N_1+N_2}$$ **$C_{12},C_{21}$ is default as 1 & 1:** ![](https://i.imgur.com/PkrJsFB.png) **Note:** 1. $\vec u_1$ , $\vec u_2$ are mean vector of positive &negative training data, respective. 1. ∑ is the covering matrix of the training data 1. $π_1,π_2$ are priori probability(先驗水平) 1. $c_{12},c_{21}$ are the perally for the first class training error ## Matlab use LDA calaulate iris dataset **iris dataset:** 由左至右分別為特徵一、二、三、四及類別標籤 ``` 5.1000000e+000 3.5000000e+000 1.4000000e+000 2.0000000e-001 1 4.9000000e+000 3.0000000e+000 1.4000000e+000 2.0000000e-001 1 4.7000000e+000 3.2000000e+000 1.3000000e+000 2.0000000e-001 1 4.6000000e+000 3.1000000e+000 1.5000000e+000 2.0000000e-001 1 5.0000000e+000 3.6000000e+000 1.4000000e+000 2.0000000e-001 1 5.4000000e+000 3.9000000e+000 1.7000000e+000 4.0000000e-001 1 4.6000000e+000 3.4000000e+000 1.4000000e+000 3.0000000e-001 1 5.0000000e+000 3.4000000e+000 1.5000000e+000 2.0000000e-001 1 4.4000000e+000 2.9000000e+000 1.4000000e+000 2.0000000e-001 1 4.9000000e+000 3.1000000e+000 1.5000000e+000 1.0000000e-001 1 5.4000000e+000 3.7000000e+000 1.5000000e+000 2.0000000e-001 1 4.8000000e+000 3.4000000e+000 1.6000000e+000 2.0000000e-001 1 4.8000000e+000 3.0000000e+000 1.4000000e+000 1.0000000e-001 1 4.3000000e+000 3.0000000e+000 1.1000000e+000 1.0000000e-001 1 5.8000000e+000 4.0000000e+000 1.2000000e+000 2.0000000e-001 1 5.7000000e+000 4.4000000e+000 1.5000000e+000 4.0000000e-001 1 5.4000000e+000 3.9000000e+000 1.3000000e+000 4.0000000e-001 1 5.1000000e+000 3.5000000e+000 1.4000000e+000 3.0000000e-001 1 5.7000000e+000 3.8000000e+000 1.7000000e+000 3.0000000e-001 1 5.1000000e+000 3.8000000e+000 1.5000000e+000 3.0000000e-001 1 5.4000000e+000 3.4000000e+000 1.7000000e+000 2.0000000e-001 1 5.1000000e+000 3.7000000e+000 1.5000000e+000 4.0000000e-001 1 4.6000000e+000 3.6000000e+000 1.0000000e+000 2.0000000e-001 1 5.1000000e+000 3.3000000e+000 1.7000000e+000 5.0000000e-001 1 4.8000000e+000 3.4000000e+000 1.9000000e+000 2.0000000e-001 1 5.0000000e+000 3.0000000e+000 1.6000000e+000 2.0000000e-001 1 5.0000000e+000 3.4000000e+000 1.6000000e+000 4.0000000e-001 1 5.2000000e+000 3.5000000e+000 1.5000000e+000 2.0000000e-001 1 5.2000000e+000 3.4000000e+000 1.4000000e+000 2.0000000e-001 1 4.7000000e+000 3.2000000e+000 1.6000000e+000 2.0000000e-001 1 4.8000000e+000 3.1000000e+000 1.6000000e+000 2.0000000e-001 1 5.4000000e+000 3.4000000e+000 1.5000000e+000 4.0000000e-001 1 5.2000000e+000 4.1000000e+000 1.5000000e+000 1.0000000e-001 1 5.5000000e+000 4.2000000e+000 1.4000000e+000 2.0000000e-001 1 4.9000000e+000 3.1000000e+000 1.5000000e+000 1.0000000e-001 1 5.0000000e+000 3.2000000e+000 1.2000000e+000 2.0000000e-001 1 5.5000000e+000 3.5000000e+000 1.3000000e+000 2.0000000e-001 1 4.9000000e+000 3.1000000e+000 1.5000000e+000 1.0000000e-001 1 4.4000000e+000 3.0000000e+000 1.3000000e+000 2.0000000e-001 1 5.1000000e+000 3.4000000e+000 1.5000000e+000 2.0000000e-001 1 5.0000000e+000 3.5000000e+000 1.3000000e+000 3.0000000e-001 1 4.5000000e+000 2.3000000e+000 1.3000000e+000 3.0000000e-001 1 4.4000000e+000 3.2000000e+000 1.3000000e+000 2.0000000e-001 1 5.0000000e+000 3.5000000e+000 1.6000000e+000 6.0000000e-001 1 5.1000000e+000 3.8000000e+000 1.9000000e+000 4.0000000e-001 1 4.8000000e+000 3.0000000e+000 1.4000000e+000 3.0000000e-001 1 5.1000000e+000 3.8000000e+000 1.6000000e+000 2.0000000e-001 1 4.6000000e+000 3.2000000e+000 1.4000000e+000 2.0000000e-001 1 5.3000000e+000 3.7000000e+000 1.5000000e+000 2.0000000e-001 1 5.0000000e+000 3.3000000e+000 1.4000000e+000 2.0000000e-001 1 7.0000000e+000 3.2000000e+000 4.7000000e+000 1.4000000e+000 2 6.4000000e+000 3.2000000e+000 4.5000000e+000 1.5000000e+000 2 6.9000000e+000 3.1000000e+000 4.9000000e+000 1.5000000e+000 2 5.5000000e+000 2.3000000e+000 4.0000000e+000 1.3000000e+000 2 6.5000000e+000 2.8000000e+000 4.6000000e+000 1.5000000e+000 2 5.7000000e+000 2.8000000e+000 4.5000000e+000 1.3000000e+000 2 6.3000000e+000 3.3000000e+000 4.7000000e+000 1.6000000e+000 2 4.9000000e+000 2.4000000e+000 3.3000000e+000 1.0000000e+000 2 6.6000000e+000 2.9000000e+000 4.6000000e+000 1.3000000e+000 2 5.2000000e+000 2.7000000e+000 3.9000000e+000 1.4000000e+000 2 5.0000000e+000 2.0000000e+000 3.5000000e+000 1.0000000e+000 2 5.9000000e+000 3.0000000e+000 4.2000000e+000 1.5000000e+000 2 6.0000000e+000 2.2000000e+000 4.0000000e+000 1.0000000e+000 2 6.1000000e+000 2.9000000e+000 4.7000000e+000 1.4000000e+000 2 5.6000000e+000 2.9000000e+000 3.6000000e+000 1.3000000e+000 2 6.7000000e+000 3.1000000e+000 4.4000000e+000 1.4000000e+000 2 5.6000000e+000 3.0000000e+000 4.5000000e+000 1.5000000e+000 2 5.8000000e+000 2.7000000e+000 4.1000000e+000 1.0000000e+000 2 6.2000000e+000 2.2000000e+000 4.5000000e+000 1.5000000e+000 2 5.6000000e+000 2.5000000e+000 3.9000000e+000 1.1000000e+000 2 5.9000000e+000 3.2000000e+000 4.8000000e+000 1.8000000e+000 2 6.1000000e+000 2.8000000e+000 4.0000000e+000 1.3000000e+000 2 6.3000000e+000 2.5000000e+000 4.9000000e+000 1.5000000e+000 2 6.1000000e+000 2.8000000e+000 4.7000000e+000 1.2000000e+000 2 6.4000000e+000 2.9000000e+000 4.3000000e+000 1.3000000e+000 2 6.6000000e+000 3.0000000e+000 4.4000000e+000 1.4000000e+000 2 6.8000000e+000 2.8000000e+000 4.8000000e+000 1.4000000e+000 2 6.7000000e+000 3.0000000e+000 5.0000000e+000 1.7000000e+000 2 6.0000000e+000 2.9000000e+000 4.5000000e+000 1.5000000e+000 2 5.7000000e+000 2.6000000e+000 3.5000000e+000 1.0000000e+000 2 5.5000000e+000 2.4000000e+000 3.8000000e+000 1.1000000e+000 2 5.5000000e+000 2.4000000e+000 3.7000000e+000 1.0000000e+000 2 5.8000000e+000 2.7000000e+000 3.9000000e+000 1.2000000e+000 2 6.0000000e+000 2.7000000e+000 5.1000000e+000 1.6000000e+000 2 5.4000000e+000 3.0000000e+000 4.5000000e+000 1.5000000e+000 2 6.0000000e+000 3.4000000e+000 4.5000000e+000 1.6000000e+000 2 6.7000000e+000 3.1000000e+000 4.7000000e+000 1.5000000e+000 2 6.3000000e+000 2.3000000e+000 4.4000000e+000 1.3000000e+000 2 5.6000000e+000 3.0000000e+000 4.1000000e+000 1.3000000e+000 2 5.5000000e+000 2.5000000e+000 4.0000000e+000 1.3000000e+000 2 5.5000000e+000 2.6000000e+000 4.4000000e+000 1.2000000e+000 2 6.1000000e+000 3.0000000e+000 4.6000000e+000 1.4000000e+000 2 5.8000000e+000 2.6000000e+000 4.0000000e+000 1.2000000e+000 2 5.0000000e+000 2.3000000e+000 3.3000000e+000 1.0000000e+000 2 5.6000000e+000 2.7000000e+000 4.2000000e+000 1.3000000e+000 2 5.7000000e+000 3.0000000e+000 4.2000000e+000 1.2000000e+000 2 5.7000000e+000 2.9000000e+000 4.2000000e+000 1.3000000e+000 2 6.2000000e+000 2.9000000e+000 4.3000000e+000 1.3000000e+000 2 5.1000000e+000 2.5000000e+000 3.0000000e+000 1.1000000e+000 2 5.7000000e+000 2.8000000e+000 4.1000000e+000 1.3000000e+000 2 6.3000000e+000 3.3000000e+000 6.0000000e+000 2.5000000e+000 3 5.8000000e+000 2.7000000e+000 5.1000000e+000 1.9000000e+000 3 7.1000000e+000 3.0000000e+000 5.9000000e+000 2.1000000e+000 3 6.3000000e+000 2.9000000e+000 5.6000000e+000 1.8000000e+000 3 6.5000000e+000 3.0000000e+000 5.8000000e+000 2.2000000e+000 3 7.6000000e+000 3.0000000e+000 6.6000000e+000 2.1000000e+000 3 4.9000000e+000 2.5000000e+000 4.5000000e+000 1.7000000e+000 3 7.3000000e+000 2.9000000e+000 6.3000000e+000 1.8000000e+000 3 6.7000000e+000 2.5000000e+000 5.8000000e+000 1.8000000e+000 3 7.2000000e+000 3.6000000e+000 6.1000000e+000 2.5000000e+000 3 6.5000000e+000 3.2000000e+000 5.1000000e+000 2.0000000e+000 3 6.4000000e+000 2.7000000e+000 5.3000000e+000 1.9000000e+000 3 6.8000000e+000 3.0000000e+000 5.5000000e+000 2.1000000e+000 3 5.7000000e+000 2.5000000e+000 5.0000000e+000 2.0000000e+000 3 5.8000000e+000 2.8000000e+000 5.1000000e+000 2.4000000e+000 3 6.4000000e+000 3.2000000e+000 5.3000000e+000 2.3000000e+000 3 6.5000000e+000 3.0000000e+000 5.5000000e+000 1.8000000e+000 3 7.7000000e+000 3.8000000e+000 6.7000000e+000 2.2000000e+000 3 7.7000000e+000 2.6000000e+000 6.9000000e+000 2.3000000e+000 3 6.0000000e+000 2.2000000e+000 5.0000000e+000 1.5000000e+000 3 6.9000000e+000 3.2000000e+000 5.7000000e+000 2.3000000e+000 3 5.6000000e+000 2.8000000e+000 4.9000000e+000 2.0000000e+000 3 7.7000000e+000 2.8000000e+000 6.7000000e+000 2.0000000e+000 3 6.3000000e+000 2.7000000e+000 4.9000000e+000 1.8000000e+000 3 6.7000000e+000 3.3000000e+000 5.7000000e+000 2.1000000e+000 3 7.2000000e+000 3.2000000e+000 6.0000000e+000 1.8000000e+000 3 6.2000000e+000 2.8000000e+000 4.8000000e+000 1.8000000e+000 3 6.1000000e+000 3.0000000e+000 4.9000000e+000 1.8000000e+000 3 6.4000000e+000 2.8000000e+000 5.6000000e+000 2.1000000e+000 3 7.2000000e+000 3.0000000e+000 5.8000000e+000 1.6000000e+000 3 7.4000000e+000 2.8000000e+000 6.1000000e+000 1.9000000e+000 3 7.9000000e+000 3.8000000e+000 6.4000000e+000 2.0000000e+000 3 6.4000000e+000 2.8000000e+000 5.6000000e+000 2.2000000e+000 3 6.3000000e+000 2.8000000e+000 5.1000000e+000 1.5000000e+000 3 6.1000000e+000 2.6000000e+000 5.6000000e+000 1.4000000e+000 3 7.7000000e+000 3.0000000e+000 6.1000000e+000 2.3000000e+000 3 6.3000000e+000 3.4000000e+000 5.6000000e+000 2.4000000e+000 3 6.4000000e+000 3.1000000e+000 5.5000000e+000 1.8000000e+000 3 6.0000000e+000 3.0000000e+000 4.8000000e+000 1.8000000e+000 3 6.9000000e+000 3.1000000e+000 5.4000000e+000 2.1000000e+000 3 6.7000000e+000 3.1000000e+000 5.6000000e+000 2.4000000e+000 3 6.9000000e+000 3.1000000e+000 5.1000000e+000 2.3000000e+000 3 5.8000000e+000 2.7000000e+000 5.1000000e+000 1.9000000e+000 3 6.8000000e+000 3.2000000e+000 5.9000000e+000 2.3000000e+000 3 6.7000000e+000 3.3000000e+000 5.7000000e+000 2.5000000e+000 3 6.7000000e+000 3.0000000e+000 5.2000000e+000 2.3000000e+000 3 6.3000000e+000 2.5000000e+000 5.0000000e+000 1.9000000e+000 3 6.5000000e+000 3.0000000e+000 5.2000000e+000 2.0000000e+000 3 6.2000000e+000 3.4000000e+000 5.4000000e+000 2.3000000e+000 3 5.9000000e+000 3.0000000e+000 5.1000000e+000 1.8000000e+000 3 ``` 以類別2與類別3的第3特徵與第4特徵做分類 ``` %% 讀取.txt資料 dataSet = load('iris.txt'); rawData = dataSet(:,1:4); % 原始資料，75筆資料 x 4個特徵 label = dataSet(:,5); % 75筆資料所對應的標籤 %% Part1、LDA % 選取每類別前半，合併為training set train_positive = rawData(51: 75, 3: 4); train_negitive = rawData(101: 125, 3: 4); % 選取每類別後半，合併為test set test_positive = rawData(76: 100, 3: 4); test_negitive = rawData(126:150, 3: 4); c12=1; c21=1; [w, b] = LDA(train_positive', train_negitive', c12, c21); [TP, FP] = LDA_classificate(test_positive', w, b, true); [TN, FN] = LDA_classificate(test_negitive', w, b, false); classification_rate = (TP + TN)/length([test_positive; test_negitive]); fprintf('前25類做training data 分類率 : %.4f \n', classification_rate) plot(train_positive(:,1), train_positive(:,2), 'ro',... test_positive(:,1), test_positive(:,2), 'r*',... train_negitive(:,1), train_negitive(:,2), 'go',... test_negitive(:,1), test_negitive(:,2), 'g*'); x= linspace(max(min([train_positive; train_negitive])),max(max([train_positive; train_negitive]))); y= (w(1)*x + b)/(-1* w(2)); plot(x,y) title('LDA Plot'); legend('train positive', 'test positive','train negitive', 'test negitive', 'classification'); xlabel('Feature3'); ylabel('Feature4'); axis tight set(gcf,'position',[0 0 500 500]) saveas(gcf, 'Part 1-1.png'); [w, b] = LDA(test_positive', test_negitive', c12, c21); [TP, FN] = LDA_classificate(train_positive', w, b, true); [TN, FP] = LDA_classificate(train_negitive', w, b, false); classification_rate = (TP + TN)/length([test_positive; test_negitive]); fprintf('後25類做training data 分類率 : %.4f \n', classification_rate) close all; plot(train_positive(:,1), train_positive(:,2), 'ro',... test_positive(:,1), test_positive(:,2), 'r*',... train_negitive(:,1), train_negitive(:,2), 'go',... test_negitive(:,1), test_negitive(:,2), 'g*'); x= linspace(max(min([train_positive; train_negitive])),max(max([train_positive; train_negitive]))); y= (w(1)*x + b)/(-1* w(2)); plot(x,y) title('LDA Plot'); legend('train positive', 'test positive','train negitive', 'test negitive', 'classification'); xlabel('Feature3'); ylabel('Feature4'); axis tight set(gcf,'position',[0 0 500 500]) saveas(gcf, 'Part 1-2.png'); ``` **output:** ``` 前25類做training data 分類率 : 0.9400 後25類做training data 分類率 : 0.9400 ``` 前25類做training data ![](https://i.imgur.com/e3wuLEa.png) 後25類做training data ![](https://i.imgur.com/G7y5Jtb.png) **function LDA** ``` function [W, b] = LDA(u1, u2, c12, c21) u1_avg = zeros(size(u1(:,1))); u2_avg = zeros(size(u2(:,1))); for count=1:length(u1(1,:)) u1_avg = u1_avg + u1(:,count); end u1_avg = u1_avg/length(u1(1,:)); for count=1:length(u2(1,:)) u2_avg = u2_avg + u2(:,count); end u2_avg = u2_avg/length(u2(1,:)); %計算covariance all_u = [u1, u2]; u_bar = zeros(size(all_u(:,1))); for count=1:length(all_u(1,:)) u_bar = u_bar + all_u(:,count); end u_bar = u_bar/length(all_u(1,:)); covariance = 0; for count=1:length(all_u(1,:)) covariance = covariance + (all_u(:,count) - u_bar) * (all_u(:,count) - u_bar)'; end covariance = covariance/(length(all_u(1,:))-1); %pi1 & pi2 pi1 = size(u1(1)) / (size(u1(1))+size(u2(1))); pi2 = size(u2(1)) / (size(u1(1))+size(u2(1))); W = (u1_avg - u2_avg)'*inv(covariance); b = (-1/2)*(u1_avg - u2_avg)' * inv(covariance) * (u1_avg + u2_avg) - log((c12 * pi2)/(c21 * pi1))/log(exp(1)); end ``` **function classifier** ``` function [CorrectCount, FalseCount] = LDA_classificate(point, w, b, positive) CorrectCount = 0; FalseCount = 0; for count=1:length(point(1,:)) H = w * point(:,count) + b; if positive == 1 if H>0 CorrectCount = CorrectCount +1; else FalseCount = FalseCount +1; end else if H<0 CorrectCount = CorrectCount +1; else FalseCount = FalseCount +1; end end end end ```