Support Vector Machines (SVM) Basic concepts and Algorithm
Support Vector is one of the strongest but mathematically complex supervised learning algorithm used for both regression and Classification. It is strictly based on the concept of decision planes (most commonly called hyper planes) that define decision boundaries for the classification. A decision plane is one that separates between a set of data having different class memberships.
It performs classification by finding the optimal hyperplane that maximizes the margin between the two classes with the help of support vectors.
Hyperplanes and Support Vectors
Hyperplanes are the decision boundaries that divide the data points into different classes. Data points falling on either side of the hyperplane can be attributed to different classes. The dimension of the hyperplane depends upon the number of features or attributes. If the number of input features is 2, then the hyperplane comes to be a line. It means to say it can be separated linearly. If the number of input features is 3, then the hyperplane becomes a two-dimensional plane. As the number of input features goes on increasing from 3, it becomes harder to imagine the hyperplane.
Support vectors and Hyerplanes
Support vectors are data points that are located closer to the hyperplane. They affect the position and orientation of the hyperplane. With the help of these support vectors, we tends maximize the margin of the classifier. Any change in the support vectors will change the position of the hyperplane. These are the points are the vital points that help us build our SVM classifier.
Classification using SVM/Finding a right hyperplane
When data are linearly separable:
For linearly separable data, classification is done by finding an optimal hyperplane between the classes.The distance between the hyperplane and the nearest data point(Support vector) from either set is known as the margin. The goal is to choose a hyperplane with the greatest possible margin between the hyperplane and any point within the training set, giving a greater chance of new data being classified correctly.
Linearly seprable data (SVM)
When data are linearly inseparable:
For non-linearly separable data, kernel functions are used. Kernels can be defined as functions that take non linear data as input and transform it into required form(i.e. separable form). There are several Kernal functions like linear kernel, Polynomial kernel, RBF Kernel,Sigmoid kernel.
linearly inseparable Data to linearly separable Data
In the SVM algorithm, the kernel SVM takes a kernel function and transforms it into the required form that maps data to a higher dimension that can be separated.
Some of the most common types of kernel function are:
Linear Kernel: K(Xi,Xj) = Xi.Xj
Polynomial kernel: K(Xi,Xj) =( γXi.Xj+C)d , where d is the degree of the polynomial that should be specified.
RBF Kernel: K(Xi,Xj) =exp(- γ|Xi -Xj|2), it is used for non-linearly separable variables. For distance metric squared Euclidean distance is used.
Sigmoid kernel: K(Xi,Xj) =tanh( γXi.Xj+C), it is similar to logistic regression is used for binary classification
Kernel trick usesthe kernel function to transform the data into a higher dimensional feature space to make it possible to perform the linear separation for classification.
So, it is better to use linear SVMs for linear problems, and non-linear kernels such as the sigmoid kernel, Radial Basis Function kernel for non-linear problems.