What is Logistic Regression?
Logistic Regression is one of the most widely used binary classification algorithm in the field of machine learning. It’s a simple algorithm that you can use as a performance baseline, it’s easy to execute, and it works well in many situations. Like other machine learning algorithms, the knowledge of statistic, linear algebra and calculus is needed to understand this algorithm.
Despite its name, it’s not a regression problem algorithm where you want to predict a continuous outcome. Instead, logistic regression is the go-to technique of binary classification. It provides you a discreet binary result between 0 and 1. To put it in simpler words, the result is either one thing or another.
Understanding the basic concepts of Logistic Regression can also be helpful to go through deep learning while building neural networks. So, every machine Learning engineer should be familiar with its concepts. In this post, you will learn what Logistic Regression is, how it works, what are advantages and disadvantages and much more.
A easy instance of a logistic regression problem would be an algorithm used for cancer detection that takes the screening image as an input and should say if a patient has cancer (1) or not (0).
How Logistic Regression works?
Logistic Regression measures the relationship between the dependent variable (commonly called as label, what we want to predict) and the one or more independent variables (our features, or attributes), by estimating the probabilities using the underlying logistic function. These probabilities must then be transformed into binary values in order to actually make a prediction. This is the task of the logistic function (commonly known as sigmoid function). The Sigmoid-Function has an S-shaped curve that can take any real-valued number and map it into a value between the range of 0 and 1, but never exactly at those limits. This values between 0 and 1 is then transformed into either 0 or 1 by considering certain threshold.
Logistic function (sigmoid function) and its graph:
Steps involved in logistic regression can be understood from the image below:
Logistic Regression VS. Linear Regression
You might be asking yourself what the difference between logistic and linear regression is. Logistic regression provides you a discrete result but linear regression gives a continuous outcome. A nice example of a continuous outcome would be a model that predicts the value of a house. That valuation will always be different based on features like it’s size or location of the house. A discrete result will always be one thing (you have cancer) or another (you have no cancer).
When to use Logistic Regression?
As mentioned earlier, Logistic Regression classifies your input into two classes by a linear boundary, one for each class. Therefore it is required that your data is linearly separable, like the data points as shown in the picture below:
Note: Logistic Regression can also be used in multiclass classification but more commonly it used in binary classification.
Advantages and Disadvantages of Logistic Regression:
It is one of the widely used classification technique as it is very efficient, does not require too much of computational resources, it’s highly interpretable, it doesn’t require input features to be scaled, it’s easy to regularize, and it outputs well-calibrated predicted probabilities.
You can remove attributes that are unrelated to the output variable as well as attributes that are very similar (correlated) to each other while performing Logistic regression, its result is same. Therefore Feature Engineering plays an important role in regards to the performance of Logistic and also Linear Regression.
Another advantage of Logistic Regression is that it is incredibly easy to implement and very efficient to train. Because of its simplicity and the fact that it can be implemented relatively easy and quick, Logistic Regression is also a good baseline that you can use to measure the performance of other more complex Algorithms.
If you have good understanding on Logistic Regression, than it becomes easier for you to understand the deep neural networks of deep learning.
A disadvantage of Logistic regression is that we can’t solve non-linear classification problems with logistic regression since it’s decision surface is linear.
Just take a look at the example below that has 2 binary features from 2 examples.
It is obvious that we cannot draw a single line that separates these 2 classes without a huge error. To use a simple decision tree would be a much better choice.
Logistic Regression is not one of the most powerful algorithms, it becomes much slower and doesnot perform well if there is huge amount of data with many number of features and the output is multiclass. Another disadvantage is its high reliance on a proper presentation of your data. This means that logistic regression is not a useful tool unless you have already identified all the important independent variables. Since its outcome is discrete, Logistic Regression can only predict a categorical outcome. Also, Logistic regression is prone to overfitting.