What is Confusion Matrix?
Have you been in a situation where you expected your machine learning model to perform really well but it sputtered out a poor accuracy? You’ve done all the hard work — so where did the classification model go wrong? How can you correct this?
There are plenty of ways to gauge the performance of your classification model but none have stood the test of time like the confusion matrix. It helps us evaluate how our model performed, where it went wrong and offers us guidance to correct our path.
In this article, we will explore how a Confusion matrix gives a holistic view of the performance of your model. And unlike its name, you will realize that a Confusion matrix is a pretty simple yet powerful concept. So let’s unravel the mystery around the confusion matrix!
The million-dollar question — what, after all, is a confusion matrix?
A Confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the number of target classes. The matrix compares the actual target values with those predicted by the machine learning model. This gives us a holistic view of how well our classification model is performing and what kinds of errors it is making.
For a binary classification problem, we would have a 2 x 2 matrix as shown below with 4 values:
Each row in a confusion matrix represents an actual class, while each column represents a predicted class. For more info about the confusion matrix click here.
The confusion matrix gives you a lot of information, but sometimes you may prefer a more concise metric.
precision = (TP) / (TP+FP)
TP is the number of true positives, and FP is the number of false positives.
A trivial way to have perfect precision is to make one single positive prediction and ensure it is correct (precision = 1/1 = 100%). This would not be very useful since the classifier would ignore all but one positive instance.
- true positives (TP): These are cases in which we predicted yes (they have the disease), and they do have the disease.
- true negatives (TN): We predicted no, and they don’t have the disease.
- false positives (FP): We predicted yes, but they don’t actually have the disease. (Also known as a “Type I error.”)
- false negatives (FN): We predicted no, but they actually do have the disease. (Also known as a “Type II error.”)
Let’s make the following definitions:
- “Wolf” is a positive class.
- “No wolf” is a negative class.
We can summarize our “wolf-prediction” model using a 2x2 confusion matrix that depicts all four possible outcomes:
A true positive is an outcome where the model correctly predicts the positive class. Similarly, a true negative is an outcome where the model correctly predicts the negative class.
A false positive is an outcome where the model incorrectly predicts the positive class. And a false negative is an outcome where the model incorrectly predicts the negative class.
Machine Learning in Cybersecurity
But in 2018 alone, there were 10.5 billion malware attacks. That’s too much volume for humans to handle. Fortunately, machine learning is picking up some slack.
A subset of artificial intelligence, machine learning uses algorithms born of previous datasets and statistical analysis to make assumptions about a computer’s behavior. The computer can then adjust its actions — and even perform functions for which it hasn’t been explicitly programmed.
And it’s been a boon to cybersecurity.
With its ability to sort through millions of files and identify potentially hazardous ones, machine learning is increasingly being used to uncover threats and automatically squash them before they can wreak havoc.
Software from Microsoft reportedly did just that in early 2018. According to the company, cybercrooks used trojan malware in an attempt “to install malicious cryptocurrency miners on hundreds of thousands of computers.”
The attack was stopped Microsoft’s Windows Defender, a software that employs multiple layers of machine learning to identify and block perceived threats. The crypto-miners were shut down almost as soon as they started digging. There are other examples of Microsoft’s software catching these attacks early.
The massive French insurance and financial services company AXA IT relies on the cybersecurity firm Darktrace to deal with online threats. And Darktrace relies in part on machine learning to drive its cybersecurity products.
The company’s Enterprise Immune System automatically learns how normal network users behave so it can spot potentially dangerous anomalies. Other software then contains in-progress threats.
“We’re not being attacked by human beings anymore,” Yorck Reuber, CTO of AXA IT North Europe, told Darktrace. “Computers are attacking us, software is attacking us. The only way forward is using artificial intelligence.”
In addition to early threat identification, machine learning is used to scan for network vulnerabilities and automate responses. And in the cybersecurity realm — where a reported one-third of all chief information security officers are totally reliant on AI and unethical hackers are always on the prowl for new ways to exploit security vulnerabilities — that’s proving to be a huge plus.
Cyber Attack Detection and Classification using Parallel Support Vector Machine
Support Vector Machines (SVM) are the classifiers that were originally designed for binary c1assification. The c1assificatioin applications can solve multi-class problems. The result shows that pSVM gives more detection accuracy for classes and comparable to the false alarm rate.
Cyberattack detection is a classification problem, in which we classify the normal pattern from the abnormal pattern (attack) of the system.
The SDF is a very powerful and popular data mining algorithm for decision-making and classification problems. It has been using in many real-life applications like medical diagnosis, radar signal classification, weather prediction, credit approval, and fraud detection, etc.
A parallel Support Vector Machine (SVM) algorithm was proposed for the detection and classification of cyber attack datasets.
The performance of the support vector machine is greatly dependent on the kernel function used by SVM. Therefore, we modified the Gaussian kernel function in a data-dependent way in order to improve the efficiency of the classifiers. The relative results of both the classifiers are also obtained to ascertain the theoretical aspects. The analysis is also taken up to show that PSVM performs better than SDF.
The classification accuracy of PSVM remarkably improve (accuracy for Normal class as well as DOS class is almost 100%) and comparable to false alarm rate and training, testing times.
KDD CUP ‘’99 Data Set Description
This data set is prepared by Stolfo et al and is built based on the data captured in the DARPA’98 IDS evaluation program. DARPA’98 is about 4 gigabytes of compressed raw (binary) TCP dump data of 7 weeks of network traffic, which can be processed into about 5 million connection records, each with about 100 bytes.
For each TCP/IP connection, 41 various quantitative (continuous data type) and qualitative (discrete data type) features were extracted among the 41 features, 34 features (numeric), and 7 features (symbolic).
To analysis the different results, there are standard metrics that have been developed for evaluating network intrusion detections. Detection Rate (DR) and false alarm rate are the two most famous metrics that have already been used. DR is computed as the ratio between the number of correctly detected attacks and the total number of attacks, while the false alarm (false positive) rate is computed as the ratio between the number of normal connections that is incorrectly misclassified as attacks and the total number of normal connections.
In parallel SVM machine first, we reduced nonclassified features data by distance matrix of the binary pattern. From this concept, the cascade structure is developed by initializing the problem with a number of independent smaller optimizations and the partial results are combined in later stages in a hierarchical way, as shown in figure 1, supposing the training data subsets and are independent among each other.
- True Positive (TP): The amount of attack detected when it is actually attacked.
- True Negative (TN): The amount of normal detected when it is actually normal.
- False Positive (FP): The amount of attack detected when it is actually normal (False alarm).
- False Negative (FN): The amount of normal detected when it is actually attacked.