Security buzzwords for the millennium have been machine learning (ML), deep learning, and artificial intelligence (AI). This new technology boasts learning systems with impressive results in marketing fluff, but not all machine learning is created equally. Algorithms are the decision-making formulas that allow the application to identify known threats and make informed guesses on unknown threats with a level of confidence. Implementing this technology can have a marked result on the efficacy of detection systems; however, there may be a cost. Here are 5 questions to ask your manufacturer of choice to understand the methods they use and potential downsides to their solution.
Why use the private cloud?
First, there are a few variations of the private cloud to consider. For this blog, a private cloud is defined as a dedicated (Private Network), a highly available and scalable infrastructure usually consumed under operational expense.
Based on this definition, the private cloud can deliver a lot of the same benefits as the public cloud, all with an architecture designed specifically for a single enterprise. The resources are not shared with others and cost management is typically more predictive. (For example, flat rates for connectivity offered by private cloud providers versus ingress or egress costs when in the public cloud.)
When it comes to security private cloud providers tend to shine. That’s because knowing that the data center has a findable address and seeing that the physical security of the data center is sound gives a lot of IT leaders peace of mind.
Most private cloud providers are smaller and offer dedicated resources, offering a better opportunity for traditional relationship building. This adds a layer of intangible security that most business still value today.
Cloud access is also more secure on a private cloud environment because it is accessed through private and secure networks links instead of the public internet.
And finally, private cloud providers tend to offer a more comfortable path to achieving compliance objectives—HIPAA is a great example.
Moving from on-prem infrastructure to the private cloud? Here are some ways to avoid problems in the future.
1. What is the corresponding false-positive rate to your detection rate?
Machine learning algorithms use a level of confidence to determine whether an encountered file is malicious or benign. This confidence is almost never 100%, nor is it ever 0%, but somewhere in between. However, much like your younger self, even if you are 100% confident on a test doesn’t mean you are right. The algorithm can be tuned to optimize false positives or valid detections to a level that meets the need.
The Receiver Operating Characteristic (ROC) curve is created by plotting the true positive rate against the false-positive rate at various threshold settings. If you select a true detection rate on the curve, you will also be selecting a corresponding false-positive rate. Knowing how the manufacturer has optimized their detection rate and showing its ROC curve goes a long way towards identifying if they have done proper in-house machine learning research. This will also tell you how much human intervention will be needed to ensure valid files are not being blocked or malicious files passing through.
2. How often does your model need updating and does this impact accuracy between updates?
It’s easy to find examples of AI algorithms being manipulated or corrupted by false data. Microsoft famously failed with an AI chatbot named Tay that started posting inflammatory and offensive tweets within 16 hours of launch. This is in part a result of positively enforcing incorrect data leading to drift.
As the machine learning training data ages, the effectiveness decays over time. Additionally, if the machine learning does not get manually adjusted periodically, it can wrongly positively re-enforce bad decisions reducing efficacy long-term.
A good model is initially trained on a large and diverse training set which can establish a stable model and prolong the efficiency of the system. Even the most robust, diversely trained AI may have to be replaced periodically.
3. Does the machine learning algorithm make decisions in real-time and locally?
There are two ways machine learning can act on a possible threat; retrospective analysis or inline blocking. With retrospective analysis, the AI will either block until verdict and leverage cloud lookups and resources to render a decision. This method can often add latency and require an online connection.
With inline blocking, the AI is working in real-time to make decisions and the model needs to be small enough to reside in memory and able to work in an off-line mode. Additionally, if your systems are offline or have limited internet access often, having a system that is able to adapt and protect you locally can be vital to the success of the solution.
4. What sources do you base your training set on?
A diverse training set reflective of real-world threats your organization may face is key to a good model. The training set will impact everything from performance, accuracy, to the ability to adapt to change. Many AI-machine learning companies leverage training data from VirusTotal or similar sources; this data is a good start but may not offer the diversity that is needed long term. If the AI has never seen the types of threats your organization encounters, it may not work as effectively as advertised. Look for organizations that may also have access to their own threat intelligence or pull from many active sources.
5. How does your machine learning system scale?
Threats are constantly changing; sometimes in surprising ways. The sheer number of threats and types of attacks are not going to reduce over time. This gives the machine learning algorithm plenty of samples to ingest to grow from, but can the back-end database handle the data load over time? How does the machine learning scale, yet remain inline and real-time? Understanding how the manufacturer has considered these changes and how that will change performance over the life of the product is important in your purchase decision.
Machine learning, AI and deep learning are variants of the same technology and bring new and exciting changes to cybersecurity. With those new changes, we need a new language to help understand how these systems work and new questions to ensure we are getting the expected results. With these 5 questions, we can start to quantify the efficacy of the solution and ensure it fits the environment moving forward.
