Two talks at last week’s FIRST conference looked at how Artificial Intelligence might be used in incident response. In both cases, the use of AI improves user privacy directly – by reducing or eliminating the need for human responders to look at user data – and also indirectly, by producing faster detection and mitigation of security/privacy threats.
Both talks stretched my knowledge in fascinating ways, so apologies if anything in the following isn’t correct. I’ll add links to the recordings when they are publicly available…
CK Chen looked at more traditional approaches, giving an excellent walk-through of how to construct a Machine Learning pipeline with the characteristics needed to support human threat hunters. Human threat hunting approaches involve looking at lots of user activity data, and generate lots of false positives. Can Machine Learning do better? As source data, the demonstration pipeline uses Windows process creation and command line events, gathered during (a) normal operation and (b) simulated attacks using APT3 and Metasploit.
- Stage 0: training an algorithm using these two – good and bad – datasets still produces lots of errors. A few processes/commands only appear in a single dataset, but things such as ‘netstat’ and ‘whoami’ are in both. Whether these are good or bad depends on the context. So…
- Stage 1: noting that a human threat hunter will look for significant sequences of events, try grouping parent and child processes into clusters, then get a human to label those clusters as good or bad, and train the algorithm using the clusters. This gives much fewer false positives, but still needs significant effort and data inspection. So…
- Stage 2: noting that attacks are less common than normal behaviour, add an anomaly detection stage to the pipeline. Processes common in the ‘bad’ dataset that appear in anomalous clusters give a strong signal. But that same pattern applies to novel legitimate activity, so this algorithm tends to generate false positives that are hard to explain. So…
- Stage 3: noting that attackers probably have to start from one of a small number of vulnerable processes, look for abnormally densely connected clusters around those processes. Interestingly, this algorithm performs slightly less well, in terms of number of false positives, but it provides a storyline that is much easier for a human to interpret: here’s where they gained initial access, escalated privilege, exploited the vulnerability.
For an AI that is designed to work with a human – protecting the privacy of (most) legitimate activity from human eyeballs, and passing the rest on to a human investigator – easy comprehension is more relevant (to both privacy and security) than highest numerical performance.
Holly Stewart, Anna Bertinger and Sharada Acharaya from Microsoft looked at approaches that go further, and take the human out of the loop (except when invited in) entirely. When millions of new spam and phishing attacks appear every day, and last no more than an hour, full automation is essential from a security, as well as a privacy, point of view. In each case, the aim is to build AI that can classify previously unseen threats, based on self-reported data. The first approach they described was data obfuscation: eliminating personal data that you don’t need. For example if someone reports malware hiding files in their Favourites directory, that information is sufficiently well structured that you can remove the username from the path, benefitting privacy without affecting either training or detection. Next is “eyes-off training” where human data scientists see the “shape” of submitted data (flows, volumes, etc) but only the AI sees content for both training and detection. This can work well, but raises a problem when investigating false positives and negatives, and understanding what the model is doing, since the investigator can’t look at live content to understand its behaviour. Finally, one of the first practical uses of homomorphic encryption I’ve seen. This is a class of encryption algorithms that preserve arithmetic operations, so adding or multiplying two encrypted values produces the encrypted version of the true sum or product. It turns out that it is (just) possible to write machine learning classifiers within these constraints. So, when inviting users to submit screenshots of phishing pages, features of the pages are extracted and encrypted on the local machine, the encrypted values are submitted to a classifier in the cloud, and the user gets back a “don’t go there” message, fast enough that it should prevent them being tempted. Seriously impressive, both from a privacy and security point of view!
