>> : It's DEFCON time. Yeah, all right. This is secure because math a deep dive on machine
learning based monitoring. So please join me in welcoming Alex Pinto. [Applause]. >> : Thank
you guys. Glad to see you all here and just before we start I just want to ask for all of you
that are from the internet and you like to do the Twitterings we have a hash tag for the talk
so make sure when you make fun of me you hash tag it or something like that because I
also do want to laugh. I actually check Twitter while you're talking so I can read the
funny stuff you are talking about. But anyway thank you so much. Let's get started. Just a
little about me. Chief Data Sciences. The cool thing about doing your own stuff is that you
can just make up any title that you want. I've pretty much been doing machine learning research
learning and training for a while now and just for anyone who is interested in this
subject I found out machine learning training is different from Pokémon training. I might
have got the wrong brochure or something but anyway... You guys might enjoy it. Anyway I'm
focusing my research on national security and a little bit of response. As a child I never
wanted to do that again so I'm trying to figure out a way where we have to do less or maybe do
it in the more smart way. Anyway... If for some reason you think I have something and you
need an attribution from me I am Caffeinated Capybara. All right. Let's get going. Anyway I want
to talk about what is upcoming, security singularity later because of all the amazing
products launched. And I just want to make sure that I give you some context, specifically
about the messaging of, I obviously don't know precise detail technically what these
companies are doing, it's secret and saucey. It's so secretive and stuff like that but I can
guess based on their marketing materials and I just wanted to break down for you what I think
that those people were doing and if that's what they are doing. So maybe these are the questions
you should be asking them because this stuff is hard. I mean honestly math is hard. And
there's a lot of potential pitfalls and a lot of kind of due diligence in your
experiments and I just really want to spend quality time together talking about math here
while we can try to decode a little bit. Anyway... Let's get going. So I specifically would
like to thank all of you because you guys are on vacation now. Nobody has to work anymore and
you guys actually are coming here to see someone talk about security singularity. Why. This
has been solved for a while now. It's sad because the guys that did this network security thing
have nothing to do with machine learning. I thought this was so amazing I had to put it out
there. Maybe she should persue machine learning products because that would make it even
more awesome I think. Anyway... Of course you kind of get the point where I'm coming
from. Before I continue I just want to make a quick side note. If you ever do a Google images
search for network security solve, Jack Daniel was the first hit. That's branding. I'll tell
you that man. It's just amazing. I wish I was okay, who knows maybe Pokémon will train us some
day. The point is there's a lot of confusion. Right? And this is understandable. There's always a
lot of confusion about specific technologies we're working on. We're always like for some
reason there's a cycle where we stop trusting and we start looking for new so people have
to come up with new stuff. And the point is that I'm frequently being asked a bunch of questions
because of the kind of work I'm trying to do. Hey I'm talking to this guy and he tells me he's
doing the math or something like that. Can you help me out and try to understand? I'm not
talking about someone at the street. These are people who have been doing extensive
product research or have been actual running security programs for a while. And they are
failing to grasp, it's a pain to grasp what the hell the companies are doing. It's almost
like, you know, there was some sort of marketing employ to make it kind of obscure to make
people think it's better than then it is. I'm not saying that. But maybe you can argue the
point. And I think that's kind of bad for everyone. Right? And this is pretty much the point
I'm coming from here. By the way is if any of you ever want to do a big data presentation there's
this big data fix tumbler. So waves with bits. Best data ever. But anyway no. I'm here to
teach something. Right? If you pick something out of this talk it's the data fix tumbler.
Anyway I guess my point is are we even trying to explain what's the security. You guys know that
like if it's more than 3 you're already hyper right.? Artificial intelligence. It's like the
third level of the matrix. What is that? I mean honestly. It kind of gets to me because I
mean isn't enough to go to the cyber wars and things like that. Do we have to scare people? I
mean... Yeah. And I wasn't able to find I don't know if I dreamed about it or if it was a
company but I think it's perfect. It's exactly that. It's secure because of math. Math.
The point, okay. This is hurting us. Right? We are unable to differentiate the products. The
investors have no idea they're funding stuff. Oh I have this security magic there here, okay
here is $10 million. And the point, it's like we're not even sure it's the same words yet in
many levels and things. And I mean I don't know about you guys, I don't have a lot of time
to waste. I don't want to be like beta testing a bunch of this stuff. I want to make sure
the people that are trying to do the research or work I can have some way to identify more easily
if what they're talking makes sense or not. I have people argue to me the point that it's
all about communication really. Right? If you get the people from the technical side of
things, right, and they try to explain [Laughing]. Come on guys I have to keep a straight
face here. And they explain to the marketing field. This is what we do. So we're actually
working on this big selection process ... No. No. No. That sounds advanced. Yeah let's go
with advanced. So maybe it's that. Maybe I'm being unfair. Maybe there's a lot of very
different good stuff happening under the hood but I sure as hell cannot figure it out.
Right? By reading what they're doing. And anytime we try to get a little closer it's like no,
no, no. That's the secret sauce. It's like someone telling you honestly my perspective is like
no, no, no I'm using this proprietary crypto algorithm. To a point it's that. So let's do
an exercise. I want you guys to guess the year when this was written. So I will make it more
simpler because there's not like there's going to be a lot of interaction. Of the 3 which one
do you think was written like today or like 2014? I would like a show of hands for number 1.
Okay there's like very, very few people. Show of hands for number 2. Show of hands for number 3.
Okay it looks like you guys don't like to play a lot of games. There's no right or wrong
least all these suck so don't worry about it. Anyway... The first one is actually from 10
years ago. I don't know if there's any ISS bugs here. Guys released a product and I used to
work for one of the largest integrators they had in Latin America. It was like the hugest
thing, we sold a bazillion. We never could get the shit to work. I don't know maybe it was
just us but... If you go to the middle one is actually from this year. I'm not telling you who it
is but you can do research. And the third one is actually when it gets interesting. It's
actually from 1995. It's from actually some research that is being done University research
and they created a product pretty much like it was right there. That sounds very
familiar. Right? And there's this woman, Dorothy Denning, a respected professor at the
graduate school she did the first paper that came up with what the IDS should look like in
86. Okay there's two parts of this. One is it's a rule based engine where we have signatures
and the other will be an anomaly detection engine that will help pick up the stuff that we don't
have signatures for. Maybe this will give you a hint. Let's keep people informed that they
shouldn't trust this part as much as that part. So 86. And anyway they actually built it
out to the intrusion detection expert system. I think people drop the E because too many
letters, too confusing. But the point I find funny actually is that from 95 was perfect
colleagues all males stole her work and made it like the next generation idea. The next
generation has been for a while here. So anyway let's get with the times. Right? And the point
everything changed because of a 3 letter acronym really. So there was a 3 letter acronym
that did something that was very significant for information security and that actually
changed the way that we do research and we do a lot of things. It's probably not who
you're thinking of. It's the KDD. It's actually a program program? It's kind of a research
track or conference from ACM and guided by DARPA, because you will probably never need another
signature engine. Boy were they right. And they decided to start funding and creating data sets
that people could use. So they had their own data sets which was focused on user anomaly
detection. Man I'm talking like, soloris whoa. This is what the specific organization looks like
for like 6 weeks. Let's see if you can pick it up here. It was a great success. It was a great
success in the sense that a lot of people started using those data sets for research and there
was nothing previously there that people could reliably use and repeat. So a lot of the
research was based on this and they used this to actually try to improve their algorithms. I
mean at first it's okay but after a while if you go through this I mean it's like one
bazillion in 99 and as you go across the years it looks like everyone is trying to warm up.
The other one I got a little better percentage the data set and that's fine. I don't know
what the fuck they do there but what bothers me is that if you do a search in Google today
there are 300 papers in 2014 they are still using this 15 year old data set to come up
with conclusions with what is good you know what is bad and what is promising in anomaly
detection. I mean, I have no idea what exactly is the modifying information based
selection but don't do it in a 15 year old data set. I'm pretty sure things have kind of
evolved, things are kind of changed. Right? And I understand that there are limitations.
Right? I understand the reproducibility thing. For some respect I'm going to get to that
but I want you guys to think about this. I want you guys to think about this. If you were
going to med school and you're going to learn anatomy and all that you have to go for is this
picture of Rembrandt. Right? Granted the human body hasn't changed that much. Right? But
it's like hey professor what if we cut a fresh one open? That would be a privacy violation.
Okay I'm not advocating like unlawful sharing but I'm pretty sure we can do better than
public data sets right now. I'm pretty sure there's enough people interested in this that
we can generate more stuff that people can use for this research. Right? And a funny
side note is that actually there was a professor that was pretty much in 2000 raving about, okay,
guys this was a bad identify. The data sets actually sucked. They are not representative of
what should be going on and he was raving about reproducibility because, yeah, I mean it's not
clear what technique these guys are using. And they are like like they like to one up the
other one. They have a point percentage improvement so it doesn't seem like we're having
success here. That's pretty much how everything works in academia but I don't know. It's like for
me it's mind blowing and people talk a lot about potential disconnect from academia in some
instances. Why are we using a 15 year old data set? I don't know. Maybe it's a joke. Maybe it's
like if you're researching this stuff you actually have to go back to 1999. I'm not here to
bash them. Right? They're all very scary people. Right? Yeah man. You should meet my friend
Kyle who is a math smuggler. But anyway I just want to put you in the mind set. Right? You've done
your research and let's assume for a point that I'm making up unnecessary fun of them and you
have this publishing protocols and everything which you had to try to focus to do specific data
sets. What I think could potentially happen, I'm not sure if this has actually happened
before but I think a potential outcome from something like that and with apology to the guard
life cycle I think the guy will get to grad school. Right? And he will be like, he will go
there and go to the party and everybody would be like turn down for math. You know? And
then they're going to come and it's like man I got this sweet, sweet data set. You guys want to
have a look? You know? And he looks at the data set and gets hooked on it. Okay maybe there's
interesting stuff in the data set. Let's do research. Low and behold he gets a bunch of
results. These kind of things we talk about in machine learning is called over fitting which is
pretty much that you know a data set so much inside out that you're actually writing instead
of doing machine learning you're writing a program to exactly parse the data set. Right? And
it's a very it's a very real problem. It happens all the time. And it's one of the
reasons why in model design you're always like okay just get random stuff. If you don't know
want to get, get random stuff because it doesn't have an immune system and you need to
try to keep it on its toes. And that's the point. The guy publishes a bunch of papers then
goes to his business friend and they're like oh my God you're going to be rich. Right? The
security field is so hard. And then they go and they actually get customer data. I don't know
how they actually convinced these guys. And we get to the results, you know? I was
expecting this thing to be awesome. Right? But then when I tried to realize data it's
something that doesn't look like it's going to work because you're actually training a very
biased and very weird kind of model then you actually expose it to real life data. It doesn't
know what to do. What ends up happening in my belief, right, is that people start, okay,
let's not do this math thing anymore. Right? The guys get frustrated. The business guy,
like hey bro we have to make some dough and yeah, okay let's do people or something like
that. And low and behold math is hard. Let's go shopping. By shopping I mean selling. Right?
I don't know you become general consulting company. You do response for people. But then
that thing you said you were going to do specifically about machine learning is not actually
what you do anymore. For some reason it still lingers in your marketing materials. That's not
who you are. Right. I maybe sound a little too righteous. If you get money to build something
and you don't, you are going to fail. I'm sorry... Right. What do I know? I mean there are
these guys that are filthy rich. I had to think this morning if I was going to take the bottle
from the mini bar. But the point is, guys, I'm pretty sure you're successful and doing great work
but let other people play with the ball in a sense. If you're not doing machine learning don't
crowd your marketing materials with machine learning because it gets people confused. Right?
There are people who are actually trying to build programs. There are quite a few
startups. And I don't know if the worst part is them or the guys that are like hey, machine
learning we should put that in our brochure. It's like yeah we deliver lemons by bicycles with
machine learning. How does that work? But I mean I'm pretty sure you guys would be able to figure
that out. I mean those two different types of companies. Right? There is a sweet spot.
But there are people; ain't not like I'm talking about myself. A bunch of people that are doing
research. What I'm trying to do now for this next part of the talk is to go through a little
bit more technical about if people are telling you about X this is probably what they're
doing and what are the potential pitfalls. I want to start with anomaly detection. Anomaly
detection is kind of interesting for me as a solution to something because when you are
doing machine learning research when you're doing modeling when you're trying to figure out what
you're going to train and predict against, alomaly detection like stuff I'm pretty
much talking about the clustering, I'm talking about decomposition, it's actually
very, very important for your exploratory phase because you just got a bunch of data and you
have no idea what it looks like and what's the shape of it and what you can do with it so we
start asking the computer like questions. Okay is this weird in some way? Is there a normal
thing going on here? And then you use the feedback that the model gave you so you can
actually design something that could potentially work. Okay. And so my point is the kind of
stuff you do you would have no idea what's going on and it's weird for people to say we're
doing anomaly detection. You're like what? What are you actually doing? What is normal? What is
weird? I mean what's the measure? What are all those different things? And to be
honest this actually works very well for when you have a very well defined process. Right? So
this is a factory, you're putting together bolts, right? And you want to measure if the
bolts you're pumping out from your bolt making machine, however it's done is not like
too big or too small. It extends to that standard that you want to do. They know the standard
and you have an effective way of measuring that. And even so when you check your balance, it
actually still works well. A lot of historical work has been done. It's way more complicated
than that now but historical work was based on this. How much was this guy standing and has he
just been to Vegas and spent ten thousands of dollars? Never mind he's been to Defcon. So but
this works because you measure one thing like money S you have one operation goes in or it goes
out. Right? And so this was clearly picked up quickly by the dev guys. And because when you
think about it they're actually running a pretty tight shop. Right? They got all this bunch
of server farms and different things they can measure about the performance of the servers
and anyway how many databases and all those things and when you look at those independently,
stuff like anomaly detection, most people would do rooting averages. They actually work.
Okay I know what I'm measuring and I know what I'm thinking about. I know what or I should
expect to be the norm. So it makes very easy for a human being to make a decision. Right?
And it's not like but it's not like if the bolt is like a little bit bigger and a little
thinner that means it's (unknown) threat from China. Right? That's the leap of
thinking that sometimes I do not understand. I want to talk in more detail. Here when I talk
about that I'm specifically talking about network/netflow behavior analysis and also this
new user behavioral analysis thing which is a new hotness and there's 3 challenges I want to
talk about. The curse of the missionality, the life of ground crew and the last is my
favorite. So the curse of missionality is actually something real. The point is if
you're trying to measure a bunch of different things at the same time and trying to see how
similar they are or how different they are if you think about anomaly detection the
actual problem you're trying to solve is I got this space, let's say I have a square. Right? And
I have a bunch of points drawn in the square. So which one of these is the closest together?
That's probably normal. Right? Because most of them are there. But the things that are further
apart are potentially those anomalies. The fact is you need to measure the distance. Right?
Okay. So what's the point? How do you separate a point? And you can do some very simple stuff
like distance which would be like draw a straight line or the Manhattan distance where you
have to go straight angles. Right. I think this is a stupid name but it's intuitive. The
point is when you start growing the number of dimensions like when you go like okay 3D, 4D,
like 10 dimensions the actual distances they stop meaning stuff. They stop meaning
anything because it's such a huge space. Right? And until the regulation, on the real, Las
Vegas, United States actually allows expanding you will not be able to see it properly. It's
very hard to imagine but it's like very far away. Everything is so, so far away. It doesn't
make any sense. It becomes very hard to measure distances between things. Right? And
another way of seeing that is if you calculate like what's the size of a sphere, right? So you
have a cube. What's the size of this sphere inside this? Which you can argue is like what's the
unit distance between my point and this other point that I'm trying to measure. Right? The
actual volume of this sphere compared to the cube becomes very, very close to zero very,
very fast. And this is a graph I stole, because... But the point is the practical result is
everything looks the same. Right? So what's the kind of stuff I'm talking about? Can you
give us a more practical example? Well let's do Netflow data. Right? I have a company
with N nodes. Right? Okay. All machines can talk to them. All machines can potentially talk to
any machine, NTC port, UDP prot. Right? It chooses ports for each one of these. Pretty much means
if I have a thousand nodes I potential have I half a trillion possible dimensions. So
measuring how much packets went from one place to the other. Okay. That's very hard to figure
something out. Right? To be honest this is a very, very open problem and people have been
trying to work. I mean this is not like security stuff. This isn't math stuff, this is
actually what people have been trying to figure outweighs to represent the data and better
represent the matrix to actually solve this. And those solutions sometimes there's breakthroughs
and okay we have this new idea which is going to help a bunch of these problems and some
become very specific. And there's actually some companies, right, they are basing their
claims on anomaly detection on a lot of research they have been doing on trying to account for
this problem. Some of them believe I mean after 20 years of research they actually have
some very good solutions involving subspaces et cetera, et cetera, et cetera. If that's
true that is actually very, very, very awesome because there's a whole class of
problems that can be solved by this. Right? This is not just security. I mean I do get a
little bit, you know, sinai glare. If people had actually solved this, right, we would be
I don't know first of all we would be doing a killing in the ad selling market. That's for
sure. Second of all we could potentially be literally free of cancer with stuff like this
because I mean Google just did project baseline which is okay let's get this DNA data and put
the data sets together and do some anomaly detection to figure out what are the gene that is
are actually not represented. So Google with all the resources are going to start something
similar to this. Because it's hard. Because it's not a solved problem at all. Anyway people
might have solved aspects of the problem. It might be a good part of their research. Right? I just
wanted to ask. I just wanted to try to understand and you might not be able to understand the
actual technical part of it. I certainly don't. When I say interesting that means I
couldn't get past the abstract otherwise I would totally tell you about the ideas. But what I
don't want is that you guys fall prey to this guy. We do not want the goal to have this one trick.
Right? And anyway the point I'm trying to make here guys is that this is not something that's
been new. It's been here forever. Right? And it's something that it's something
that you can wave your hand away. You just have to deal with it. Right? Just deal with it.
Right? And try to come up with some solutions for that. I have to put the other glasses on
otherwise I can't see. So anyway... The second class is what I talk about normality
attacks. Which pretty much the problem that you have in anomaly detection in general which is
there is no real labeling. There's no real ground truth. You don't know what you're
holding against and again it comes back to what is normal. When you do the other machine
learning which is label you know that okay I think that these things are good. I have a very
good feeling things are good, let's try to get them apart. So some of the end the problems of
this is there's asymmetry because there's usually much more stuff that looks kind of
normal than stuff that looks anomalous. So, you have to, sometimes you have to push the
models too hard to actually recognize what's anomalous and they become problems. If you
have detected anomaly you know what I'm talking about. It's hard to fine tune and also very
easy to tamper with. Some of you are familiar with waves and people like the cross source. Oh
there was a car crash here. So I have a bunch of friends they're about to leave work and there's
a bunch of accidents around and they can get out and go. Anyway... I'm taking too long
here. I have to rush a little bit. Finally and I think is the most important thing even if
you okay my anomalies are very accurate. What does an anomaly even mean? Right? Why is it
necessarily evil? Right? And this is a thing I saw all the time, is that you turn on
something like anomaly detection engine and it picks up all sorts of weird stuff that is not
necessarily security. This is mostly a problem of process, of how do you actually put this in
and it's funny you will never see an anomaly detection company specifically only marketing
itself as security. It will also use the word performance because that's what anomaly detection is
good for. I guess the point I'm trying to make is if there's a spy on the production server
it's using a bench of memory, who is more likely to have done this? Is it the evil hacker or
is it the hipster developer that said yeah let's bring the machine closer... I mean
honestly guys. It can't be all of that. So I just want to get to user behavior quickly. User
behavior actually works but you've got to have a limited scope. So everyone here who does
the product security, so yeah we are the start up or we have this product on the web, you will be
doing user behavioral analysis and fraud detection things in your product. It works
surprisingly well because you have a limited scope. You know exactly your application inside
out so you can actually program relatively easy all the shortcuts you need to have to go
through some of the problems that I described. What bothers me, right, and again people use
anomaly detection to actually then view the classification model. Right? But what bothers
me is can this be generalized? Right? Can we come to people and say, oh, I just filed these. It
works like a charm, the math is amazing. You just have to have role based access to all your
users and do classifications. If I do that I do not. This is something that bothers me. I
guess a lot of these companies actually come from more of military background where this
is all a given. Right? And I'm pretty sure it potentially has good results in that
environment. It's just that that's not how the thing works. Anyway... The other point is if
I'm actually doing user behavior do I average it all out like all the different oh, yeah. I was
putting a bunch of stuff in my expense system here because I've been to DEFCON then it blocks my
access to the work order because I was dos.ing this. How does this work, how does this
actually interact with the other? How does one thing actually direct? There's a lot
of open questions. I really wish there was a way to build a general user behavioral thing.
Right? But I mean question mark for me too. Anyway I'm very late. I just want to quickly go
through classification. I'm sorry this is a repeat slide but there's no way to talk about
this to someone. So you're trying to talk about cats and dogs and you are trying to teach
the program what is a cat and what is a dog. The point is it's really about how many cats and
how many dogs and how many different cats and how many different dogs. If you only got
grumpy cats in your data set you have a serious case of bad labeling, you will definitely
not be able to pick this cat. It's a happy cat. There are no such things as happy cats
because all the cats, so because all the cats I've seen are grumpy. It's a problem that's
similar. In the sense that you don't have a good sample. It's a little biased. In the same way
if it's all majestic and you will miss this guy. I mean come on. Poor guy. I love teen tuna.
Tuna is awesome. Anyway there's been malware activity. The point is everyone has been doing this.
There's been a lot of public data sets and a renaissance of a sort of research because a lot
of people are publishing those hundreds and hundreds of gigabytes of malware so it's
productive of doing this. My opinion, right, is that we've actually got pretty good in
telling malware apart. I already know this is malware so I can know pretty much what family it
comes from by analyzing code paths and things like that. I've seen interesting work in that
field. But actually detecting it, I mean just like that based on this stuff it's not there.
One of the arguments is companies have been doing that forever. I mean we all think
this is brand new but the people who started doing this, I don't know, like 5 years ago I was
hearing about the implementation about story and processing the malware. I don't think we're
much better off. I don't think they've actually cracked the problem. Right? People tell me
the lead researchers here and there left for this other company because the guys at the
av's didn't believe that was the way to go. But I don't know. Maybe there is something here. I
just really, really wanted to be better than those av's on my computer because that crashes my
computer every single time. Let's all hope for that. Right? Again we have to make sure about
the bad data. The bad specific labels that you're picking up. So I mean the problem is if
you're writing a paper and you're doing classification and you're like, yeah, I'm comparing
this evil effort I'm pretty sure your models will be great. But that's not representative of
what is really out there, what people actually use. So imagine what could we possibly use to
compare a piece of evil malware from the internet, the file system, that can access the
camera and send data to a remote controller, if there was legitimate problems that look
exactly like this and do exactly this kind of thing, right, and the point is that looking at
that I almost thought I had it figured out because oh my God I know the thing that tells them
apart is the features. Browsers have sand boxes. So that's what we should be looking for as a
feature. Then I remember, ah, sorry FireFox, we almost had discovered but anyway... Point
is good data. It's not so much having all the bad samples. You also have to have good
representative good samples for the research that you're trying to do. And I'm really beating
myself up here like everyone makes mistakes. Right? How the fuck do I do training of my
machine learning models and all that I have to show for good is stuff for I don't think I even
got to use Chromium. It was not only different classes of things I was comparing at the time and
were more representative of stuff you're going to get, right? And of course the model
was like, oh yeah here is a banana and a fire engine. So can you tell these apart? Yeah I'm
pretty sure I can. So, um, anyway the point is don't use bad data for good samples. I
mean there's much more stuff and anyway you can go for that. But let me try to wrap it up here.
How's it going then in right? How have things been? So what I've been trying to do is really
to extrapolate information from intelligent feeds. So the problem I'm currently trying to
solve is if I got a bunch of samples of bad data and just like mal ware the amount of
information we have increased exponentially for the last years or so. So there's good samples
there that can be used and I'm talking about indicators and domain indicators and things
like that. Right? And so models with this information and give it the same data that an analyst
would use. Get addresses for analyzing. Where are they coming from? Why should you care? You
should care because I think it's much easier to do everything in the computer to do the first
triage. If you want to do it yourself I have some tools you can use to reach them and do
statistical thinking with them. This was covered in the other talk that I did this year. But I
think having everything automatic is kind of cool. And the actual ground truth I use
like I said is the data from the anomaly sorry the data from the feeds and there's a lot of
things I've tacked on. Using Alexa. The good thing about designing this model the way I
did is that I don't have to worry too much about data tampering because the only thing
that I can actually change is the log data inside the company which they would be sending me
to compare this. If they're changing the log data there's a whole bunch of different
problems much right? But they cannot inject something that would look more normal because
I'm not really doing anomaly detection here. All the stuff I'm using is actually external
to any other company. And they're kind of hard. [Indiscernible]. It's hard to
change that then just to try to inject random stuff into your environment. False positives.
The point is it's intrinsic and will always be. It's like the 100 percent in machine security
thing. There's no 100 percent accuracy. If you do model it's 100 percent you've done
something wrong. Go back to the drawing board. That's the biggest thing you can find. I
believe it's all about you creating an actual process around this. Right? And the
idea what I'm presenting this as is it's a way to facilitate triage. You still have to have a
human being to do the last leg of the investigation and they'll be able to actually tell us for
feedback that something was bad or something was not bad. And based on all that I propose a
buyer's guide. Specifically these questions I've talked about, right, these are the
questions you should be asking the machine learning. And for some reason you do not use them.
If they give you bad answers if for some reason you try to argue and they are like no, no, no
we're different, you don't understand. That's the other guys. That's not me. I am very
different. You know, you can hash tag them not all algorithms. Right? Because
that's pretty much the argument they're making and feel free to tag me as well of course. But
just make sure you disagree with me first. Anyway this is us. Don't take my word for it. You
can try it out. We're trying to get this data up and running, we have some limited capacity. Take
your time. I mean it might take a little while and that's pretty much all I have. Thank you very
much, guys. [Applause].