ZOZ: I'm Zoz. This is "Hacking Driverless Vehicles."
My academic background is autonomous robots, but I'm probably best known for a little cable
TV show I did with Kingpin called "Prototype This!" We did some autonomous robot stuff.
Tried to hack together cool demos. For example we did a UAV that would protect
unguarded beaches. So press a button if you are out in the ocean in trouble and it would
fly out and drop of you a life preserver. We learned a lot about unmanned air systems
on that episode and a lot about it racing in the design process.
We also turned our hand to driverless ground vehicles for one of the most pressing technological
challenges in the ground space. Which is maintaining the American lead in high‑speed pizza delivery.
Here is the solution for local delivery. Sharing routing space with pedestrians. Here is the
long distance method. Operating within the shared automobile network.
This was the first ever autonomous crossing of a U.S. highway bridge. Can be hard to convince
human delivery drivers to make this trip. So screw them. Let's let the robots do it.
One of the things I've 'been doing is hosting autonomous vehicle competitions for robo nation.
These are student competitions, university level and below. Ground vehicle competition
two air vehicle competitions ones for boats and one for submarines.
I want to say a few words at the start about the motivation for this talk. I am a huge
fan of unmanned vehicles. I love robots. I think they're the future. And they are definitely
coming. Because there are so many advantages. Energy efficiency. Not having to carry a human
driver. Not having to carry food and water, bathrooms or go out of its way to acquire
them. Same with time efficiency. Not having to deal with fatigue, boredom, taking rest
stops, operator changeover and so on. And all the new applications that are going to
be enabled where it wasn't practical with a human driver.
The revolution is coming. You can't stop it. Even if you want to, it's here.
(Laughter) But like everything else that humans have
ever made, these systems are going to be hacked. So I want to start that conversation now.
I want to start talking about it before the systems are too entrenched for us to go back
on decisions. So what I am definitely not trying to do here
is spread FUD ‑‑ fear, uncertainty and doubt. It's not going to be some kind of alarmist
anti‑robot propaganda. You like that one? (Laughter)
How about this one? That's not it. It's not every presentation where I get to make two
Hitler jokes on the same slide. I couldn't resist. Hope it's not too soon.
But I think that the, you know, this revolution is coming. And really got a sense of this
recently ‑‑ this is footage I shot a year ago at SAUC‑E. This is fire scout.
It's autonomous, unmanned robot helicopter. And it's doing a takeoff run there. And it
just looks like it's sitting still on the ground right. It's so stable. You look at
that and think, yeah, that's how to do it. Let's let the robot drive. I'm not trying
to interfere with this process. This is my point. DEF CON is not a security conference
per se, it is a hacking conference, a hacking convention. To me that's hacking in the old
school sense, right. Figuring out how things work. What is wrong with them if there's stuff
wrong with them and how to improve on them. This talk is about getting people excited
about contributing to that conversation. The analysis, discussion, design and acceptance
of driverless vehicles. So you can think of this as a recruitment talk. Sort of like General
Alexander's NSA keynote last year. The big difference being of course I don't want to
see what you fab to on the internet. You can keep that to yourself.
We're going to talk about vulnerabilities and interfering with driverless vehicles.
But it's out of love. It's kind of tough love, right. Kind of like when you teach people
to swim the Australian way. By putting them in a pool and throwing in a crocodile.
(Laughter) This is true. This is how we all learn.
Speaking of managing vulnerabilities, right, there's the fire scout, it's very stable,
but notice the people operating that robot. They're hiding behind the start CART. Just
in case anything goes wrong. That's a security mindset. So when I talk about exploits and
counter measures I want you guys to think about counter counter‑measures, right. Stuff
that I say, well okay here's how we design around that. Here is how we fix that problem.
Increase the robustness of the system. So unmanned system space is kind of wide.
Means basically no human driver or pilot on board but doesn't necessarily mean you don't
have someone off board controlling or supervising it for an unmanned system. Not actually autonomous.
You might actually have a safety pilot on board or might be carrying passengers. There
might be people on there but they don't have a direct control role.
Of course the military are early adopters. Most of this field has been adopted by military
spending and applications for a number of reasons I'm sure you can guess many of them.
A pretty significant amount of uptake has been the airspace. For example this is global
hawk. If you look at global hawk flight hours, it's looking pretty much like an exponential
curve, right. That's because it works well. There's a lot of simplifications that apply
to the airspace but they really want to push the changes down to other domains. 12 years
ago congress insisted one‑third of all operational ground vehicles were to be unmanned by 2015.
Now congress can put up the money and say whatever the hell it likes. Doesn't mean they're
going to get it right? We're clearly not going to make the 2015 deadline but the will is
there. I'm not going talk a lot about military vehicles in this talk but the capabilities
are largely classified so it's very speculative. They already have a really active interest
in resistance to adversarial engagements. So they already think a lot about the security
stuff. They have the highest quality sensors and, of course, hopefully most of us will
never encounter one. Unless Edward Snowden is in the audience. Hopefully we'll not encounter
one of these. Here's just a quick example of a military
UGV. A driverless ground vehicle specifically designed with threats in mind. You can see
a lot of sensors. Designed looking for people. It's got weapons on board. You just have to
get close enough to this thing to press one of these kill switches.
(Laughter) It's kind of like putting an unshielded reactor
exhaust port on a death star right. Presumably they'll remember to remove those in the final
version. Let's start thinking where these things are
going to show up in our backyards. So we've got transportation. Nothing bad about these
guys by the way. We love these guys. They're friends. Not singling them out but they're
pioneering stuff in transportation. Oceanography. Film making is big. If you see the film making
conferences you see UAV's everywhere. Some of the weird esoteric ones you wouldn't think
of like power line inspection with UAV's and, of course, logistics like the pizza delivery.
Lots and lots more. And there are two main priorities that industry
advocacy group has for unmanned systems in the civil sphere. Precision agriculture. A
lot of combine harvesters now are practically mobile robots. And secondly self‑driving
cars. This is big right. Wide applicability over the entire country.
Roadblocks to up take the civil domain. Shared infrastructure. They have to share the stuff
with humans. It's much tougher if the robots have to interoperate with humans. Hand in
hand with that goes acceptance. Do they trust the safety. You got to convince the public.
Also privacy is important. So we're talking about safety and robustness,
the fun stuff here is failure. Let's take a look at that. Here is a couple of classic
failures. First of all a UAV failure. This is the OCU 3 star. There's a surviving example
in the Smithsonian. It was supposed to cost $10 million a unit ultimately. The first
couple were very expensive. The first prototype failed on its second flight test. On it's
second takeoff actually. I've seen the video of this crash with my
own eyes but was not able to obtain it for you guys. So you just have to imagine. Here
is a non‑crash takeoff. Just extrapolate from that and imagine if it didn't take off
like that and its keeping down the runway and starts to wobble up and down and oscillate.
And the oscillations get more and more and it pitched its nose up and comes down hard
and it's a huge fire ball and is millions of dollars down the trade. This is a quote
about what happened. Here's what as it was explained to me by researchers involved this
is what happened. They had modeled the takeoff run with a flight control system on an asphalt
runway. The second flight takeoff was on a pre‑fabricated concrete runway. The cracks
between the concrete panels were under damped and they set up this oscillation that eventually
caused the failure of the vehicle. Right. So just small impulses.
The moral of that story is the expectations of the designers are critical. Even a seemingly
trivial detail like runway composition can mean the difference between success and failure
of these systems. If there's going to be exploitation there's a good chance it's going to happen
at these cracks between the boundaries of the designers expectations.
Here's a second example. This was the favorite vehicle to win the first challenge in 2004.
This is the desert race from Los Angeles to Las Vegas fully autonomous. This vehicle
was done by a CMU off‑shoot called red team racing. And it got a few miles. Seven miles
or something like that, before it tack a hairpin turn wrong and ran off the side of the road.
Its engine caught fire and it was all over. While we wait for that video to catch up,
let's see if we can play both of them at the same time. Hey look at that. There it is.
Failed. What went wrong? Apart from the fact that
diesel engines don't like running on weird angles? They had a huge team. They had extensively
mapped the course beforehand. Even though they only got the final route two hours beforehand
they knew every road in the area. Very precise. They had people walking that course with GPS
receivers. One of the people on that team told me their map was so good they could have
just about made it on map data alone. The big problem was they paid too much attention
to the other sensors. If they just ignored the laser range finder they would have gotten
through just fine. The moral of the story is the robot faces
a constant battle of deciding what it knows best. What information is reliable and what
isn't. Correctly estimating the state you can't observe is the key to all decision making
on uncertainty. So hacks and exploits have some of their best chances of succeeding if
they subvert or undermine that state of summation process.
Let's take a look at some of these logic structures at a higher level. Just like with humans we
can think about behavioral logic of a robot in hierarchal fashion. At the bot we have
control loops, stability maintenance. This stuff typically runs independently at a high
cycle rate. Might run on a completely different computer from other stuff. Sometimes I'll
see a robot that's completely crashed. It's not doing a damn thing but maintaining perfect
stability in the water and air. Just hanging out there. Above that you might see some of
that collision avoidance. That's preservation of the robot. Kind of like (inaudible) third
law. Taking precedence over everything except low
level control. Above that navigation and localization. That might be all of the mission in it's just
navigation. Then above that high level mission task plan, stuff like that.
So what can we take away from that arrangement? First of all, there's an implicit dependency.
If you attack at lower level in the hierarchy you can defeat everything above it. If the
robot can't maintain functionality there it can't at the higher levels. I like to think
of it for those people who have office jobs no one thinks about filling out their TPS
report while they're actually being kicked in the balls. You can go back to work and
try that and see if it's true. But, secondly, more engineering effort might
have been spent programming ‑‑ guaranteeing robustness at lower levels. Stability is super
important but getting lost once in awhile you might be able to recover from that. So
the lower layers might be juicier attack targets but might be better defended and harder to
find bugs in. So a couple of examples that I mentioned before
from prototype this just looking at the way things were arranged, the life saving drone
had an auto pilot that did all the stability maintenance, has low level control loops for
all the basic air worthiness for the different environmental conditions you might expect
to encounter. Nothing in the way of collision avoidance; this is missing from just about
all UAVs. That's one of the things that needs to implemented and designed in order for the
shared space arrangement to happen. Navigational localization is GPS based. That has way points.
And this involves also control loops. PID loops controlling all the aircraft approaches
those way points and when it discovers ‑‑ when it decides whether or not its hit one
or not or whether it should go back around for another try.
Then at the top we had our bumming run planner that would set up a temporary weigh point.
So the system of course fully vulnerable to collision. Because there was no effort to
not be. And the high level logic depends on one single sensor ‑‑ the GPS. That single
point of failure is a big vulnerability. And they're really common in the robot fear.
Local pizza delivery has to have all kinds of control for stability maintenance because
it's balancing on two wheels. It's got to do weight shifting for when the pizza gets
removed and the center of gravity changes. Lots and lots of collision avoidance. Pretty
much almost everything that the system does. The main strength is dealing with those dynamic
obstacles that aren't on the map. At the high level, navigation by route planning from a
map that's pre‑generated using simultaneous localization and mapping. That's where the
discrimination between static and dynamic obstacles happens. Then the high level task
is a simple one. Just to dispense the correct pizza when the correct credit card gets given
to it. So this kind of system is vulnerable to redirection,
trapping and map confusion attacks. All those things that attack where the robot thinks
it is. And, of course, you can always try to get the pizza out if you didn't pay for
it. Now that we're thinking about that logic hierarchy
let's look at the operation of the hierarchy and what kind of logic is going on there.
There's usually some form of state machine. Represents the mission and what the designers
envisioned. So the robot considers itself to be in a state and can stay in this state
or transaction to a new state. They define what logic the robot runs at any
given time. These states may correspond to tasks and the transitions may be task completion
or contact switches caused by things like priority shifts. Might be simple time‑outs,
for example, anything like that. They may contain subordinate states.
For math people watching, if this looks like a Mark off chain, there's a good reason for
that. These feature a lot in robot control systems. The thing to be aware of with these
machines is the machine is ‑‑ the state machine is not necessarily deterministic.
Right? Just because we think we're in a state doesn't mean we're actually there. And so
there's this hidden state that the robot has ‑‑ that it can't necessarily observe and has
to figure out. That's where things get tricky. So to put some labels on this stuff and not
have it be totally abstract here is the robosub mission. I've chosen this because it's kind
of linear and easily broken down into various mission states. First the sub has to navigate
through a start gate. This is often done open loop. Just point it in the right direction
and drive it for a certain amount of time. But then you have to start making decisions.
The sub's got to start looking for a buoy then trying to touch it. It's got to decide
can it see the buoy? Has it touched it yet? Should we try again? You got some choices.
And then you've got to start looking for a path on the bot of the pool. And you've got
more choices once you find it. All the different sub tasks you might want to do. The obstacle
course. Identifying the targets and dropping markers on them. Finding the torpedo targets
and firing torpedoes through them. Then underwater manipulation task. You've got to determine
the state of that in terms of finding it and whether you have managed to complete that
task or make progress on it. You can time‑out from all those. A lot of these are vision
guided but a certain point you can transition from anywhere in the graph. It uses a different
sensor. So you've got to find the pinger and go and retrieve a package.
So looking at this from the point of view of second guessing the designers, where are
the vulnerabilities and potential exploits? There's the package. They may be in the state
estimation. What does the robot think it's trying to do versus where it is. The transition
between states. Can we spoof them or prevent them from occurring? Or are there bugs in
the states themselves? Just unexpected conditions or results. Here is a key thing. When designers
watched the robot in action they don't necessarily know even though they programmed the whole
thing why it's doing what it's doing. They can only guess from that output. So until
you see the logs you don't necessarily know. So the would‑be exploiter also has to put
themselves in the mind of the designer and think about what might they have been thinking
about and what might they have got wrong? But I don't want to talk about attacks that
would work on any vehicle, even human driven vehicles, because there's no point in that.
Like digging a big pit and camouflaging it I. Want to talk about relevant physical attacks.
The input mechanisms on the sensors. They can be active or passive. Which is a important
distinction that we'll cover as we go. Some common examples are GPS of course. Laser range
finder, cameras, millimeter wave radar, visual compass and measurement unit, wheel encoders.
Then for the specialty vehicles like subs we have things like Doppler velocity bloggers,
scanning sonars, pressure fuses for the air. In addition to all this there's the map. We'll
definitely talk about that, too. So sensors don't give a perfect picture of
the world. Just the best guess. And you've got plenty of sources of uncertainty. They
all know about noise of course. A constant battle associated with noise is drift. Latency
and update rate come into play. So when we were doing the life saving drone, we had one
update on the GPS. That meant with unknown time stamp on that. That meant that the vehicle
could be anywhere within, like, a 70‑meter distance when we got that position estimation.
Getting a position estimation that could is not valid to the tune of up to 70 meters
that's hard to do a five‑meter precision bombing run.
You have to model these uncertainties under various assumptions. You need to know what
the underlying noise models are. A lot of people don't use GPS because they don't ‑‑
they can't get from individual units what the noise model of the GPS is so they can't
develop a noise model that's super reliable. You might think fusing sensors together might
be more useful than a single sensor. That's true in many cases. Fusing and registering
together can be more useful than taking separate sensors.
But what do you do when a sensor disagrees? Which one do you trust and how much? The robustness
of the robot in the end may come down to how smart is it at discounting one single bad
or spoofed sensor? Even though it might have a whole suite of sensors on board.
So let's look at sensor attacks. Two basic kinds ‑‑ denial, basically preventing
the sensor from recovering useful data. Then spoofing. Causing the sensor to retrieve information
that is specifically incorrect that the attacker wants it to retrieve. Then you've got a basic
attack mode choice. You can directly attack the sensors. Give them instantaneous bad data
or try to mess with the aggregated data they're accumulating over time. I'm going quickly
go through most of the common sensors and leave the specialized for another time.
GPS we know is a major reference for vehicles that have access to it in the atmosphere.
Denial is straight up jamming. You can buy a jammer from a number of sketchy Chinese
websites. You can also find plans online if you want to build your own. It's blow throwing
a big bucket of noise at the frequencies. The transmissions are weak from the satellite
so you just overpower them. You can also spoof them. You need to generate fake GPS signals
at a higher power than the satellites themselves to override the receiver. This has been demonstrated
with UAVs. This is a group from UT Austin demonstrating taking over a UAV using GPS.
So the attacker on the left is broadcasting a GPS signal that is at a higher power and
first is aligning to the GPS signal that the UAV is really receiving. Those three dots
are match filter trackers on that output that's finding that peak. And then you start to move
your signal off and you take the tracking points with you. So now convincing the UAV
it's in a position it's not really in and its flight control system is trying to correct
that and moving it somewhere else. Here is some video of doing it to the real
UAV. They're going to take control over it and convince it via GPS that it is moving
upward at a certain speed. And the flight control system is going to try to account
for that and move the helicopter down. You can see if the safety pilot doesn't take over
here, you could drive it straight into the ground.
This technique was claimed to be used by the Iranians when they brought down an RQ seven
surveillance. They said they spoofed night the ground. This is widely believed to be
a model. Because the original crashed. But seems unlikely to me. Because the military
systems use encrypted GPS and the military don't rely on GPS for UAVs because there are
jamming and less likely spoofing for military GPS. But we do know for civilian systems it's
easily jammed due to the weak amplitude. Easily spoofed, I mean.
In the civilian realm the GPS is used as a primary sensor. Take the GPS and maybe resolve
the last couple of meters with the laser range finders. It's important to point out those
sensors all have filters on them. The GPS will drag the vehicle slowly from incorrect
trajectory rather than just snapping it off its path.
Here is from 2005. Again without access to the logs it's hard to know exactly what's
happening. But looks like because there's nothing on the road looks like we're seeing
GPS drift off the road that is then being corrected by laser range finders. So that's
typical of what you might see if something's relying on GPS.
Here's another challenge example. Looks like a GPS run that is drifting off the road here.
Who knew that a van could drive over a Josey barrier?
(Laughter) Something maybe for the next DEF CON canon
ball run to keep in mind. Next up laser range finder. Originally a sensor
for industrial automation but then the robot guys got hold of them and were like this is
awesome so we're going on use these outside. Mechanically scanned via rotating mirror.
They measure time of flight. It is an active sensor. Depends on a return signal coming
back. These are primarily used for collision avoidance and map making. I don't know why
I do these bullets because I always forget to advance them. They return a point cloud
of reflected distances within the laser range. So you can do denial on them by actively overpowering
then or by preventing a return signal with dust, smoke, mist. You can also spoof them
by manipulating the surface absorbance of the things they're looking for. So, basically,
manipulating absorbance and reflectivity to give the receiver incorrect information about
what it's looking at. After a 2D sensor. It's highly orientation
dependent. It's often mounted in a push broom configuration. What that means is you are
looking for obstacles nicely but if the ground slopes up it can look like a brick wall. You
can see ‑‑ this is from pizza delivery. On the left is the Light R. You can see that
you get these ranges stopping when you see an obstacle. If you look over there to the,
you know ‑‑ as it sort of turns off to the right where the street slopes up, you
can see that range just drop away to nothing. Kind of looks obstacle like. In addition if
it's shooting down over the low obstacle or like a curb it can miss it entirely. So it
can fall in a ditch if it is orientated right. Active emission sensor. It only return that's
active signal back to the receiver. So no return means that it assumes nothing is there.
Think about that. Over the horizon and out of range returns no signal. So most of the
world returns no data. What that means is thing that's absorb in the laser frequency
look exactly like nothing. Things that are transparent in the laser frequencies also
look like nothing. So if you were to paint an absorbent tunnel on a wall it's just‑like
Wile E. Coyote. The robot would not see that. If you were to make obstacles out of glass,
it sees right through them. Glass is transparent but also reflective. There's a limit to this.
Might miss a bottle but probably going to see a (inaudible). Some also have this multi‑echo
suppression so if you have glass close to them it's designed to ignore them.
It's all about what gets returned to the sensor. Reflective things confuse it. For example
a puddle on the road is very reflective. They can make far away things like obstacles look
near if the angles match up. Right? So this is something that robot actually has to deal
with. Or if there's nothing to be reflected the signal goes out into space and that lack
of return makes it look like a big hole in the road. Not just water in puddles. Even
fresh asphalt cannot give a return and look like a big hole in the road.
One of the challenge vehicles I won't say which one ran into a brand‑new black SUV.
Because it was so shiny and reflective and recently washed that it looked like nothing.
So even a new car can be a problem. That's what the millimeter wave radar is for. You
can use reflective surfaces to make things look like a switch the vehicle won't consider
safe to drive over and will have to take a different route. People out there, bad guys
out there have good reason to try and use these kind of techniques.
This is just something for fun. I found this on my travels on the internet. These are documents
from al Qaeda in the Arabia Peninsula that were captured in Timbuktu. You have to trust
me what it says in Item 2 referring to a GPS jammer. This is a document on how to avoid
UAVs and drone strikes. Item three advises them to place reflective plates on their vehicles
to reflect off the laser designator. To make the missile miss slightly that. Could be the
difference between life and death. Of course for this to work you would need a material
reflective in the laser wavelength. But it's on the list of techniques.
Laser reflectance is also a feature. The road mostly gives a decent return unless it's fresh
asphalt. But the white lines are quite reflective. They look like gaps in the road. They use
this to do road line detection. So a fun consequence of this is that you could make fake road markings
in a way that's invisible to the human like black on black but the robot is going to see
them perfectly. So you could paint some black on black swervy lines, for example. A human
does not know why the robot is swerving all over the road. Or you could even try to be
more kind about this and leave the lines as they are but do the black on black as hidden
messages for the humans back at base when they go and look at the map.
(Laughter) Cameras are used as well. But not as much
as you might think. Vision is really hard. Specialized object detection sometimes, sometimes
stereos used to get a def map but it's noisy. Often what people do is colorize the data
with cameras. You are registering your data to this color information from your camera.
Why isn't this video going? Come on. So this is stop a challenge from '05. This
is Stanley. The way that Stanley drove so fast is it used lasers to get an idea of what
was road and what wasn't road. Then used the camera information to match colors and say
everything that looks like road in front of me I'm going extrapolate that based on color
and that's where I can drive. I'm going compute my path based on that. It was a really nice
technique. Of course, cameras are easily dazzled and subjected to blinding attacks just like
we always talk about with anti‑surveillance stuff. You can also do spoofing with it. Right.
Because like camouflage works for the Mach 1 human eyeball you can mess with the color
reception. If something is saying there's stuff that looks like road everything that
is that color is road just make obstacles out of road color stuff. Repeating patterns
confuse the hell out of stereo cameras. They don't know what matches with what.
Millimeter wave radar. This is used in vehicle applications to get through obstacle avoidance
stuff. You have probably seen it in one form or another because this is the stuff at the
airport that shows off your junk to the TSA. Primarily used for collision avoidance looking
for things that will reflect the radar well like signs and other vehicles. Lower resolution
than the Light R. Produces fuzzy images and lives in a weird world where everything is
a mirror. So lots of stuff is reflective. You can't use it a lot to find decision making.
Like any raider you can confuse it with chaffing. Spitting out thing that's reflect the signal.
Also gets a big return from things like sign. So an overhead sign the robot might be programmed
to ignore that as an obstacle because it's getting a huge return from it. If it happens
to be a dynamic obstacle underneath that it might miss it.
Compass stands for inertia integration unit. This is the primary navigation sensor for
a lot of systems because they can be very, very robust and because they can be very resistance
to any kind of spoofing or attack. You can get everything from hi‑fi dealt models to
these hobbyist ones that are often pretty noisy. A commercial aircraft IMU or commercial
robot IMU like a Boeing 777 IMU has a cumulative error of .1 percent of total distance traveled.
So they're used on a lot of arctic UAVs. You travel 300 kilometers and when you get to
the destination you pop up and get a GPS fix your cumulative distance error is about 300 meters.
That's easy to deal with. Very difficult to interfere with. They're all on board. They're
fully encapsulated. You are just recording what the robot feels. Which is why military
depends on them. However, the compasses very susceptible to magnetic field.
Another part of doing this dead reckoning is wheel odometery. Encoders on the wheels
and giving rotation information that you can integrate out. They're really key component
because they give you good speed information relative to the ground and let you know when
you are stopped for sure. One of the only sensors that can let you know when you are
stopped. Which you might not know how to do especially in a tunnel. So important what
happened when we did our pizza delivery we had to get interrupted at the end because
it took this turn tight and scraped off the wheel encoder. That was bad enough coming
off the bridge we had to rescue the vehicle at this point. There it is hanging off the
side. Very sad. So there are some things you can look at to increase the wheel odometery
uncertainty or just trying to remove it. Odometry drift by changing the wheel diameter. Slippery
surfaces might cause drift. And, of course, when you remove them potentially unpredictable
behavior or stoppage. Now we've talked about all these physical
attacks we can do. Sounding a bit like a James bond car package right. We've got our GPS
jammer to knockout the absolute localization. We've got smoke, dust or vapor ejectors to
confuse the Light R, the Light R lasers perform particularly bad in mist. We've got
chaff dispensers for the millimeter wave radar. Could be sparse enough that a human wouldn't
see anything but the fine metal particles that make the robot stopped suddenly. Glass
Cal drops. If you are James bond you have to have Cal drops. Then, of course, an oil
slick to prevent it from telling when the car's really stopped. Nice if you got a James
bond budget and, of course, an Aston Martin to put them on. But there's another really
important thing to talk about besides all these sensor attacks ‑‑ the map. The
old school mobile robots went into the world pretty much knowing nothing about it and just
using sensors. Lots of emphasis on doing slam and using sensors to build up the map as they
went along. But that aggregated map data is so cheap and ubiquitous there's a huge emphasis
on pre‑acquired map data. Think about for example what Google does as another huge part
of their business, right, mapping. So the map is so comprehensive it's treated like
the ground truth. Here's an example, a video of mapping with sensor data.
This is the kind of map we created for the treasure island pizza delivery. The map is
so comprehensive it's treated as the ground truth. It's really powerful because it reduces
the recognition load on the robot in realtime. The robot can instantly map its sensor data
to static features such as traffic lights, trees, vegetation, speed control and traffic
signs, speed bumps. But reliance on one single thing even a big thing like this cab also
be a weakness. A single point of failure. So there are potentially all kind of things
that we can make use of if they're relying too much on this map. Traffic lights for example.
This is how some vehicles locate traffic lights. 100 percent robustly locating a traffic light
is hard right. They could be anywhere that you can see. You've got to do vision to find
them. And you've got to have 100 percent. You can't just have your robot go around occasionally
blowing through a red light. Because, you know, the robot, you know, it doesn't have
to get ‑‑ it's got to do this. Because you are on the same road as humans. The human
is expecting that if the light's red that guy is going to stop.
But if you've got a map of every single traffic light and it's registered to GPS so from everywhere
you see it you know exactly where to look to see that traffic light, then detecting
the color of the traffic light is trivial. You know where it is. You just have to look
for that blob. But now you've got a potential schism between the human and robot's assumptions.
Because the human assumes the robot can see the light under any conditions. But the robot
assumes the light is where the map says it is. So if it gets moved, shifted around, human
drivers have no problem with that. But the robot isn't going to recognize the new state.
Registration detection is another example. Let's say the robot has some kind of rules
to determine what is vegetation and what kind it is. So you might have some blob of vegetation.
You've got the colorized Light R looking for green stuff. Oar you might have some kind
of transmission classifier. How much of the laser is coming back? The vegetation is not
100 percent reflective. And you know that if it's on the ground it's grass and you can
drive over it. And you've got all your trees mapped out. So you know where those are all
and you treat them as static obstacles. So big deal. Since the last time you drove that
route the trees have grown and the foliage is overhanging the road. And suddenly it's
now spotted by the row about the as dynamic obstacles. Here is this vegetation and the
robot is going crazy and stopping everywhere even though it's light vegetation it could
drive right through. So the meta‑point here is that the rules
for guessing what things are that the human designers have come up with are often brutal.
They represent the best efforts that the designers have been able to do to design acceptable
tests. But when you have this great thing that looks like the truth in the form of a
map, you come to depend it on. And dependence on the map may exacerbate the brutalness of
the rules in the way that opens the door for exploitation.
You got to have constant updates. This video is an example of simulation of what robot
vehicles could do at an intersection if they have completely reliable realtime local information.
There's no need to stop they just go through the intersection. Totally terrifying right?
(Laughter) So you could do that if you have that map
update. But if you have a local map, then you can't do that kind of stuff and you're
vulnerable to unexpected real world features. If you have a remote map, then you are vulnerable
to all those attacks on the network. Right? So you can do denial. You can jam your 4G
map updates and also spoofing. Man in the middle the map dater as it comes through.
All of the techniques we're familiar with from other parts of DEF CON.
So looking at some of the general vulnerabilities here let's talk about the logic structures
and how we might craft and exploit. So we want to maximize the uncertainty facing the
vehicle in order to cause mission failure. Some of the maneuvers that a vehicle needs
to do when it can only do on‑board sensing or more uncertain and, therefore, more fragile
than others because of the geometry. One example is a right turn on right. Upcoming
traffic from the left and the view could be blocked from other vehicles. The same problem
a human driver has. The robot is going to be necessarily more cautious here and this
provides an opportunity to trick it. So we might want to force the robot to require manual
assistance, right. To be unable to continue without supervision. We might want to confuse
or annoy the occupants so they abandon the robot vehicle transportation. Even regularly
dropping the vehicle back into manual mode might do that.
Inconveniencing the other road users, right. If the robot stops and blocks traffic, you've
got robot road rage. So getting back to these fragile maneuvers like the right turn on red,
if you can make it too uncertain so it sits there and blocks traffic or if it ventures
out at the wrong time and gets T‑boned, then that's the vulnerability to be exploited
here. Now if you have physical access to the physical, you can do these kind of physical
attacks on the logic. Kind of like a 21st Century version of slashing the tires. Obviously highly
dependent on the configuration and mission. But if you have the abilities for example
to get near the compass and stick a device only a millimeter away if it has a strong
electronic magnet that's got a 4G (inaudible) you can figure out where it is and mess with
its compass at just the right time. An obvious style of attack is re‑directing
the robot away from where it's suppose to go. Or even trapping it in a spot it can't
get out of. Attack on the collision avoidance and navigation layers forcing it to postpone
its high level tasks. Have obstacle that's move and you can force the robot to stop.
If you can put obstacles around it so it can't get out. Or you can have moving obstacles
that guide the robot off the path somewhere you want to take it.
(Laughter) You could even have obstacles formed of other
robots. Artificial traffic lights is another one. The robot is depending on the map. You
can't put up a fake light but if you can use the real light and modify it so the robot
thinks nights a different state the human would figure it out right away but the robot
has to stop. Another attack is clobbering. This is a term from the cruise missile world.
Make the robot run into something. Like a piece of terrain or something. Subverting
its collision avoidance ultimately to incapacitate the vehicle perhaps. So you might want to
completely crash into something or might want to scrape off sensors. You can do this by
doing subtle deviations from the map, changing things on the map, especially near fragile
maneuvers. Or by changing things post mapping. You can do it by imitating light vegetation
so it thinks it can go through it but it can't. Simulating obstacles at speed. So the robot
has to stop suddenly or swerve. Disguising entrance walls. Like a fake tunnel. Putting
materials within the localization noise so it goes too close to one side or overhanging
piece that scrapes off the top sensors. Or obstacle as I mentioned underneath big
radar reflectors so the robot is normally programmed to ignore. Like big overhead traffic
signs. So now that I've said all of these things
that you could potentially do to mess up a robot, mean and nasty things to driverless
vehicles, I want to reiterate driverless vehicles are cool. Don't do any of these things.
(Laughter) I'm saying this: Don't hassle the Hoff. I
mean, don't ax saw the bot. But instead if you are into autonomous vehicles and getting
involved in the future of transportation why not get involved in the hard challenge of
actually making them work. Screwing them up is easy. Getting them right is the cool part.
So for any students here I'd like to close out by just mentioning some stuff about the
autonomous robot competitions. SAUC‑E and Robobot and Robosub. I want more DEF CON
people involved in these competitions. Because DEF CON people like to push the envelope.
So here's just a quick run through of the tasks that are done and what you guys might
be interested in. SAUC‑E wave point navigation, search for
and identifying secret symbols on the ground and connecting to an area Wi-Fi network and
downloading secret codes. This is DEF CON right here ‑‑ secret codes. Also coming
soon, hopefully, package dropping. Legit excuses to write bombing runs for UAVs. Cool challenges
involve visual map making, registering with GPS. If you are into Panorama stitching or
automatic target ID not a lot of teams are doing this. So there's lots much opportunity
to put together a sophisticated entry. Roboboat is one of the most difficult competitions.
We've got channel navigation, directing con canons on to a target, identifying thermally
hot items on ground stations, disabling water sprays, deploying a rover and retrieving a
package. A team this year had a boat that launched
a quad‑copter to retrieve a package. Capture the flag from another boat. So this is all
DEF CON stuff, right? Camera and light sensor integration.
This year a team had a Light R they couldn't afford one off the shelf so they hacked one
out of a robotic vacuum cleaner and reversed engineered it. That's what we need more. Discrimination
between vegetation and water and detecting when the robot is stuck on things. People
haven't been good at that. That's where we need people with a security mindset to think
about it. Robosub. The big one. Underwater is the poster
child for autonomy. Because you don't have the communications bandwidth to remote control.
So 3D navigation, target recognition, shooting torpedoes, manipulating objects and package
recovery with a sonar pinger. All without GPS. I'm flying through this because I know
I'm running over time. One big thing again I think like we need the
hacker mindset for is all these thing that's people don't think of before they go into
water. Like thermal management the thing I most want people to be involved in is because
I think the rules need to be hacked. They're there to have loopholes found. That's what
people in this room do. This UAV is the scan eagle. It doesn't need
a runway. Planes need runways? Hell no. It's fly night ‑‑ a cable and catch it. This
is what I want to see. Nontraditional vehicles. There's dimension limits but they apply at
the start. Who is to say you can't change the day mentions while doing it and hack things
that way? Swarms of vehicles. Let's get Voltron on this stuff. I think this is the ultimate
hacker sport. It's technologically awesome. It's bloody hard and there are loopholes to
be exploited. I hope people here in the audience who are
students and have eligibility for this will check them out. There's a Big Daddy Roboboat
next year in Singapore. They'll give you a $50,000 boat and $25,000 for sensors if your
team is selected to complete in it. That's it. At the start of this talk.
(Applause) Just before the goons drag me off I want one
more propaganda poster. When I was doing my disclaimer about I didn't want to spread fear,
uncertainty and doubt. Here is a propaganda poster about how we might let the row live
the Austin powers dream. Letting the robot take care of driving while we get down to
business in the back seat. I hope you will get involved in making that come true.
Thank you!