>> Here's Mr. Tal. I'll let him talk to ya and start the talk. So, enjoy. [Applause] >> This is
working, great. All right, good morning. Good morning everyone. Good. I'm getting an interactive
audience, good. Um, so this talk is titled "I Hunt TR-069 Admins." Let's get it started
off. So just a quick disclaimer here. Corporate legal wouldn't let us you know, actually do
most of this stuff but you'll see. So first of all I would like to credit my personal role
model which, and first thank Taylor Swift for providing inspiration you know, and words
of wisdom which I've tried to incorporate into some of the slides. [Applause]. So yeah. But
seriously if you're not following this account, you should because it's hilarious.
Okay. So and obligatory who am I? My name is Shahar Tal. Yeah, such profile pic, much coolness,
very professional. I have an amazing wife and daughter waiting for me back home. I've
just spent almost 10 years as an officer in the Israeli defense forces. Last year I joined check
point where I lead a team in the malware vulnerability research group. We occasionally do some
interesting things and hopefully you will get to see us more this year. So this is my right leg.
Is that, can you see that? I mean, ok. This is my right leg. My left one has a hexadecimal
dump. And if you want to see it just come after the talk. So starting off with an agenda, I'm
gonna give you an introduction to what TR-069 is. Okay. [Laughter] And I'm going to tell
you why you should care and I'm gonna try to give you a sense of what the landscape feels like.
We'll move on to show a few examples of footage and conclude with a short summary. So just
beginning with you know, residential gateways. Sometimes referred to as home routers or
Soho routers. And we all know that this is an incredibly unprotected domain. In recent
years it just gave us more and more good examples of that. So we have the Internet census
which showed us that there are literally millions of devices out there waiting to be
controlled with default passwords. We've seen rapid severance, great coverage, VPN
exploitation, and last year showing us it's also quite easy to run code on so many devices.
We've also seen campaigns targeting devices for criminal activity and you know, there are
really too many to mention but some of these guys, maybe some of them are actually here, have
been doing great work demonstrating really, what, that the state of affairs of this
domain. And I'm here to kind of extend this notion. So what is TR-069? TR stands for Technical
Report, just kind of like a, kind of a RFC and you know, contrary to what you'd expect,
the title and numbering were not given by a giggling teenager but a more or less respectable body
called The Broadband Forum, which is previously known as The DSL Forum and The Frame Relay
Forum. And they're basically known as-- they are a consortium of many players in the broadband
market and they're attempting to find standards and common goals. So in 2004 they released this
CPE WAN management protocol and which is CWMP or more commonly referred to as simply TR-069.
And the specification went through five amendments actually which were released late last
year, and to tell you the truth it's a pretty boring 228 page document filled with, one sec,
filled with fancy words like mechanism specifications or schema definitions and you know,
they actually describe pretty simple things. So let me try and summarize 200 pages for you.
This is how TR-069 sessions or provisioning sessions work. Okay, on the right side we have
the CPE. Which is the TR-069 client. That's your home router. On the left side you have the
TR-069 server which is called an ACS or an auto configuration server. And they talk in basic
SOAP RPC and XML or HTTP and it's that simple. It's important to mention that the client
always initiates the session which is a single TCP connection over which RPC's are made back
and forth. The client begins with an inform which is studying the reason or why is the session
is initiated and the ACS follows with provisioning functions such as get parameter values, or set
parameters values. It really is that simple. And there is a dual authentication mechanism. The
CPE should make sure it has a verified ACS and the verified ACS should only accept sessions
from authenticated CPE's. And here's a quick example just to get, just to be-. This is an ACS
set parameter value call for a certain CPE. And this is in fact -- let me see if I can -- yeah.
Okay. So this is set parameters value call and this is assigning a new value for the management
server URL key which basically means this is updating the configuration ACS URL for this
device for ACS super-securized.com, whatever. So these are only some of the
names -- oh, sorry. You see some really big names up there have active TR-069 deployments. And
as you can understand this is not some obscure protocol. This is a growing trend to adopt it
everywhere. And this is a widely used defacto device management standard. Yeah so research in
2010 estimated 115 million TR-069 devices online, 70 percent of which are Gateways
and you know, today it's probably more. How about this? Some very interesting results
from the ZMap guys, Zachary and his friends at University of Michigan. And they show that
7547 which is a default WMP port is actually the second most popular open port in the world.
This, I mean, this is no joke. This is a serious, serious protocol. So let's talk about
some of the functional use cases for TR-069. So what can you do with ACS's? You can provision
devices which is what you call zero configuration. Just remote management by the tech report
guys, you know, when you call the call center and they fix your router, whatever they do
there, can monitor for faults, for malicious activity even, diagnostics, performance
measurements, you can replace, fix faulty configurations. And also deploy upgraded firmware.
So I'm betting at least some of you are surprised with that last bullet you know, and this is so
you can understanding the power that's entrusted in these sessions. Right? So this, and
this is a power that service providers don't necessarily want you to know about. I mean, think
about this for a second. You know, without getting too deep into the trusting your compiler
issues you know, who do you really trust to run code on your devices on demand? You know? How
about when it's done silently? Remotely? With elevated permissions? You know I might
trust some of these you know, very big vendors like Microsoft, they have heavily protected
update servers, whatever, I might trust them, but my ISP, I'm not sure. So yeah. ACS's
have that trust and they can remotely manage all their things. So here's a short story.
This is my home router or at least it used to be. You know I'm starting to learn about
TR-069 and I'm poking around looking for the TR-069 configuration and there's no
evident pages with these settings. I know you can't really read it from there but
nothing says TR-069. And you know oh, well I should go into expert mode. Right? Oh why not?
I'm an expert right? So I go into expert mode but still nothing saying TR-069. So maybe
it's under remote management. I clicked it and then no, that's only for web management, which
is, you know, properly turned off by the way. And this doesn't makes sense to me because I
know, I know these devices have TR-069 enabled and on the specs I've heard that my provider uses
this. So as it turns out my provider customized the firmware to hide the fact that they were
in control this was done using the highly sophisticated HTML commenting technique.
[Applause]. So you know, I entered the URL, and then there I get the details and I have the
set up screen and you know suddenly I see that there is no legitimate way to turn this off.
So the disabling, the informs is just like the periodic hello messages, it just, I mean, it
still has no real effect. What you actually could do is change the ACS URL but never mind about
that. So you know in the later firmware version they actually changed this page to be you
know, just a status page, now there really is no legitimate way without you know, getting a
root on your device, to turn off TR-069. And so the point is someone out there knows that
they have a lot of power in their hands and they clearly don't want me to know about it.
So with that context in mind let's take a broader look on TR-069 architecture. And, sorry.
Again on the right side we have a CPE, the client which can also act as a proxy from any other
home devices, such as VOIP phones, Nest devices, set up boxes, whatever wants to be
controlled. And on the other side we have the ACS. That's the, you know, the one server
that's in control. On its north bound side it is attached to ISP internal assets like the OSS,
BSS, Billing, Policy, the call center of course, and this south bound interface is where the
CPE's go. So, wait. What? There's a single machine in this design you know, that's in
control of an entire fleet of devices for that ISP which would make it in fact the single point
of ponage. So if ever there was a high value target in an ISP that would be it. Well luckily
this south bound interface is using the most trusted and secure line that you can think
of which is the public Internet. [Laughing]. So you know, looking at this from the APT or you
know, nation state attacker's perspective, so you can take control of this one target, one
accessible target and you're in complete control by design over millions of devices. This is
like, that's the NSA's dream right there. So now that we all understand that TR-069 is very
powerful what could you actually do if you had control of an ACS? I see my shot coming up. So I'm
just gonna pause for that one sec. It's very, very dramatic, very dramatic. [Applause]. >>
Keep talking. >> I'm gonna keep talking. All right. So what could you actually do? >> I see
you've already taken your shot. >> I apologize for that. [Laughing]. It was a long night.
You know, I copied all those Swift On security quotes like late last night. So what could
you actually do if you had control of an ACS? You know, and that's a big if right now
because we don't know how protected this is. But you can actually get private data, you
can get the SSIDs, you can get the host names, Mac addresses, usernames, you can get phone
numbers. With some vendors you can even get complete configurations including
passwords. You can set every parameter you know, starting with the classic DNS change.
Right? We all know that one. You know, but you can go for more sophisticated stuff like adding
your hidden WiFi. >> Shut up! >> I'm shutting up. [Laughter] >> Come on you guys know the drill.
How about a little love for the brand new speaker? [Applause]. Scumbag ACS? >>Yes sir. >> What
would an attacker do if... [Laughing]. Maybe I should attend this talk. >> That's
right. Okay so you can add the new hidden SSID, you can add like a new hidden WiFi to that
router and just connect to it whenever you're around or you can even you know, replace the
WAN service with a tightly controlled tunnel. You can download logs, configurations,
firmware from the device, and you can actually you know, go all the way and custom make your
own firmware for all the clients of that. You can do this selectively, you can do this for
everyone. This has crazy abilities. So now, you know, we understand this is a critical
component in modern Internet infrastructure and with this understanding let's go out and
research TR-069. So a couple things just with cropping up Google when you look it up. And
you know, there's Luka, which is the main developer for CWMP, that's an open source TR-069
implementation. But his talk is mostly about interoperability issues, and much less about
security. And just a nice blog post by these guys, 3S Labs and they actually make some few good
points but they never follow it up with anything afterwards. And that's about it. And I mean when
we were seeing this, we were really surprised. Because you know, this is why we think that
this talk is so important because no one is talking about this yet and this is important.
We think more people should and hopefully you guys will. And so why is that? And the problem is
this is software in a niche market. So this is service provider world. This is out of
most consumer's and researcher's reach. You know, where is the TR-069 community? Right, I mean,
are there like TR-069 parties out there? I don't know if I'm not invited. Maybe I should. I
don't know. I mean you know how there's a subreddit for everything, I mean, nothing for
TR-069, this is-. You know, we were looking for this, you know, a TR-069 media outlet, something
that covers you know, TR-069 happenings. And you know what? Apparently there is one. So this
actually is a website called TR-069 Central. It's a WordPress blog. And you know, don't laugh
about it because you know, they have 16 followers on Twitter. [Laughter] Yeah, let's see if I
can -- yeah they're following 23 people including me. [Laughter] And, you know, of course I'm
kidding because they actually do a decent job of covering TR-069 happenings and they have
independent reviews and such. But, and when I say they I mean a pretty nice guy called Max and
they compiled a list of ACS vendors, some smaller names, some bigger players and, you
know, we started browsing the vendor sites. As you go along you get this certain feeling.
Let me try to show you what I mean. Let's hope this works. >> This is a presentation on SS/SI
for TR-069 auto-configuration service. >> This is great music, by the way. >> Let's begin by
logging into ACSI and enter your username and password. >> Okay so maybe that's enough... So you
know the feeling is much ACS vendors, much TR-069, many features have such a 1999 look
and feel. Right? So you know, the more we read, the more we understand that ACS software,
are basically web applications that are filled with CPE controlled values that go
through lots of processing, and you know, and we all know that clients can be like totally
trusted. Right? So we can look at TR-069 problems in two planes. So we have insecure
configuration and we have insecure implementation. Of course we have the combination
of both. More words of wisdom there. [Laughing]. So how do you find ACS's in the wild? So first
off you can hack a single router. Right? We all know that's kind of easy. And then
you just get the ACS URL, that one. But you can do more interesting stuff like you know,
scanning, you know, ZMap, Masscan, for the win, you know. Scanning 7547 and friends.
There's also UPnP which are a lot of fun actually. And there's public data that you can search
like the Internet census and the DNS census. Of course there's some fun Google dorks if you're
looking for a specific product and of course you can also go for, let me show that for you,
and look for interesting stuff there. But anyway there are vulnerable ACS's online. I
guarantee it. So how is ACS authentication performed? So SSL is recommended. [Laughter] Which
means that a whole bunch of lazy ISPs avoid the trouble and just go with the shared secret which
is HTTP authentication. Right, so either basic whatever. And we've actually done a little
survey, we've collected a few hundred ACS URLS belonging to a few hundred ISPs and would you
like to guess how many of those were using HTTPS or SSL? So yeah so here we go so, 81 percent
were actually not encrypting any of this, you know, super critical provisioning sessions.
And this is like -- I mean, what -- [Laughing]. You know that means, that all it takes is one
freaking you know, NSA quantum packet, you know, single quantum TCP packet to take control of a
provisioning session and give you a new firmware or change any of the other parameters I've
just described. Here's a nice trick. So our interfaces, right, we take the ACS passwords, which
is, you how the ACS is supposed to know your, the real CPE and, but as I mentioned before they
allow you to change the ACS URL, which basically means you can change it to a web server that
you control and you know, even downgrade to basic authentication and just make it
even easier to parse. So yeah, nice Jerry. So let's talk about certificate validation. And the
technical report states that if TLS1.2 is used then the CPE must authenticate the ACS using the
ACS provider certificate. You know, that sounds reasonable. That actually sounds right. But
this is how I imagine this is perceived in most ISP management forums, and I did a little field
test. This is obviously a fake certificate right, it's a self-signed certificate that we
just created for this, for testing purposes. And so you claim to validate SSL
certificates but the fact you accepted my self-signed one determined that that was a lie.
Yeah, so first big hard face palm for the day. [Laughing]. So let's run through a short recap
until this point. So I hope you're convinced that TR-069 is you know, dangerously powerful
and TR-069 servers are high value targets and, you know, we've also seen actual proof
that a lot of implementations just aren't serious enough. And now we're gonna take it to the
next level. So at this point I have to apologize in advance. I wish I could give some more
details on some of the following slides but however we do realize that there are many unpatched
instances out there and we don't want to make it too easy for the bad guys. So our first target
was a Java based open source project, it's called OpenACS, you know, and we started
auditing the code and three days later we get remote code execution. So we have reflection
and file upload and this is like you know, at this point we're like -- and this is exactly what
we expected to find. Right? We expect to find poor code, this is awesome. I mean, let's keep
it up. So we move forward. We pick Genius.acs, which is you know, another open source
project. This one, this is fairly modern. It's under active development. No JS, MongoDB, all
the new hip stuff. And we started editing, auditing and two days later we got a
functional remote code execution exploit. So this one actually is kind of nice. Let me try to show
you that. So this is a regular expression that's filtering out bad characters. And an input
that follows to JavaScript eval but they forgot the G at the end of the regular expression
meaning that you put in one bad character and it just removes that and lets you do whatever.
And the real bad thing is this is running as root. No JS servers, yeah. This is bad. Lets
you dump like AC shadow and stuff. [Laughing]. Yeah, that would be our faces when we saw
it. Moving on, some more ponage. Here's some green text for you. Scanning for IvP4 ACGS and we
detect an instance, a vulnerable instance in Middle Eastern ISP, and when I say Middle Eastern
I'm talking about a major, a very Muslim state and you know, the northbound interface is
exposed which means the internal API, I mean you can actually make calls to internal API,
which means that we own that. And then we are like, you know, face palm. But OP actually
delivers the vulnerability report. The ISP support center was not thrilled with an Israeli
calling about the vulnerable infrastructure. [Laughing]. [Applause]. But we showed them
this, and you know, practically all blanked out. It's a few thousands devices. Eight out of
10 would report again. So yeah, I mean what if you're leaving a critical Internet internal asset
listening to the entire world with no protection whatsoever? The next case is kind of a big
one. We can't disclose the vendor at this point in time but he has a massive install base,
locally, globally, and our findings include the internal API authentication bypass code
injections there. And denial service attack. The combination of which allows writing
arbitrary files to any location including C slash I net to ASB enabled Internet server. So you
know, remote execution, right? And we went ahead and contacted one large provider to test this
with permission. And eventually we ended up owning just over half a million devices and this
is a major ISP. And this is just an example provider. We just picked one. And you know this
has been unpatched for at least two years now so it is very possible that we're not the
first to discover that flaw but we don't really care. Appreciate it. Appreciate it. [Applause].
So you must be asking yourselves what can I do? What can I actually do? If you don't trust
your gateway there really aren't many things that you can do. But, you know, adding another
security -- well first of all you can hide under the couch and never connect to the Internet,
but that's obviously not a good solution. So the first thing you can do is you know, audit your
own TR-069 settings. You know, actually ensure that you know, SSL is turned on, that there is
proper certificate validation. You know, if you're unsatisfied you can disable it, if you can
disable it. And, you know, adding another network security layer is not bad. You know you
can add another router with NAT or firewall capabilities. I mean we sell these devices but
actually, quite frankly you can use open source firmware alternatives and they'll do just
great work. And you know, ask your provider about the TR-069 configuration. That's, I mean,
it's their responsibility. So you can't fix the problem, so there is no easy fix because the
bad implementations are out there. So TR-069 has to mature more. Awareness is key. You
know, this is what we're here to do. TR-069 can be secure and this is up to the security
community which is you guys. We're hoping to get as many you know, new research blogs out
there as we can find. And ACS vendors they have to start you know, writing better software.
Put your money in secure coding. I mean, show your security stance. I've never seen a bug
bounty for TR-069 software and I think we should. Service providers, I mean you're the
owners of the traffic and you have to protect your customers. That's your responsibility. So
for some future directions. We're actually currently looking to extend this research. So you
remember how 7547 was the second most popular port in the world? And his is by design. So we're
looking into some client ponage, stay tuned for CCC so this opens up an interesting exploitation
vector. Right. Thank you very much. [Applause].