Innocent Mistakes, Huge Security Holes – That’s the Story for IoT and Edge Until Something Changes…

” Rotate datacenter credentials every few minutes or hours. Repave every server and application in the datacenter every few hours from a known good state. Repair vulnerable operating systems and application stacks consistently within hours of patch availability. Faster is safer. ”  — Justin Smith

A story went out a few days ago about a security flaw in an enterprise networking system – it had a hardcoded password in the device that was used as part of remote installation and maintenance of software for certain lines of voice

and video products. Its a relatively quiet notification in an (unfortunately) noisy world of security updates. But I bring it up because it points to a significant problem that, while easy to address in a datacenter of equipment, becomes a major problem for IoT and edge computing.

For enterprise organizations, you’d first have to get through significant perimeter security of the datacenter to take advantage of this exploit. In fact, its arguable that physical access to the box (i.e. sneaking into the datacenter) might be easier than finding a way to remotely log in.

But what if the perimeter were gone? If the compute doing the serving function were actually an industrial computer sitting in a field running a solar panel farm running a real-time app? “Perimeter security” from a networking standpoint is gone and even physically accessing the box itself is infinitely simpler. This is edge computing dissolving the concept of “perimeter” by moving server functions out of the secure datacenter to perform real-time functions.

What is a hardcoded password and how does it get there? Well, hardcoded password basically means that the product developer needed a way for you to login to the box when you first buy it and bring it home. A default password is programmed into the firmware in the box so that you can perform the initial setup. How do I know?

In the late 90s when a current large purveyor of home networking equipment was just a wee-start up, they hired their first product manager on embedded computing – young, bright-eyed, devilishly handsome product manager who was in charge of launching their first ever “wireless home gateway”… while my wife may not recognize the description, I’m talking about me…

A legend in his own mind

So I had to come up with a way to have a user login, set the initial configuration for their 56Kbps Modem Router (aka Home Gateway – a term we originated to compete with 3Com at the time) or first generation wireless router. I also needed to have a way for Manufacturing QA teams to setup “burn-in”, i.e. where they tested the product before it shipped to make sure there were no DOA units. There was no “always-on” Internet or ability to “call home”. So we created a “default” password for the admin. In fact, I chose it. Then, in GIANT letters, I wrote a manual that said “CAUTION: Change this password or anyone can login to your home”. Done and done. We shipped, be beat 3Com in the retail channel, and now its a $3B company… Yup, all me.

But here’s the disturbing part. Literally for YEARS after I stopped working there I could go to folks homes, see one of the default SSIDs, and login in with the very same ADMIN username and password I had set as the original PM of the product! RTFM is real and its epidemic. So that is how these “hardcoded passwords” come about.

This brings me to IoT, edge computing, and the need to rethink embedded computing deployment, updating, and security. And it needs to happen NOW before these “things” are shipped in the billions. Username and password combinations are rudimentary form of identity and credentials. Clearly not secure since best case you can write it on a Post-It and someone can steal it. Worst case, you’re the PM of a mass produced embedded product and you create a default that the factory uses to do burn-in testing. The Internet of Things and the edge infrastructure that runs them needs to start thinking about its security in terms of credentials and identity not just in terms of user names/passwords.

This is where “cloud native” comes in to solve a MAJOR problem with embedded computing and the manual, labor intensive model of “burn-in” and remote login to test configurations. The cloud native view of the world promotes a simple (for cloud native platforms) principle that embedded never had the ability to think of and is summarized in this article:

” Rotate datacenter credentials every few minutes or hours. Repave every server and application in the datacenter every few hours from a known good state. Repair vulnerable operating systems and application stacks consistently within hours of patch availability. Faster is safer. “

Now for the “cloud native edge”, you can replace “datacenter” with “edge system” and you can already start to see the realities of running a secure edge environment on traditional embedded computing. And you have to have a software platform that can do this right out of the box. If “Rotate” is too complicated and is skipped, the other 2 “Rs” won’t cut it! As platforms for deploying, maintaining, and securing edge systems evolve, providing the three Rs of cloud native is priority one.

How to “Repair” and “Repave” every few hours is a whole topic unto itself. But starting at avoiding the dreaded “hardcoded password” and be able to “Rotate” that credential is not difficult in a cloud native edge. It starts with a system root of trust being at the hardware level (via TPMs or similar technology). For embedded systems this part is pretty familiar as secure mobile phones start here. How to “Rotate” credentials can come in various forms leveraging the root of trust at the hardware level. The most recent discussions around centralized control of credentials like CredHub – a proposal by Pivotal to the open source community could serve as a model to facilitate “Rotate”. This would allow credentials to be securely changed periodically very easily without any significant increase in operational costs of a highly distributed edge system (example – a network of ATMs from Bank of America). And it would be faster and hands-free. And remember “faster is safer”.

My example above is for a consumer router so its not an exact apples to apples comparison to a network of ATMs…except that it is!! The only difference is the bank PROBABLY remembers to change the default passwords before deploying the ATMs’ embedded computers.