Keeping that software up to date: how hard could it be?

Think you can keep everything up to date with the latest security fixes? Think again.

One common piece of advice you will hear from cybersecurity experts is that you should always keep your software up-to-date, so that vulnerabilities can be corrected. We like to believe that consumers are the biggest offenders in terms of keeping old software around. After all, Grandpa doesn’t always know how to upgrade his iPhone.

This isn’t so easy for professionals. Let’s take a web site. Your average web site is composed of numerous servers, each running all manner of code, from the operating system to standard applications like Apache to web support code such as Python, PHP, and back end services, like MongoDB. ALL of it has to be updated.

What does the inventory look like? This has been the work of a group of people who have created something known as software bills of materials or SBOMs. One could think of an SBOM as the ingredient list on the back of a can of soup, only there’s software inside.

So how does one generate the inventory? And this is where things can get a bit interesting. On any reasonable UNIX machine there exists a package manager. In the case of Ubuntu, that would be apt, which is built on top of Debian’s dpkg. One can simply type “apt list –installed” to get the installed packages on a system, right?

WRONG.

Of course that will get you software that apt has installed, but if your site runs with python, then you might need to get a software list by using Python’s package manager, pip. In fact, “pip freeze” provides this information, and that will get you most, but likely not all, python packages. Now repeat for node.js’s npm, and others (assuming you can find all of them).

Now you might want to update those packages. That was the problem statement, after all. This should be simple enough, right? Well, let’s start with apt. It generally is simple enough, at least to start with: one simply runs, “apt upgrade -y” and software package upgrades happen. Of course, you have to test all of your code after this, as apt won’t do that for you. Your CI/CD process is hopefully doing this for you.

Next we go to Python. pip’s upgrade command requires a list of packages. Conveniently, we have one that we froze from above. However, it turns out that not all python packages are managed by pip; and it knows this, and pip will happily blow chunks when you try to upgrade a package it does own that requires an upgrade for something it doesn’t own. In the Ubuntu world, there are a few good examples of this, Cairo and MongoDB to name a few. Instead, these packages are managed by apt. Well that’s all good, since we can use the apt upgrade process, right?

WRONG.

Often times these packages are installed have or meet dependencies of their won that might go unmet as part of an upgrade.

Alpine Linux has applied an interesting solution to all of this: provide common python packages in their own ‘apk’ package manager. By using apk, you assuredly will not get the latest version of a package, but you will get consistent upgrade behavior. But this presents its own set of problems:

A publication hierarchy, starting from your source on Github, and then going who knows where..

What you are looking at is a publication hierarchy. Now we have software in multiple distributions, deriving from a repository, probably from different branches. It’s possible that there are even multiple repositories for the same code. If we are lucky, the repo owner is doing the integration to push updates to Ubuntu, Pypi, and Alpine. But we are almost never going to be that lucky, if for no other reason, a lack of authorization for all of them. And so we’ll end up with a mix of push and pull. It’s the pull that causes the version skew, and this is just a single distribution.

Now, I’ve picked on Python in this post, but pip and pypi actually do a pretty reasonable job of managing what packages it installed, so long as there aren’t dependencies to stuff it didn’t install.

So what does this mean in terms of managing your product or package?

I’m not ready to make any sweeping recommendations about what to do. One option is to just deal, and upgrade when you can. The problem is that your velocity will be slower than if you pull all source from their repos directly. On the other hand, doing the latter is a whole lot of work, requiring a whole lot of expertise when builds blow up.

This is a good example of where OSS needs to mature a bit more.

Chicken Noodle Soup by By Willis Lam – Campbell’s Chicken Noodle Soup, CC BY-SA 2.0.

Who has access to that smart home you’re buying?

You got the keys to the house, but someone else may have the keys to all of the systems inside the house, including the door locks.

You’ve just moved into a lovely house. It has these wonderful smart lights, with a wonderful smart oven, fancy smart thermostats, with a smart refrigerator, smart locks, and a smart security system. There’s only one problem: not only do you not have all that fancy access for your apps, but perhaps the old owner still does, and he didn’t leave willingly, something you don’t know. Sounds crazy? We sure have come a long way from just getting the keys and the garage door openers, and one cannot just call a locksmith.

Many – but not all – IoT-enabled devices have some form of factory reset capability. Often, this amounts to inserting a paperclip into a pinhole and holding it for 10 seconds or so, but as we’ll see the procedure varies by device type, and it is not possible for some devices. Your stove is unlikely to have anything to stick a metal object in, for instance, nor will outdoor lights. In these cases, there will generally be some vendor instructions. In the case of Philips Hues, the only available reset option is to reset the bridge that is used to communicate with the lights. If the bridge is fastened to the wall, as demonstrated in the picture, this means removing it first. This, by the way, requires not only that the bridge be re-paired with the lights and with your app, but that all configuration for the lights be re-established.

Yale Assure Lever Lock — Yale Assure Lever

What about smart locks? Clearly one of the highest priorities upon taking possession of a home is to control who can enter. If you are leasing a home, some smart locks have master codes that the landlord will set and maintain. In this case, all is “good” (assuming you don’t mind your landlord having access) unless the landlord loses the code. If you bought your dwelling, or if the landlord did lose the code, what to do? Again, this will vary by vendor. For example, here are the instructions for the Yale Assure Lever (YRD256):

Remove battery cover and batteries.
Remove the interior escutcheon to access the reset button.
Locate the white reset button near the PCB cable connector.
Press and hold the reset button for a minimum of three (3) seconds while simultaneously replacing the batteries.
Once batteries are replaced, release the reset button.
Reassemble the lock.

You might be wondering what an escutcheon is. According to Google, it’s a flat piece of metal for protection and often ornamentation, around a keyhole, door handle, or light switch.

Next, let’s have a look at the oven. Let’s say that you have a Signature Kitchen Suite Double Wall Oven such as the one pictured to the left. According to the instructions, all it says is… follow the app instructions. You better hope there are some, or a service call to SKS will be in order. By the way, one might reasonably ask what could happen if you don’t reset this device? First, the device itself won’t be able to receive security updates, assuming the company issues any to begin with. This means the oven could be vulnerable to attack. If the oven app was used by the previous owner, then the chances are, it has joined and would be looking for the old Wifi network. But we really can’t say, because there’s no clear documentation. This holds true for many “smart” devices.

Oh and then there’s that garage door. Here’s the Genie StealthDrive 750 Plus, featuring what they call Aladdin Connect. Their stated “advantage” is that you can “Control and monitor the status of your garage door from anywhere with your smart device.” Or the previous owner can. Or your ex-husband can. The good news is that garage door manufacturers have been in business for a long time, and understand the need to deal with lost or misplaced remotes. The bad news is that they haven’t been in the Internet security business for very long, and there are indeed no instructions on how to reset Aladdin Connect, other than to unplug it.

Oh dear.

How does one take possession of that house?!

While it is impossible to provide a comprehensive guide about all smart devices, here are here are some guidelines that will help.

First, learn about what IoT devices are in the house prior to entering a contract, or by including full disclosure and assistance as a contingency of sale. Having documentation and a customer support number available will help to assess what effort is required to shift control from the old owner to you. The simplest case may be for the old owner to transfer control to you in whatever application controls the smart appliance. Otherwise, a reset will be required.

You might want to use a simple table along the lines of the following to assist.

System	IoT Enabled?	Manual located?	Known how to reset?	Customer Service contact	Handoff Complete
Smart Locks
Door Bell
Climate Control
Garage Door
Lighting
Oven
Fridge
Sprinkers

Smart device handover checklist

It may not be possible to reset certain devices, as we discussed. In this case, what is important is that you read the documentation and understand when you have received the necessary supervisory access. You should be able to understand who has control and who doesn’t. If there are passwords involved, you should be change them. If there is a list of authorized users, you should be able to view them and disable the ones you don’t know. If you can’t perform these features, it may cost money to correct the situation. You should know about that cost in advance.

Is all of this Smart Stuff worth it?

While it may help to think about what benefit you will gain by having smart appliances in the house, increasingly the choice may no longer be yours, as IoT capabilities diffuse through the industry. If you are moving into a place, you don’t want to have to worry about who has control of the door locks. If you are installing door locks, you may want to think twice about the headaches that may occur when you move out. Whatever you do, keep all manuals! They will be needed later.

I should point out that the vendors I named in this post are not bad vendors, but in all likelihood representative of where the market is today. Few vendors are likely to do better than them.

Is there hope for the future?

Yes. Smart home device capabilities are still evolving. Just like we had universal remote controls for televisions in the 1980s, at least some access control functions are likely to be aggregated into one or two control systems. The reason this is likely is that no manufacturer really ever wants to hear from you, because phone calls have to be answered by people whose salary takes away from their profits. This means that incentives are likely aligned for manufacturers to cooperate on standards to facilitate handover.

It’s Not the Doorbell, It’s the Cloud

Your password in the cloud was weak, not the IoT device this time. But there are emerging IoT standards like DPP that can help do away with passwords.

You have to have been hiding under a rock over the last week not to have heard about scare stories about kids being tormented by perverts and others being violently extorted through various Ring products. Not exactly what you were expecting from your security product, was it?

With so many reports of IoT devices being vulnerable to attack, one might leap to the idea that the Ring device itself has been poorly designed, and thus broken into, but one would be wrong. That is because, like so many IoT devices, Ring products make use of the cloud to offer a service. Here’s how it all works.

When you establish an account, you are doing this not on the doorbell, but on a service somewhere on the Internet to which the doorbell connects. This is evident, because when you go to ring.com, you can log in with the account that you have previously established in the app.

Later during device setup, the doorbell is registered with the service, using the phone’s setup app. This is likely the only time the phone would directly communicate with the doorbell. All other communications flow through the service, as drawn above.

So how did someone else get to control your device? If you are not using two factor authentication, an attacker requires two pieces of information to control your device: your email address and your password. Your email address can easily have appeared in public if you have joined a public mailing list, or had made a comment on a poorly designed web site. An attacker may also be able to guess your password if you have used that same password on a service that has been compromised (hint: many have), or the password itself is obvious.

Some recent research has found that long or complex passwords aren’t good because people write them down or forget them. On the other hand, Ring will accept “12345678” as a password, and quite a number of other commonly used passwords that can be found on this list of stupid passwords. First piece of advice in this article: don’t use those passwords!

Ring also offers the option to register a cell phone with your account, so that when you log in, you will receive a code via SMS that you must enter to access your account. This two factor authentication (or 2FA) is stronger, and well worth the mild inconvenience, given that this is your house and its security we are talking about.

All of this is about securing your online account. The only reason that the EvilBadDoer can bother Little Johnny and take over your doorbell or security camera, at least in this moment, is that EvilBadDoer hacked your online service password to the service controls the device.

Could this marriage of IoT devices and online services be used to provide a stronger authentication? Possibly. Because a device communicates with the cloud once it’s set up, and because your phone communicates with the cloud after the doorbell is setup, it is possible for the device to provide the doorbell a token. However, for that to work, communications must be secured between the device and the doorbell during setup. Earlier this year, researchers found that this was not the case, the reason being that the doorbell was simply using unencrypted HTTP to share information about your wifi network. Bad Ring! No Ring biscuit!

Luckily, there are some onboarding standards that Ring and others could leverage to help improve matters. One is EasyConnect by the Wifi Alliance, otherwise known as Device Provisioning Protocol (DPP). Here’s how DPP works:

With DPP, you can use an app to scan a QR code printed on a label that came with the device that contains the public key that was installed during the manufacturing process. The app then looks for the device and authenticates using that key. Look, Ma! No passwords. DPP was primarily intended to be used for Wifi connectivity, but there’s no reason that the same trust couldn’t be leveraged to do away with Ring passwords. This is something that Amazon and others should consider.

There are some remaining challenges. For instance, what happens if you lose your phone? Can you repeat the exercise, and if you do so, would you have to do so with all the Ring devices in your house? To me this is best handled with some sort of backup before one loses one’s phone.

The key point here is that IoT can actually help itself if we adopt stronger onboarding technologies, like EasyConnect. This will take some time to get right. As a customer, you might want to ask about EasyConnect to help ease password problems so that Little Johnny can sleep easier.

Would you want your cousin using a connected oven?

Recently my cousin installed a smart oven into her home. It is top of the line. She wrote on social media that it texted her to tell her that it needed to clean itself, which it did before her second cup of coffee. How cool is that?

I immediately feared for her safety. Here is a slightly edited version of what I wrote to her:

IoT is a nice convenience, but there are a few things you should know. First, I guarantee that there are vulnerabilities in the device, even if some have yet to discover them. This is true for *any* connected device. Those vulnerabilities may be exploited at some point. What will happen then?
First, it’s possible that attacker could simply disable the oven. They probably won’t do this unless they are able to communicate with you. But since the oven seems to be sending you messages, it’s possible that they will do this and ransom you to re-enable it. (If that happens, don’t pay.)
Whether or not you can control the oven from the app, don’t think for a moment that hackers won’t be able to gain that level of control. That presents a far more serious risk: a fire, especially if the hackers are able to detect that the cooking temp is supposed to be 350, and turn the thing up to broil or clean.
The other thing that will happen is that the oven will attack other Wifi-enabled devices in your house or elsewhere. If you have a Wifi-enabled thermostat, maybe it will attack that. Some of those devices have cameras and microphones. The attackers aren’t going to be nice about what information they collect. They’re out to make money or worse.
Will any of this happen? Yes – to many people. Am I being paranoid? Maybe a little. Appliance manufacturers may know how to make excellent oven mechanisms, refrigerator compressors, stove top elements, etc, but they generally know very little about Internet security and their risks. Even those who know a lot get it wrong all the time, simply because we’re human.
And so are you gaining any great convenience by having the Wifi turned on, apart from a 5:30am wake up call to let you know that it needs to clean itself? If yes, you have a trade off to make. If not, just disable its darn Wifi.

This is how I feel about technology and the ones I love. Presumably you have some of those. There are definitely times when IoT is necessary, and when convenience is probably worth the risk. But consumers really need to think about this long and hard, and we professionals need to provide them a decent decision framework. I’ll talk about that next.

Addressing the Department Gap in IoT Security

People in departments outside of IT aren’t paid to understand IT security. In the world of IoT, we need to make it easy for those people to do the right thing.

So, Mr. IT professional, you suffer from your colleagues at work connecting all sorts of crap to your network that you’ve never heard of? You’re not alone. As more and more devices hit the network, the ability to maintain control can often prove challenging. Here are your choices for dealing with miscreant devices:

Prohibit them and enforce the prohibition by firing anyone who attaches an unauthorized device.
Allow them and suffer.
Prohibit them but not enforce the prohibition.
Provide an onboarding and approval process.

A bunch of companies I work with generally aim for 1 and end up with 3. A bunch of administrators recognize the situation and fit into 2. Everyone I talk to wants to find a way to scale 4, but nobody has, as of yet. What does 4 involve? Today, it means an IT person researching a given device, determining what networking requirements it has, creating firewall rules, and some associated policies, and establishing an approval mechanism for a device to connect.

This problem is exacerbated by the fact that many different enterprise departments have wide and varied needs, and the network stands as critical to many of them. Furthermore, very few of those departments reports through the chief information officer, and chief information security officers often lack the attention their concerns receive.

I would claim that the problem is that incentives are not well aligned, were people in other departments even aware of the IT person’s concerns in the first place, and often they are not. The person responsible for providing vending machines just wants to get the vending machines hooked up, while the person in charge of facilities just wants the lights to come on and the temperature to be correct.

What we know from hard experience is that the best way to address this sort of misalignment is to make it easy for everyone to do the right thing. What, then, is the right thing?

Prerequisites

It has been important pretty much forever for enterprises to be able to maintain an inventory of devices that connect to their networks. This can be tied into the DHCP infrastructure or to the device authentication infrastructure. Many such systems exist, the simplest of which is Active Directory. Some are passive and snoop the network. The key point is simply this: you can’t authorize a system if you can’t remember it. In order to remember it, the device itself needs to have some sort of unique identifier. In the simplest case, this is a MAC address.

Ask device manufacturers to help

Manufacturers need to make your life easier by providing you a description what the device’s communication requirements are. The best way to do this is with Manufacturer Usage Descriptions (MUD). When MUD is used, your network management system can retrieve a recommendation from the manufacturer, and then you can approve, modify, or refuse a policy. By doing this, you don’t have to go searching all over random web sites.

Have a simple and accessible user interface for people to use

Once in place you now have a nice system that encourages the right thing to happen, without other departments having to do anything other than to identify the devices they want to connect. That could be as simple as a picture of a QR code or otherwise entering a serial #. The easier we can make it for people who know nothing about networking, the better all our lives will be.