Software updates for IoT edge Linux devices
To successfully deploy an Over-the-Air software updates mechanism on our Internet of Things Linux based devices (Raspberry Pi, BeagleBone Black or industrial custom system on module board running Yocto based Linux distro), we must first understand the differences between such devices and any other device that capable of receiving updates remotely. Therefore, let’s start the article by explaining those differences, continue with the acceptable approaches regarding the IoT domain and finish by providing some tips and techniques on how to successfully implement an OTA updates mechanism for your next IoT project.
The difference between OTA updates on edge Linux devices
OTA updates technologies exist in any industry that involves internet-connected devices, certainly not only the IoT one. Any smartphone receives OTA updates, every server worldwide also supported remotely and maintained through OTA mechanisms and plenty of others. Each and every one of those products are working in the same and specific way, thus the method of sending an OTA update for a particular server is exactly the same as for any other server on earth. Similarly, the method of sending an OTA update for every mobile application is the same.
As opposed to this, when examining the IoT sector, the situation is different. Every device is behaving and working in unique architecture, hence the mechanism for sending OTA updates must be “smart” and generic enough to deal with whatsoever situation. The next few questions may provide a taste of the main differences IoT products have:
Does the product intend to be utilized by an end-user or to work autonomously? Does the product operate 24/7? Does the product’s power source is a battery? Does the product has a stable internet connection? Is it connected to a cellular provider or a cable one?
The answers to these questions influence how the OTA updates mechanism will be implemented for a particular IoT product.
We must keep in mind that opposed to servers or mobile applications, where if an update fails there is always somebody that can react to fix the issues by restarting the device or even calling a support center, in many situations at the IoT industry this is not the case. There is nobody that can even get close to the device in the field and the potential damage is much higher. Also, a downtime of IoT devices sometimes may cause true harm, for example, a factory that is paralyzed as a result of a stuck assembly line due to a failure of a software update.
Software updates approach for IoT and connected Linux devices
Despite the risks of sending OTA software updates, it is obvious to every IoT manufacturer that there is no option to retail a smart product these days without appropriate OTA updates mechanism. There are 2 main approaches for such a mechanism:
The old method — Before the connection to the internet, the standard way of updating embedded products was by reaching every device physically, connecting it with a cable to a computer, and burning on its memory the new software. Since this procedure is pretty complicated and costs a lot of money, the number of updates that were deployed in practice was extremely low. Actually, most of the products would never receive even one software update throughout their lifetime.
As soon as those products were connected to the internet, naturally, many corporates implemented an OTA mechanism based on the method they already were familiar with. Meaning, each update, they send the whole operating system(OS) and replacing the existing one with the new. The only advantage of this implementation is that it has no surprises. Whatever exists inside the new OS is what will run on the device(as long as the update succeeded, obviously). However, there are plenty of disadvantages to this method:
The size of the update — since we send the whole OS, each update weights between hundred of megabytes to multiple gigs. This fact influences also on the deployment time but more importantly, it uses a huge amount of internet data plan. As we all know, countless IoT devices are connected to a cellular network with a SIM card and have quite low data plans. Therefore, sending each update the whole OS just makes no sense. In case something goes wrong with the update, the product probably won’t power on anymore, this situation is called a “brick” device. As a result of this, there are always concerning feelings when speaking of sending OTA updates. The update requires a reboot. Rebooting a device causes unwanted downtime which in various circumstances cost an enormous amount of money(such as with the factory example I presented in the past paragraph).
In general, whoever uses this method of sending OTA updates, the actual number of updates he sends is minor.
The new method — Sending update only of the part inside the OS we are interested to change. For example, sending only the updated binary of our application or sending an update to upgrade a specific package or module which our application uses. As opposed to the old method, this one has only one disadvantage. If not managing the updates in an organized and clear way, you may very quickly get into a situation where you don’t know which version deployed on which device. Though, there are several advantages when using this method:
In most cases, the update size will be extremely small. Since we are sending only the updated part of the OS, the whole update will be tiny. This fact allows continuous software deployment for devices with limited internet data plans. The update doesn’t require a reboot. In certain conditions, there is no need to even stop the running application while deploying such an update. This, of course, prevents unwanted downtime and users’ inconvenience. If something goes wrong with the update, in 99% of the cases, the device won’t be bricked. Thus, a smart OTA updates mechanism will recognize an update failure and revert to the previous software version. Security — Sending continuously software updates enables us to keep the product secured and drastically reduces the chances of potential breaches like happened with the famous Mirai bot.
Implementing OTA updates mechanism
For those who choose to implement the old method, note that there is a need for low-level programming while modifying components such as the bootloader and the Linux kernel itself. Although there is an open-source project named swupdate which provides a partial solution for this matter. Notice that even when using swupdate, you still have to integrate it and develop specific features to address your needs.
For those who choose to implement the new method, the most used one, there are 2 options:
1. Developing an OTA updates mechanism in-house. If you choose to develop such a mechanism by yourself, take into consideration the next points: You must develop both the client-side as well as the server-side. The server side will run on a could server(i.e AWS, Azure, Google Cloud, etc) while the client-side will run on your devices. The communication between them depends on multiple factors but mostly on the next point. Most of the IoT devices are working behind a NAT, basically, it means that they don’t own a global IP address on the world wide web, therefore no one can contact them directly. The only way your server can communicate with them is when they are the first to start the conversation. As a result of this, you would have to implement a mechanism in which only the client sends the first packet to the server and not vice versa. Rollback system — Following the points I have previously mentioned, an OTA update deployment on IoT devices has good chances to fail. For this reason, you must implement a smart rollback system that recognizes failures and revert the software to the old and working one. Note that such a system has to deal with many edge cases. For instance, if the update stopped in the middle of a package upgrading, the rollback system should check whether the old version of the package is still installed and working properly, and if not, reinstall it.
2. Using a ready-to-use commercial platform. Such platforms provide a fast yet stable and secure solution which most of the time includes all of the functionalities I have discussed above. JFrog Connect is an example of such a platform. JFrog Connect’s platform provides a management dashboard for IoT and connected devices. Amongst the tools we offer, we can find the OTA software updates system that supports sending fast and lightweight software updates remotely(using the new method) with a smart rollback engine that automatically identifies failures and reverts to the previous software version. We also provide tools to help you manage and support your IoT products fleet by giving you the option to connect remotely with a secured connection to any of your products in order to provide technical support. The registration to JFrog Connect is completely free and even includes an example device to let you play with the platform. Another great advantage of JFrog Connect is the super simple and fast integration with any Linux based device which takes no more than 1 minute and possible to be done also with already deployed products in the field.
