Best practices for deploying software on large fleets of Linux-based IoT devices

The Internet of Things (IoT) has transformed the way we interact with the world, connecting a myriad of devices to the internet, from smart thermostats in our homes to industrial sensors in manufacturing plants. A significant portion of these IoT devices relies on the Linux operating system due to its flexibility, robustness, and open-source nature.

Deploying software to Linux-based devices, at scale, is a complex and critical process that requires planning, well thought out processes, and adherence to best practices to ensure the stability, security, and manageability of the IoT fleet. In this article, we’ll explore some best practices for deploying software on large fleets of Linux-based IoT devices.

Initial deployment

For your initial deployments, consider using a containerized deployment approach. This involves packaging the software and its dependencies into a container image which can then neatly be deployed to IoT devices. Docker containers have quickly become a popular choice for deploying IoT applications as they provide many benefits, including:

• Isolation: Application isolation within a container helps to improve security and reliability
• Portability: Docker containers can be deployed to any platform that supports Docker and makes it easy to deploy IoT applications to a variety of devices
• Reproducibility: Docker containers are reproducible, meaning that they can be created and deployed consistently across different environments
• Efficient: Docker containers are very resource-efficient, making them ideal for IoT devices that have limited resources
• Mature: Docker as a technology, is mature, understood, and well-supported

Use a centralized management platform

A centralized management platform can be used to automate the process of monitoring and managing your fleet of IoT devices. Having a centralized dashboard that offers a broad overview of the fleet’s health, as well as the ability to get details, can save a significant amount of time and effort while reducing the risk of errors.

When working with hundreds or thousands of devices, you’ll want the aforementioned installed images on the IoT devices to automatically “call home” to the management platform, and self-register with the platform in a secure manner.

There are a number of different centralized management platforms available, both commercial and open source. When choosing a platform, it’s important to consider the specific needs of your organization. Can the platform support the version of Linux that you are using? Can it support the types of devices that you’ve selected? Does it provide the level of security you require? Will it integrate well with your existing and future development workflow?

Segmenting, grouping, and subgrouping deployments

When managing large fleets, the ability to create hierarchical structures for grouping and subgrouping is fundamental for scalable administration. Grouping allows for the organization of devices based on shared characteristics, which facilitates efficient monitoring, configuration, and software updates. Grouping could be organized by location, device type, operating system, version of application, etc. Being able to view specific groups from a central dashboard will make scaling much more manageable.

Automate software updates

Sometime after the initial deployment, you’ll inevitably need to update the software on the deployed devices. Again, automation is essential. You could write a lot of scripts to do this, but they’ll still have to automatically find the devices to update, securely connect them, and push updates to these devices – which hopefully update successfully. Before going down this path, consider the complexity of the task, the risk of bugs, the time and resources needed, and the potential lack of support when relying on the expertise of the developers of the scripts.

The better approach is to use an IoT management platform to automate this. There are a number of commercial and open-source IoT management platforms that can automate the software update process for you. These platforms typically offer a number of features that make them more reliable and efficient than custom scripts, such as:

• Device registration and discovery: IoT management platforms can automatically register your devices, eliminating the need to manually track and update the inventory of your IoT devices. For IoT devices located on other public or private networks and firewalls, a secure platform is needed to establish and manage connections with these devices, even when their IP addresses are concealed and protected by firewalls.
• Software update scheduling and orchestration: IoT management platforms can automatically schedule and orchestrate software updates for your IoT devices. This ensures that updates are deployed in a controlled and efficient manner.
• Over-the-air (OTA) updates: This may seem obvious, but is worth emphasizing. OTA secure updates eliminates the need to physically access the devices to deploy updates. This means your team can deploy updates to any device from anywhere.
• Segmentation and Versioning: With large fleets of IoT devices, the types and configurations of devices can vary widely, with different utilities, versions, dependencies, and even functionality. The management platform must be able to handle different updates for different segments of devices, even though there may be thousands of devices in the fleet, and the updates should be generic enough to allow different types and configurations.
• Security: IoT management platforms typically offer a number of security features to protect your IoT devices during the software update process. An encrypted connection and secure token exchange to verify their authenticity is a good start. But software updates should go beyond encryption to check the software’s compliance and vulnerabilities as well.

If you are serious about managing IoT software updates, I recommend using a proven IoT management platform. These platforms can save you time, money, and headaches in the long run.

Have a rollback plan

IoT devices can be challenging to update due to their remote locations and inconsistent network connections. That’s why it’s important to have a rollback plan in place in case something goes wrong with deployment updates. A rollback plan should quickly restore the devices to their previous state. If you don’t have a rollback plan, and a network interruption stops the update, you could wind up with a lot of devices that no longer work, requiring costly field visits. In the event of a failed update, a properly deployed device should automatically rollback to its previous working state.

You could maintain a rollback image and try to republish that image to the devices. However, that option feels quite manual and tedious, even with scripting, and it assumes you can gain access to the failed device. Some IoT management platforms deploy a smart client to help manage secure connections to the device, including the ability to restore a failed update, back to its previous state.

DevOps for IoT

IoT development may be so nascent that it may not yet be part of your mainstream DevOps processes – you may still be in the early stages of experimentation. Once you’re ready to scale, you’ll need to bring IoT into the DevOps fold. When dealing with‌ thousands of deployed devices, the scale and costs are now significant. DevOps is an important approach for ensuring the seamless and efficient delivery of software development, updates, and enhancements to IoT devices. By integrating IoT development into an established workflow, you’ll gain the improved collaboration, agility, assured delivery, control, and traceability that’s part of a modern DevOps process.

Secure deployments

It’s critical to use a secure deployment process to protect your IoT devices from unauthorized access, inadvertent vulnerabilities, and malware. A secure deployment must include strong authentication methods to access the devices and the management platform. The data that is transmitted between the devices and the management platform should be protected by encryption. The manner in which the client devices connect to the platform after deployment should always be encrypted as well.

To ensure that an IoT device is valid and that the management platform it is communicating with is also valid, there should be an exchange of tokens to validate the client and platform. In other words, the device and the platform should both generate and exchange tokens that are unique and difficult to forge. These tokens can then be used to verify the identity of the device and the platform.

Beyond encryption, you must consider the security of the software itself. Is the software free from vulnerabilities? Has it been scanned for potential vulnerabilities before deployments? Was this done throughout the development process? If you’re leveraging open source software, there are vulnerability databases such as cve.org and vulndb that offer information on specific software packages.

In addition to the software itself, consider checking for potential deployment misconfigurations that could make the devices vulnerable to attacks. To automate this process, consider using a software composition analysis (SCA) tool to scan for vulnerabilities, and perhaps a static application security testing (SAST) tool that can help developers find weaknesses in their code well ahead of deployment. While these tools are helpful, they can sometimes overwhelm developers with false positives. To reduce false alerts – and therefore wasted time and effort – find a modern tool that can take the context of the application’s use-case of the software into consideration.

Security scans should be done on an ongoing basis throughout the software development lifecycle — from coding to deployment. Building automated security processes into your DevOps processes will go a long way towards secure deployments. The merging of DevOps and SecOps is more commonly referred to as DevSecOps, and should be considered standard practice in today’s software development.

Monitor the deployment and automated alerts

Monitoring large IoT fleets is essential for ensuring their performance, security, and reliability. By continuously collecting and analyzing data from IoT devices, organizations can gain insights into their usage patterns, identify potential problems, and take corrective action. Given the scale, the ability to automate the monitoring and alerting process should be a requirement. Is a device offline? Are preconfigured device thresholds for CPU, disk, or memory usage being exceeded? Is a particular process being monitored still active?  By automating the monitoring and alerts of all devices, you can get ahead of problems before they escalate.

Remote access

Field work is expensive. The personnel costs, time, and physical travel should be avoided whenever possible. That’s why remote access to devices is key. In the event of a device malfunction that requires manual intervention, remote terminal access can save the day as the developer can seamlessly access the device as if it were on their desk. The most common method is using SSH (Secure Shell). SSH is a secure protocol that allows you to connect to a remote computer and run scripts and commands directly on the device. You’ll need to ensure that the device is configured accordingly to support this.

When devices are deployed behind firewalls, on private networks, you won’t know the device’s IP address or be able to get past the firewall with standard SSH. The common method is to use reverse SSH tunneling, also known as SSH port forwarding; a technique that allows you to connect to a remote host from a local host, even if the remote host is behind a firewall. It works by creating an SSH tunnel initiated from the device to your external machine. This tunnel allows you to access the device as if it were on the same network as your machine.

An even better option is managing your fleet with a robust IoT management platform that has a proven record of developing and deploying applications with enterprises. This platform deploys a lightweight, smart agent application for each device you wish to oversee and manage. The agent can handle the central IoT platform’s connectivity, security, monitoring, alerts, and the essential reverse SSH information required for secure remote access to the device. The central dashboard would then provide a holistic view of the entire fleet, delivering the visibility and framework necessary to accommodate your evolving scale and requirements.

Summary

As a developer, you may be tempted to build your own custom platform and smart-agent to manage your IoT fleet. This requires time, expertise, and a significant investment of trust in a few internal experts. To update, control, and manage large and mission-critical IoT fleets, it’s more practical to partner with a dedicated, established vendor with a proven track record. This will free up your development team to focus on building great IoT apps instead of having to build and maintain the infrastructure to manage them.