KubeEdge: Design and implementation of the next-generation cloud native edge device management standard DMI

Guest post by Zhao Ran (Huawei Cloud), Wang Zilong (DaoCloud)

With the maturity of technologies such as 5G, AI, and distributed cloud, concepts such as Internet of Everything (IoE), digital twin, and ubiquitous compute are extending, bringing innovations in many industry services. More and more applications run on devices and generate a large amount of data. How can we better decouple application development from device data access, provide complete lifecycle data management for devices, and unleash the value of device data? How can we efficiently manage and transmit device data while ensuring cluster availability to obtain more convenient and flexible data access modes? Cloud native edge computing can help you better cope with such problems.

1. KubeEdge Device Management Framework

The design and implementation of the KubeEdge device management architecture effectively help you handle issues encountered in device digital twins. With KubeEdge, you can abstract physical devices into digital twins and manage devices and data in cloud native ways.

Figure 1 shows the KubeEdge device management architecture design. The specific process is as follows:

1. You call the Kubernetes API to create a Device CRD on KubeEdge.
2. CloudCore, a cloud component of KubeEdge, watches Device CRD creation messages in Kubernetes.
3. CloudCore delivers the device twin information to EdgeCore through the cloud-edge websocket stream. In addition, CloudCore generates a ConfigMap containing the device profile information. The ConfigMap is mounted to the pod of the corresponding Mapper based on the node name.
4. The Mapper reads the device profile information in the mounted ConfigMap and updates the locally maintained device list.
5. EdgeCore sends the received device twin information to the specified MQTT topic.
6. All Mappers on the node receive the device twin info and check whether the device is in the list maintained by themselves based on the device name.
7. The Mapper connects to the device through the corresponding protocol based on the device profile information.
8. The Mapper reports the device status and collected data to EdgeCore through MQTT topics.
9. EdgeCore reports the device twin data to CloudCore through the cloud-edge WebSocket stream.
10. CloudCore updates the device twin data to Kubernetes.

2. DMI Framework Design

On this basis, the KubeEdge team also continuously upgrades and iterates the framework. To help users cope with larger-scale device deployment, higher availability requirements, more flexible function support, and better user experience, KubeEdge designs a more optimized device management framework, that is, DMI.

DMI integrates device management APIs, optimizes device management capabilities in edge computing scenarios, and builds a device digital twin management platform based on cloud native technologies, covering device management and device data. In addition, a unified connection entry is defined between EdgeCore and Mappers. EdgeCore and Mappers implement the server and client of upstream and downstream data flows and bear specific DMI functions.

In the DMI framework design, the device management plane data is decoupled from the device service plane data. In this way, device CRDs are only used for the lifecycle management of the device itself, and the device service plane data is directly provided for data-consuming applications through microservices. In this architecture, devices are no longer pure data sources. Instead, they are cloud native device microservices. Developers of device data consuming applications do not need to care about how to obtain device data. Instead, they focus on the service logic development of applications in a more cloud native manner. The DMI framework also provides multiple data push modes, enabling data consumers to obtain device data more flexibly and providing better user experience.

As in DMI, the device management plane is decoupled from the service plane, the service plane data can be more flexibly processed on the cloud or edge through the service plane channel. However, only a small amount of management plane information is transmitted through the cloud-edge channel of the management plane, greatly reducing the possibility of cloud-edge channel congestion, and the availability of the KubeEdge system is improved. In addition, DMI provides unified device management APIs. Both device application developers and users can carry out device-related work in a more unified, flexible, and standardized manner as long as the DMI APIs can be implemented. In this way, you can enjoy the cloud native device management powered by KubeEdge.

2.1 DMI Positioning

Figure 2 shows the positioning of DMI in the KubeEdge architecture. Similar to Kubernetes CNI, CSI, and CRI interfaces, DMI defines a group of internal APIs between EdgeCore and Mappers and unified APIs for external applications to access Mappers. The bottom layer of internal APIs is implemented by gRPC and UDS. External APIs support MQTT and REST access. Regardless of the bearer and implementation mode, Mappers can access KubeEdge and manage devices in cloud native mode as long as the uplink and downlink data APIs defined in DMI are implemented.

2.2 DMI Device and Data Management

Figure 3 shows the DMI framework architecture design. The yellow lines indicate data flow management on the device management plane, and the blue lines indicate data flow management on the service plane.

In this design, the management plane data is decoupled from the service plane data. The management plane data includes device metadata, device attributes, configurations, status, and lifecycles. The management plane data is relatively stable. After a device is created, information except statuses is seldom updated, which is closer to the data generated by pod resources, this ensures that users can use Kubernetes APIs on the cloud to maintain the lifecycle of devices in the same way as accessing pods, minimizing extra data transmission overheads caused by device management.

Under the DMI framework design, devices are no longer pure data sources. Instead, they are abstracted as microservices to provide data services for device data consumers in cloud native mode. Device data access in the DMI framework supports multiple scenarios, which is more flexible. Figure 3 shows several main data access manners, including data push and pull. Details are as follows:

1. Edge applications access device data through the REST service.
2. Applications on the cloud access device data through the REST service.
3. The Mapper pushes data to edge applications by configuring REST destination addresses.
4. The Mapper pushes data to cloud applications by configuring the REST destination address.
5. The Mapper pushes data to the edge database by configuring the destination address.
6. The Mapper pushes data to the MQTT broker by configuring the destination address.
7. Edge applications subscribe to device data through MQTT broker topics.
8. Cloud applications subscribe to device data through MQTT broker topics.
9. Edge applications process the data and upload the processing result to the cloud.

2.3 DMI Workflow

In the DMI framework, the device management process changes. As shown in Figure 4, when KubeEdge is installed, CloudCore registers the DeviceController component to listen to the CRDs of the device and device model. The DeviceController has two modules: Downstream Controller and Upstream Controller. The Downstream Controller listens to device and DeviceModel events on the cloud and delivers the events to the edge through CloudHub. The Upstream Controller receives the device status and messages reported by EdgeHub forwarded by CloudHub, and updates the device status in Kubernetes. On the edge side, during Mapper initialization, the Mapper registration API in the DMI needs to be called to register Mapper information with DeviceManager and receive the device information that has been delivered to the node and matches the protocol returned by the API. When receiving a device message from the cloud, EdgeHub forwards the message to DeviceManager. DeviceManager selects the corresponding Mapper driver based on the device protocol and sends a device creation request. The local database also stores the device information, the Mapper will convert the device twin message into the device protocol format to communicate with the actual physical device.

3. DMI API Definition

3.1 DMI API Classification

The DMI APIs implement communication between EdgeCore and Mappers. It supports REST and MQTT communications and is presented in a standardized manner, simplifying Mapper development and adaptation. In terms of data access, DMI enables applications on both the edge and cloud sides to access device data via REST services.

As shown in Figure 5, the DMI has six types of APIs. Mapper management is used for various device protocol drivers on the edge side. Device management and device data management split data on the management plane and service plane. Device upgrade management and device command management provide APIs for devices with upgrade and command execution functions. Device event management can monitor the running status of Mappers and each device managed by Mappers.

3.2 DMI API Definition Example

Figure 6 shows the definition of the DMI device management API. The v1 version is defined in gRPC proto mode. You can run the make dmi command to create the corresponding gRPC-go code.

3.3 CRD Definition Related to DMI Devices

Figure 7 shows the CRD definition related to DMI devices, which consists of Device and DeviceModel. There is a one-to-one mapping between DeviceModel and the actual device model, indicating the common attributes of the same type of device models. The common attributes include Properties (data attribute generated by the device) and Commands (command attribute supported by the device). A Device corresponds to a physical device. Each DeviceModel can correspond to multiple Devices of the same model. Devices include information about device model mapping, device protocol configuration, device deployment node, device status, and device data property collection configuration.

4. Release Plans

The DMI release plan is divided into three versions. The Alpha version provides device management functions and a Mapper Demo that supports DMI APIs. The Beta version supports device command management, device upgrade management, and device data management. In addition, this version connects to third-party platforms and provides related interconnection demos. The GA version will support interconnection with multiple platforms and protocols. In addition, the device security and event management functions will be supplemented.

Currently, KubeEdge Device IoT SIG focuses on device management and Mapper Demo development in the first phase. You are welcome to contact us through the following contact information to participate in solution design and feature development.

For more about KubeEdge, visit the following links:

Website: https://kubeedge.io

GitHub: https://github.com/kubeedge/kubeedgeSlack

Address: https://kubeedge.slack.com

Email list: https://groups.google.com/forum/#!forum/kubeedge

Weekly community meeting: https://zoom.us/j/4167237304

Twitter:  https://twitter.com/KubeEdge

Documentation: https://docs.kubeedge.io/en/latest/