Enhancing Collaborative Inference on Heterogeneous Edge Devices via Adaptive Ensemble Knowledge Distillation

The integration of edge computing with deep neural networks (DNNs) is crucial for intelligent industrial cyber-physical systems. Typically, deploying DNNs on heterogeneous edge devices relies on methods like model compression and partitioning. However, these approaches often result in homogeneous models across devices. This homogeneity limits the collective capability of edge computing systems, particularly in terms of generalization to diverse data distributions and adaptation to dynamic industrial environments. In this work, we propose to treat each DNN on an edge device as an independent model, aggregating their capabilities via ensemble learning to enhance generalization and dynamic adaptability. To realize this, we introduce the Adaptive Ensemble Knowledge Distillation Framework (AEKDF), combining cloud-based model training with edge computing based collaborative inference. In the cloud, AEKDF develops an enhanced Born Again Network that generates diverse, lightweight models tailored to specific edge devices through knowledge distillation. This process ensures model diversity which is critical to effective ensemble learning. On the edge, AEKDF employs an adaptive ensemble technique that aggregates prediction logits across devices, enabling rapid adaptation to changing environments and maintaining inference efficiency. Our extensive evaluations conducted on a realistic prototype demonstrate the substantial boost in predictive performance achieved by our AEKDF, showcasing a 4% to 10% accuracy improvement on the CIFAR-100 compared to conventional single-model approaches, while maintaining low latency.