Efficient Mixture-of-Experts Model Inference at the Edge via Adaptive Expert Merging

This paper studies the Mixture-of-Experts (MoE) model inference problem at the edge via the adaptive expert merging technique. To fill the gap between the large size of MoE models and the limited hardware resources at the edge, existing works primarily focus on static merging policies and overlook the dynamic and heterogeneous nature of edge environments. This paper addresses these gaps by proposing a flexible framework for adaptive expert merging and online inference task offloading on edge servers. Our contributions include (1) a novel approach allowing each edge server to adaptively merge experts based on its resources and task preferences, (2) an online inference task offloading algorithm with a bounded competitive ratio for pre-determined merging policies, and (3) an online algorithm for joint optimization of task offloading and expert merging, which ensures timely model updates and assignment of tasks to specialized servers. Experimental results on five datasets demonstrate that our approach reduces the overall weighted cost by at least 16% compared to baseline methods.