PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining

Computer Science

• Iteration Time 100%
• Pipelining 100%
• Experimental Result 50%
• Graphics Processing Unit 50%
• Artificial Intelligence 50%
• Optimization Problem 50%
• Performance Model 50%
• Training System 50%

Keyphrases

• Mixture-of-experts 100%
• Iteration Times 25%
• Optimization Problem 12%
• Model Size 12%
• Large Animal Model 12%
• Training Time 12%
• Communication Time 12%
• Communication Tasks 12%
• Multi-GPU 12%
• Performance Model 12%
• Distributed Training 12%
• Training System 12%
• Computational Tasks 12%
• Expert Training 12%
• Mixture of Experts Models 12%
• NLP Model 12%