Probing Diametric Coordination Graphs for Multi-Agent Reinforcement Learning

Effective coordination is the cornerstone of success in cooperative Multi-Agent Reinforcement Learning (MARL), where agents work together to achieve a collective goal. However, most existing methods for modeling agent coordination rely primarily on proximity between agents’ features or on incorporating heterogeneity at the behavioral/policy level, while overlooking the valuable diversity present at the observation level. In this paper, we introduce Diametric Coordination Graphs (DiaCoG), a novel framework that dynamically models implicit coordination of agents by integrating both consistency (shared similarities) and discrepancy (unique differences) from agents’ observations, offering a richer understanding of inter-agent relationships. Accordingly, we propose two approaches, DiaCoG-DE and DiaCoG-CE, implementing this framework under two MARL architectures of actor-critic networks, Centralized Training and Decentralized Execution (CTDE) and Centralized Training and Centralized Execution (CTCE), respectively. Through theoretical analyses based on information-theoretic measures, we show that DiaCoG offers superior expressiveness over consistency-based methods for value estimation and action selection. Empirically, we evaluate the proposed methods in the Predator-Prey and Traffic Junction environments, where they outperform baselines in terms of both final average returns/success rates and convergence speed across diverse scenarios. To further showcase the adaptability of DiaCoG across different MARL paradigms, we integrate it with a state-of-the-art value-based approach and compare it against several representative and state-of-the-art methods to demonstrate the enhanced coordination performance by DiaCoG in the StarCraft II Multi-Agent Challenge environment.