Learning Communication-Efficient Protocols in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) often depends on communication. When each agent sees only part of the environment, messages can supply missing information and support coordinated action. Yet every transmission consumes bandwidth, adds latency, and may use energy. A protocol can therefore deliver strong task performance while communicating inefficiently.

Task performance and communication cost should be evaluated together. Better coordination does not always require more communication.

This article follows the presentation from motivation to deployment. It first explains why communication belongs inside the learned policy, then introduces a unified description of multi-round protocols. Three complementary metrics reveal what success rate alone misses. Experiments in Traffic Junction then show how efficiency-aware training can improve both coordination and communication efficiency.

01 · Why Efficiency?

Communication is part of the learned policy

In a partially observable environment, no individual agent can see the full global state. Each agent acts from its local observation, while the policies of the other agents continue to change during training. From one learner’s perspective, the environment is both incomplete and non-stationary.

Learned communication extends the information available before action selection. An agent encodes its internal state into a message, receives information from teammates, and updates its representation before acting. Because end-to-end training can adapt message content and communication topology to the task, communication becomes an integral part of the policy rather than a separate external channel.

Three agents receive partial observations of a global state and exchange learned messages before acting — **Why agents communicate.** Messages expand an agent’s effective observation before it selects an action, supporting coordination under partial observability and non-stationarity. Original diagram from the EXTEND presentation.

From capability to efficiency

Learned communication research has progressively added control over when, with whom, and what agents communicate. RIAL and DIAL introduced differentiable message channels. CommNet used continuous broadcast. IC3Net added context-dependent gating, while TarMAC used targeted attention. MAGIC represented communication as a dynamic graph with repeated exchanges. More recent BVME and GACG approaches use bottlenecks or sparse topology.

Timeline of learned communication architectures from RIAL and DIAL through CommNet, IC3Net, TarMAC, MAGIC, BVME, and GACG — **Architecture evolution.** Prior work increasingly controlled message channels, recipients, and topology, while the information and resource cost of communication remained less explicit. Original diagram from the EXTEND presentation.

The dominant question was whether a more capable communication mechanism could improve task performance. Communication cost received less direct attention. A second exchange round may raise the final success rate or accelerate convergence, but it also increases message volume. Success rate alone cannot tell us whether that added communication was worthwhile.

Training curves compare one-round and two-round communication across seven multi-agent learning architectures — **More rounds can improve performance, at a cost.** A second communication round helps several architectures converge faster or reach a higher success rate, while necessarily increasing communication volume. Original slide from the EXTEND presentation.

02 · A Unified Communication Protocol Framework

A common description for diverse architectures

The proposed framework represents each agent with a local observation encoder followed by a repeated communication block. Before choosing an environment action, an agent may perform one or more communication rounds. Every round contains four operations:

Encode a message. The agent converts its current hidden state into information that teammates can receive.
Select a topology. The protocol determines which agents communicate with one another.
Aggregate received messages. Each agent combines information from its connected teammates.
Update the hidden state. The aggregated information changes the representation used for the next round or the final action.

A multi-agent framework repeats message encoding, topology selection, aggregation, and hidden-state updates before action selection — **The repeated communication block.** Changing the encoder, topology selector, aggregation rule, update rule, or round count allows different learned communication architectures to be compared inside one framework. Original diagram from the EXTEND presentation.

The framework follows centralized training with decentralized execution (CTDE). During training, the global state can support value estimation and optimization. During execution, each agent relies on its local observation and the messages it receives. That distinction matters in deployment: a protocol must remain useful when the centralized information available to the trainer no longer exists.

03 · Metrics and Training Objective Design

Three views of communication efficiency

Communication efficiency has several dimensions. Messages may be compact but indistinguishable across agents. A highly specialized protocol may still use too many transmissions. The study therefore introduces three Communication Efficiency Metrics (CEMs) that examine information content, role differentiation, and explicit communication use.

Metrics · 01

IEI, SEI, and TEI

Metric	What it measures	Preferred direction
Information Entropy Efficiency Index (IEI)	Message entropy required per unit of task success	Lower: more compact and structured messages
Specialization Efficiency Index (SEI)	Cross-agent message similarity relative to task success	Lower: stronger differentiation among agents’ messages and roles
Topology Efficiency Index (TEI)	Task success obtained per communication act	Higher: greater performance with fewer communication resources

Three panels define IEI for information compactness, SEI for role specialization, and TEI for resource parsimony — **Three complementary metrics.** IEI and SEI describe the structure and differentiation of learned messages; TEI makes the communication cost explicit. Original diagram from the EXTEND presentation.

During one-round training, the metrics develop distinct profiles across the seven protocols. IEI and SEI generally fall as messages become more compact and specialized, while TEI generally rises. These trajectories reveal differences in how a protocol communicates even when its task success looks similar.

IEI and SEI decrease while TEI increases over 2,000 training epochs for seven one-round communication protocols — **Distinct one-round efficiency profiles.** Protocols differ in message compactness, specialization, and task success per communication act throughout training. Original results from the EXTEND presentation.

Metrics · 02

Communication depth changes the trade-off

Increasing communication depth from one round to two or three generally reduces IEI and SEI. Repeated exchanges can sharpen the information in each message and encourage more specialized agent roles. TEI, however, decreases consistently because every added round produces more communication acts.

Bar charts compare IEI, SEI, and TEI for one, two, and three communication rounds across seven architectures — **The communication-depth trade-off.** Additional rounds often improve compactness and specialization, but the cost of extra transmissions lowers TEI. Original results from the EXTEND presentation.

More focused messages do not guarantee more efficient communication. Message quality must be interpreted together with the number of communication acts needed to achieve the task.

Objective · 03

Make efficiency part of the training objective

The study augments the original task loss with IEI- and SEI-based regularization. Training initially emphasizes task learning. As task success stabilizes, the efficiency terms become stronger and encourage messages that are more compact and differentiated. Smooth surrogate functions keep this adjustment suitable for gradient-based optimization.

TEI remains an evaluation metric in these experiments. Several evaluated architectures use a fixed communication topology, so the number of communication acts does not change during training and offers little useful gradient signal. IEI and SEI can directly shape message content and role specialization.

Training objective combines task loss with IEI and SEI regularization whose influence grows as task success stabilizes — **Adaptive efficiency pressure.** Agents first learn a workable task policy, then face stronger regularization that rewards compact and specialized communication. Original objective design from the EXTEND presentation.

04 · Experimental Evidence

Traffic Junction under partial observations

The experiments use Traffic Junction, a partially observable collision-avoidance benchmark in which multiple agents coordinate to cross an intersection. Each episode contains 20 steps, and the models train for 2,000 epochs. The evaluation covers MAGIC, CommNet, TarMAC, GA-Comm, IC3Net, BVME, and GACG under one, two, and three communication rounds. It also compares standard one-round training with efficiency-aware one-round training.

Animated Traffic Junction benchmark showing agents approaching and crossing a four-way intersection with local observation ranges — **The Traffic Junction benchmark.** Agents must avoid collisions while acting from partial observations, which makes timely information sharing useful. Animation supplied with the EXTEND presentation.

Result · 01

Optimize the objective before increasing communication depth

Compared with standard one-round training, adding IEI and SEI improves final task success and TEI for all seven architectures. The gains are strongest for IC3Net, CommNet, MAGIC, and GACG, and more moderate for TarMAC, GA-Comm, and BVME.

For GA-Comm, MAGIC, IC3Net, and GACG, the efficiency-aware one-round setting is preferable to adding communication rounds. The remaining architectures gain only marginally from extra rounds while their TEI falls. This suggests a practical design order: improve what agents say in the existing round before giving them more opportunities to communicate.

Success-rate and TEI curves compare standard one-round, efficiency-aware one-round, two-round, and three-round training across seven protocols — **Consistent gains from efficiency-aware training.** Objective augmentation improves final success and TEI across all seven evaluated protocols. Original results from the EXTEND presentation.

Result · 02

The objective changes the learned message structure

The modified objective changes more than the speed of task learning. For CommNet, IEI is 36% lower and reaches that level 59% faster, while SEI is 40% lower with 48% faster convergence. For IC3Net, IEI is 19% lower and converges 58% faster, while SEI is 36% lower with 56% faster convergence.

CommNet and IC3Net curves show lower IEI and SEI with efficiency-aware loss adjustment — **Compact and specialized messages.** Lower IEI and SEI show that efficiency-aware training changes the structure of the learned protocol, not only task convergence. Original results from the EXTEND presentation.

Result · 03

One efficient round improves the performance-cost balance

The performance-cost plot makes communication depth visible. The desired region combines high task success with fewer messages. Efficiency-aware one-round methods move toward that region, while two-round communication nearly doubles message volume. Under bandwidth or latency constraints, changing the objective can therefore produce a better operating point than adding another exchange.

Scatter plot places communication methods by maximum success rate and total message count, with the ideal region at high success and low cost — **The performance-cost frontier.** Efficiency-aware one-round methods move toward high success with fewer total messages. Original analysis from the EXTEND presentation.

05 · Implications and Outlook

Communication as a constrained system resource

The results suggest a direct workflow for communication-aware MARL. First, describe different protocols within a common multi-round CTDE model. Next, measure message compactness, role specialization, and explicit communication use with IEI, SEI, and TEI. When communication resources matter, optimize efficiency as part of training rather than treating it only as a post-hoc statistic.

Three-part summary describes a unified model, three communication-efficiency measures, and efficiency-aware optimization — **Model, measure, optimize.** A common protocol description enables comparison; the three metrics expose complementary costs; the adjusted objective improves task success and communication efficiency together. Original summary from the EXTEND presentation.

Additional communication rounds should not be the default response to a coordination problem. Their value depends on the useful information they contribute relative to bandwidth, latency, and energy cost. The best operating point will differ across deployments, but the trade-off should remain visible during system design.

Open questions

Can a learnable topology use explicit communication cost during training so that TEI becomes an optimization signal?
How should the metrics change when links are delayed, unreliable, quantized, or lossy?
How can compactness be separated from semantic usefulness so that low entropy does not reward uninformative messages?
How well do efficiency-aware protocols transfer across team sizes, tasks, and agent behaviors?

The power of coordination

“Alone we can do so little; together we can do so much.”
— Helen Keller

Related publications

X. Zhang, J. Yu*, and Z. Zhong, “Learning Efficient Communication Protocols for Multi-Agent Reinforcement Learning,” IEEE Transactions on Machine Learning in Communications and Networking, vol. 4, pp. 1335–1352, 2026. doi: 10.1109/TMLCN.2026.3719222.
X. Zhang, Z. Zhong, and J. Yu, “Learning Multi-Agent Communication Protocol: Study on Information Entropy Efficiency in MARL,” in The Third Reinforcement Learning Conference (RLC), Montreal, Canada, 2026. arXiv:2606.07200.