Mixture-of-Experts Reinforcement Learning

Expert-Space Exploration in MoE Reinforcement Learning

Explore alternative sparse computation paths during rollout—beyond conventional token-level sampling.

Summary of ESRL results across benchmarks. — **Main results.** ESRL improves average Pass@1 and Pass@8 across reasoning domains and multiple MoE routing architectures.

A routing perturbation activates an alternative expert path and changes the response. — **Case study.** The same prompt follows a different expert path, shifts the first-token distribution, and yields a complementary reasoning trajectory.

Introduction

Sparse expert routing introduces an overlooked exploration dimension: a model can vary not only its sampled tokens, but also the internal expert trajectories used to generate them.

Existing reinforcement-learning exploration is primarily implemented through token sampling, while the sparse computation structure of Mixture-of-Experts models remains deterministic. Yet each routing decision selects only a small subset of experts from a much larger expert pool. Perturbing this decision changes intermediate representations before decoding takes place, allowing the same prompt and token prefix to traverse alternative computation paths and produce complementary solutions.

We introduce Expert-Space Exploration Reinforcement Learning (ESRL), which makes this latent routing space directly explorable through anchored expert sampling, entropy-adaptive noise, and rollout routing replay. ESRL improves the primary Qwen3-30B-A3B setting by 3.2 percentage points in average Pass@1 and 4.5 points in Pass@8 over GRPO. The improvements extend across mathematical, scientific, and coding benchmarks and remain consistent on Sigma and Moonlight MoE architectures, supporting expert-space exploration as a general mechanism rather than a backbone-specific adjustment.

Why Expert-Space Exploration?

Empirical Analysis

We trace routing perturbations from the router itself to token distributions, complete sequences, and finally the diversity–quality frontier.

Router-level: perturbations activate different experts

Increasing routing-noise scale raises expert change rates and reduces path overlap. Layer-wise load profiles show that the effect is distributed across the network rather than concentrated in a few isolated layers. Together, these measurements verify that router perturbation changes the actual sparse computation graph.

Expert change rate and Jaccard overlap versus routing-noise scale. — **Routing path change.** Noise produces measurable substitutions and progressively lower Jaccard overlap, directly quantifying the distance from vanilla routing.

Expert-load coefficient of variation across MoE layers. — **Expert utilization.** Routing perturbation changes expert-load concentration across layers and exposes computation that deterministic top-k routing rarely activates.

Token-level: different routes shift next-token distributions

The vanilla top-1 token drops in rank and top-20 overlap decreases. PCA projections show movement in the full next-token distribution, not just a swap between near-tied candidates. Expert-path variation therefore reaches the decoder as a meaningful change in predictive belief.

Rank of the vanilla top token and top-20 Jaccard under routing noise. — **Local token shift.** Higher noise changes both the preferred token and the high-probability candidate set, widening the possible continuation space.

PCA visualization of next-token probability distributions under router noise. — **Distribution shift.** Perturbed routes occupy distinct regions in projected distribution space instead of merely reordering identical candidates.

Sequence-level: routing creates complementary response diversity

Expert-space perturbations reduce response similarity across checkpoints and decoding temperatures, so routing diversity survives autoregressive generation. The resulting differences accumulate into distinct solution strategies rather than disappearing after the first few tokens.

Sequence diversity measured by self-BLEU across checkpoints and temperatures. — **Response diversity.** Lower self-BLEU indicates less repetitive rollout groups and a broader set of complete reasoning trajectories.

Diversity–quality trade-off: useful exploration requires control

Unconstrained diversity can damage accuracy. This frontier motivates anchoring reliable experts, exploring plausible alternatives, and adapting noise to router uncertainty. ESRL is designed to move the frontier rather than maximize route changes in isolation.

Accuracy versus self-BLEU frontier under router-noise settings. — **Quality-aware frontier.** Expert-space exploration can move beyond the token-temperature frontier, while controlled sampling keeps the operating point in a high-quality region.

Method

ESRL expands routing diversity while protecting rollout quality and preserving the expert paths used for policy optimization. Its three components separately control where exploration occurs, how strongly each token is perturbed, and how the sampled path is aligned with the subsequent update.

Overview of ESRL rollout exploration and routing replay optimization. — Anchored and exploratory experts jointly form the rollout path. The selected route is stored and replayed during optimization, ensuring that the update follows the same sparse computation graph as generation.

Anchored Expert Sampling

Keep high-confidence experts as anchors and explore only within a plausible candidate pool. This preserves the model’s reliable computation path while leaving explicit capacity for alternative experts.

Entropy-Adaptive Noise

Scale routing perturbations with token-wise router uncertainty instead of applying fixed noise everywhere. Confident tokens receive conservative perturbations, whereas uncertain decisions are given more room to explore.

Rollout Routing Replay

Replay the rollout expert path during optimization to remove rollout–training routing mismatch. The policy is therefore updated on the same sparse computation graph that produced the sampled response.

Experiments

We first evaluate final-task performance across models and domains, then examine robustness and learning dynamics. Component ablations, parameter sweeps, and decoupled training further identify which design choices produce the gain and how they affect exploration quality.

Main Results

ESRL improves mathematical reasoning on the primary Qwen3 setting and consistently transfers across three distinct MoE routing architectures. The same exploration mechanism also improves science and code, showing that the benefit is not tied to a single reasoning format or expert topology.

Table 1 · Mathematical reasoning on Qwen3-30B-A3B

Method	Olympiad		AIME		AMC		Minerva		Average
Method	P@1	P@8	P@1	P@8	P@1	P@8	P@1	P@8	P@1	P@8
GRPO	51.2	70.2	21.9	43.1	41.9	72.8	40.8	52.6	38.9	59.7
GRPO-R3	52.0	70.6	22.8	46.0	43.9	73.9	40.3	53.7	39.7	61.1
GSPO	51.8	70.6	24.4	51.7	48.8	77.2	41.4	53.7	41.6	63.3
RO-GRPO	51.5	70.9	22.4	44.3	47.2	80.4	41.4	51.5	40.6	61.8
Aux-Loss	39.1	64.8	13.3	40.6	26.2	57.6	34.5	52.9	28.3	54.0
N-Sampling	33.1	62.8	16.7	37.7	18.9	52.2	27.7	52.6	24.1	51.3
ESRL	53.5	72.7	24.4	52.9	49.6	78.3	41.0	52.9	42.1	64.2

ESRL achieves 42.1 average Pass@1 and 64.2 Pass@8—+3.2 and +4.5 percentage points over GRPO. Improvements on AIME and AMC indicate that expert-path exploration is especially useful when a problem admits multiple intermediate reasoning strategies.

Table 2 · Generalization across MoE backbones

Backbone	Method	Avg. P@1	Avg. P@8
Qwen3-30B-A3B	GRPO	38.9	59.7
	GSPO	41.6	63.3
	ESRL	42.1	64.2
Sigma-20B-A0.5B	GRPO	20.4	38.6
	GSPO	20.5	40.0
	ESRL	21.1	41.3
Moonlight-16B-A3B	GRPO	27.5	47.4
	GSPO	27.4	47.7
	ESRL	28.8	51.0

ESRL improves the aggregate result on Qwen3, Sigma, and Moonlight, covering different expert capacities and routing designs. This cross-backbone consistency suggests that the method operates on a shared property of sparse routing rather than a model-specific router artifact.

Table 3 · Science and code on Qwen3-30B-A3B

Method	GPQA		MMLU-Pro		MMLU-Redux		LCB v6		Average
Method	P@1	P@8	P@1	P@8	P@1	P@8	P@1	P@8	P@1	P@8
GRPO	35.7	53.5	57.6	67.4	79.3	86.3	40.1	52.2	53.2	64.9
GSPO	38.9	66.7	60.4	76.0	75.3	90.2	41.0	53.1	53.9	71.5
GRPO-R3	36.1	62.1	59.0	74.7	78.6	89.0	35.6	51.9	52.3	69.4
ESRL	39.0	76.8	62.5	82.5	80.3	94.5	42.0	54.3	55.9	77.0

The largest coverage gains appear on GPQA and the MMLU variants, while LiveCodeBench improves in both Pass@1 and Pass@8. These results indicate that diverse expert trajectories remain useful beyond mathematical proof-style generation.

Robustness

We vary two independent rollout controls: decoding temperature and the number of sampled responses. ESRL stays above GRPO-R3 across both sweeps, showing that its gain is not tied to a single sampling budget. The consistent margin also separates expert-space exploration from simply increasing token-level randomness.

Average Pass@1 across rollout temperatures. — **Temperature robustness.** The gain persists as token sampling becomes more stochastic, indicating that routing exploration contributes an independent source of trajectory variation.

Pass@1 versus number of samples. — **N-sample robustness.** ESRL maintains a consistent margin from 32 to 256 samples, rather than relying on an unusually large rollout group to reveal rare successes.

Training Dynamics

The benefit appears in both the learning signal and expert utilization: ESRL creates more informative groups and changes load concentration over training. We track these quantities throughout optimization to distinguish a sustained learning effect from a transient change in the first few checkpoints.

Pass@1 and informative-group ratio. — **Informative groups.** More groups contain both successful and unsuccessful responses, providing non-degenerate reward contrasts for group-relative updates.

Global expert-load CV over training. — **Load CV.** CV tracks how evenly tokens are distributed across experts and whether exploration reduces concentration on a narrow subset.

Global expert imbalance over training. — **Imbalance factor.** A complementary measure confirms that altered expert utilization is sustained rather than caused by a few outlier routes.

Ablation Studies

The ablations isolate how selection weights, anchored experts, and adaptive noise protect training stability and rollout quality. Each component addresses a different failure mode: unstable mixture coefficients, implausible expert substitutions, or excessive perturbation on already-confident tokens.

Noisy weights

Method	Olympiad		AIME		AMC		Minerva		Average
Method	P@1	P@8	P@1	P@8	P@1	P@8	P@1	P@8	P@1	P@8
GRPO-R3	52.0	70.6	22.8	46.0	43.9	73.9	40.3	53.7	39.7	61.1
ESRL w/ noisy weights	51.3	69.0	23.2	47.0	45.1	72.8	40.6	53.3	40.1	60.5
ESRL w/ original weights	53.5	72.7	24.4	52.9	49.6	78.3	41.0	52.9	42.1	64.2

Noise should influence expert selection, while clean router logits continue to compute mixture weights. This separation adds discrete route diversity without unnecessarily perturbing the magnitude of the expert combination.

PPO clipping fraction for original and noisy expert weights. — **Optimization stability.** Noisy mixture weights trigger more PPO clipping and larger policy deviation, explaining their weaker downstream result.

Anchored expert sampling

Configuration	Pass@1					Pass@8
Configuration	Olympiad	AIME	AMC	Minerva	Avg.	Olympiad	AIME	AMC	Minerva	Avg.
GRPO-R3	52.0	22.8	43.9	40.3	39.7	70.6	46.0	73.9	53.7	61.1
Ksafe=4, M=16	53.5	24.4	49.6	41.0	42.1	72.7	47.3	78.3	52.9	62.8
Ksafe=4, M=32	53.3	24.6	47.6	41.4	41.7	71.8	49.2	78.3	53.7	63.2
Ksafe=2, M=16	52.6	26.0	46.9	41.0	41.6	72.9	53.1	79.4	54.4	64.9
Ksafe=2, M=32	52.6	24.8	49.5	41.6	42.1	70.5	46.5	78.3	52.9	62.1
No anchored experts	52.3	22.3	45.8	40.8	40.3	70.9	45.8	75.0	51.5	60.8

Anchoring avoids the collapse of full-range sampling; Ksafe=2, M=16 gives the strongest Pass@8 in this sweep. The anchor keeps a reliable computation core while the remaining expert slots introduce controlled alternatives.

Pass@1 versus noise scale for anchored and full-range sampling. — **Anchors preserve quality.** Full-range sampling deteriorates rapidly, while anchored sampling remains robust over a substantially wider noise range.

Adaptive noise scaling

Configuration	Pass@1					Pass@8
Configuration	Olympiad	AIME	AMC	Minerva	Avg.	Olympiad	AIME	AMC	Minerva	Avg.
GRPO-R3	52.0	22.8	43.9	40.3	39.7	70.6	46.0	73.9	53.7	61.1
Non-adaptive noise	50.3	20.8	42.7	40.9	38.7	70.0	42.8	69.6	55.9	59.6
Adaptive noise	53.5	24.4	49.6	41.0	42.1	72.7	52.9	78.3	52.9	64.2

Entropy-adaptive noise raises average Pass@1 from 38.7 to 42.1 and Pass@8 from 59.6 to 64.2 over fixed noise. Token-wise scaling concentrates exploration on uncertain routing decisions instead of perturbing all tokens equally.

Parameter Analysis

Four sweeps identify where routing exploration helps: anchored-expert count, perturbation position, exploration strength, and candidate-pool frontier. Rather than searching only for the highest accuracy point, we jointly inspect quality and self-BLEU to expose how each parameter moves the exploration frontier.

Accuracy and self-BLEU under anchored-expert counts. — **Anchored-expert count.** A small anchor set improves the frontier; too many anchors preserve the original path but restrict the diversity available to the rollout.

Accuracy and self-BLEU for perturbation positions. — **Perturbation position.** Layer placement determines how strongly route changes propagate through later representations and ultimately affect the response.

Quality and diversity across exploration strengths. — **Exploration strength.** Moderate perturbation adds useful diversity, whereas aggressive noise moves too far from the model’s high-quality routing region.

Quality-diversity frontier across candidate-pool settings. — **Candidate-pool frontier.** Restricting alternatives to plausible experts produces a better frontier than unconstrained full-range sampling.

Decoupled Rollout Exploration

Expert noise can collect alternative trajectories while optimization replays the saved route, decoupling exploration from the policy update. This experiment tests whether expert-space exploration is useful as a rollout-time search mechanism even when the optimization policy itself remains clean and path-consistent.

Joint versus decoupled training

Configuration	Pass@1					Pass@8
Configuration	Olympiad	AIME	AMC	Minerva	Avg.	Olympiad	AIME	AMC	Minerva	Avg.
GRPO-R3	52.0	22.8	43.9	40.3	39.7	70.6	46.0	73.9	53.7	61.1
Joint	53.5	24.4	49.6	41.0	42.1	72.7	52.9	78.3	52.9	64.2
Decoupled	53.4	25.2	49.2	41.7	42.4	71.4	50.4	78.3	51.8	63.0

Decoupling slightly improves average Pass@1, while joint training retains the strongest average Pass@8. Their comparable aggregate performance shows that most of the benefit originates from improved rollout coverage rather than permanently injecting noise into the policy.

Pass at N for GRPO-R3, joint ESRL, and decoupled ESRL. — **Coverage at large N.** Decoupled exploration overtakes joint training at larger N, indicating broader solution coverage without sacrificing the clean optimization path.

Citation

If you find ESRL useful in your research, please cite our paper.

@article{he2026esrl,
  title   = {Expert-Space Exploration in {MoE} Reinforcement Learning},
  author  = {He, Hongyi and Lin, Zhenghao and Liu, Xiao and Cheng, Peng and Lu, Yan and Gong, Yeyun},
  journal = {arXiv preprint arXiv:2609.13058},
  year    = {2026},
  doi     = {10.48550/arXiv.2609.13058},
  url     = {https://arxiv.org/abs/2609.13058}
}