--- title: "The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models" author: - "Zanlin Ni" - "Shenzhi Wang" - "Yang Yue" - "Tianyu Yu" - "Weilin Zhao" - "Yeguo Hua" - "Tianyi Chen" - "Jun Song" - "Cheng Yu" - "Bo Zheng" - "Gao Huang" bibliography: - main.bib --- # Abstract Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential. However, in this paper, we find that for general reasoning tasks (e.g., mathematics and coding), arbitrary-order generation may in fact limit the reasoning potential of dLLMs. We observe that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, which can lead to a premature collapse of solution coverage. This observation motivates a rethink of RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We show that effective reasoning can be elicited by simply forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, **JustGRPO**, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: # Introduction {#sec:intro} Recent research has witnessed a surge in Diffusion Large Language Models (dLLMs) [@nie2025large; @ye2025dream; @zhu2025llada; @zhao2025d1; @labs2025mercury], which challenge the dominant autoregressive (AR) paradigm [@brown2020language; @achiam2023gpt] by treating sequence generation as a discrete denoising process. Central to the appeal of dLLMs is their theoretical flexibility, which offers two distinct advantages over the strict left-to-right causal chain of AR models: efficient parallel decoding and the capability for arbitrary-order generation. While the efficiency gains of parallel decoding have been well-established [@fastdllm; @labs2025mercury; @geminidiffusion; @song2025seed; @wu2025fast], the implications of arbitrary-order generation remain relatively less explored. Theoretically, the unconstrained generation order constitutes a superset of the fixed autoregressive trajectory and has the potential to unlock more advanced problem-solving abilities. Meanwhile, some works [@ye2024beyond; @kim2025train] have demonstrated the superiority of non-sequential generation for specific tasks such as sudoku and zebra puzzles. Driven by these successes and the theoretical promise, the community has increasingly adopted Reinforcement Learning (RL) to elicit similar advanced capabilities for *general reasoning tasks* such as mathematics and coding[^1] [@zhao2025d1; @wang2025spg; @gong2025diffucoder; @rojas2025improving; @ou2025principled]. In this work, we present a counter-intuitive observation: for these general reasoning tasks, arbitrary-order generation may in fact narrow rather than expand the reasoning potential elicitable by RL. To rigorously assess this, we employ Pass@$k$ [@chen2021evaluating], which measures the coverage of solution space. Recent studies suggest that RL primarily acts to sharpen the base model's distribution; consequently, the Pass@$k$ performance of the base model serves as an effective proxy for the achievable reasoning limits of the model after RL training [@yue2025does; @liu2025prorl]. Under this metric, we compare the reasoning potential of arbitrary-order generation against standard AR decoding using LLaDA-Instruct [@nie2025large]. As shown in Figure `\ref{fig:teaser}`{=latex} (Left), restricting a dLLM to standard AR order tends to yield a *higher* Pass@$k$, and consequently a higher reasoning boundary, than its flexible counterpart. ![ **Less flexibility unlocks better reasoning potential.** *Left*: We observe a counter-intuitive phenomenon where restricting dLLMs to standard Autoregressive (AR) order expands the reasoning solution space. *Right*: Motivated by this, we propose \`\`**JustGRPO**". By foregoing arbitrary-order adaptations and adopting just GRPO, we elicit the reasoning capability of dLLMs more effectively. Results reported on LLaDA-Instruct [@nie2025large] with generation length of 256. Baseline results are quoted from their papers. We also include reproduced results under a matched configuration in Table `\ref{tab:controlled_comparison}`{=latex}. `\label{fig:teaser}`{=latex} ](figures/teaser.png){width="\\linewidth"} This counter-intuitive observation is closely related to how different orders handle uncertainty. ```{=latex} \begin{wrapfigure}{r}{0.52\textwidth} \includegraphics[width=\linewidth]{figures/illustrate.pdf} \caption{ \textbf{Confronting \emph{vs.} bypassing uncertainty.} (a) \textbf{AR order} preserves reasoning space by forcing decisions at uncertain tokens. (b) \textbf{Arbitrary order} bypasses uncertainty and resolves easier tokens first. Once future context is established, the original branching possibilities are effectively pruned. \label{fig:illustrate} } \end{wrapfigure} ``` Reasoning is inherently non-uniform: it hinges on sparse \`\`forking tokens", typically connectives like \`\`Therefore" or \`\`Since", which do not merely continue a sentence but steer the logical trajectory into distinct branches [@wang2025beyond; @cheng2025reasoning; @huang2025low]. At these forks, the reasoning path diverges, naturally manifesting as localized spikes in entropy [@wang2025beyond]. Standard AR decoding compels the model to *confront* this uncertainty (Figure `\ref{fig:illustrate}`{=latex}a). By sampling exactly at these forks, the model is able to explore different reasoning paths, thereby preserving the diversity of the generated rationales. Arbitrary-order generation, however, allows the model to *bypass* these hard decisions (Figure `\ref{fig:illustrate}`{=latex}b). This flexibility is typically exploited to prioritize low-uncertainty tokens [@nie2025large], which effectively anchors the reasoning path on easier parts before deciding the connectives that lead to them. By the time the model returns to infill the bypassed forks, the established future context has prematurely resolved their ambiguity. We term this phenomenon *entropy degradation*: the high entropy originally inherent to the fork is suppressed. Effectively, the model is no longer navigating a branching point, but aligning the connective to bridge a pre-determined gap. The above observations motivate a rethink of RL for dLLMs in these general reasoning tasks. Current methods operate under the assumption that preserving arbitrary-order flexibility is essential. This commitment incurs a heavy tax: algorithms must grapple with a combinatorial explosion of denoising trajectories [@zhao2025d1; @yang2025mmada; @gong2025diffucoder] and intractable marginal likelihoods [@ou2025principled; @wang2025spg], forcing reliance on inaccurate approximations [@wang2025spg; @ou2025principled; @rojas2025improving]. However, if arbitrary order is non-essential, or even detrimental for eliciting the potential for the target tasks, this complexity may be unnecessary. To this end, we propose a return to simplicity with **JustGRPO**. We show that eliciting reasoning potential for general reasoning tasks does not necessarily require complex, diffusion-specific RL adaptations. Instead, it can be effectively achieved by simply treating the dLLM as an AR policy during RL training. This allows us to apply standard GRPO [@shao2024deepseekmath] without modification, transforming an otherwise intractable optimization into a well-defined task. Crucially, this AR formulation only serves as a \`\`scaffold" during RL training for better exploration. The dLLM's native architecture remains untouched, e.g., no causal masking imposed, which preserves their bidirectional attention and discrete diffusion formulation. As a result, the *elicitation of reasoning ability* (which benefits from sequential exploration) and the *inference execution* (which benefits from parallel decoding of dLLMs) are decoupled. Despite its simplicity, JustGRPO is surprisingly effective. It achieves strong results (e.g., 89.1% accuracy on GSM8K, 45.1% on MATH-500), comparing favorably with methods relying on intricate diffusion-specific RL adaptations (Table `\ref{tab:main_results}`{=latex}). More importantly, as shown in Figure `\ref{fig:eb_sampler_comparison}`{=latex}, the AR-trained model remains fully compatible with parallel decoding, suggesting that the reasoning capabilities acquired via AR exploration need not compromise the signature inference efficiency of dLLMs. # Preliminaries {#sec:prelim} ## Diffusion Large Language Models Diffusion Large Language Models (dLLMs), particularly Masked Diffusion Models (MDMs), generate sequences by iteratively denoising a masked state $x_t$ initialized from fully masked tokens. The process is indexed by a continuous time variable $t \in [0,1]$, representing the masking ratio. Given a clean sequence $x_0$, the forward process independently masks each token with probability $t$: $$q(x_{t,k} \mid x_{0,k}) = \begin{cases} \texttt{[MASK]}, & \text{with prob } t, \\ x_{0,k}, & \text{with prob } 1 - t, \end{cases}$$ where $k$ is the token index. A neural network $p_\theta(x_0 \mid x_t)$ estimates the original token distribution at masked positions. During inference, generation starts from $x_1$ (all `[MASK]`) and iteratively unmasks a subset of tokens based on heuristics such as confidence scores, updating $x_t \rightarrow x_{t-\Delta t}$ until completion. As a special case, dLLMs can also perform generation autoregressively by always unmasking the leftmost token. The model is trained by minimizing the Negative Evidence Lower Bound, which reduces to a weighted cross entropy loss over masked tokens: $$\mathcal{L}_{\mathrm{MDM}}(\theta) = -\mathbb{E}_{t \sim \mathcal{U}[0,1],\, x_t \sim q(x_t \mid x_0)} \left[ \frac{1}{t} \sum_{k=1}^{L} \mathbf{1}[x_{t,k} = \texttt{[MASK]}]\, \log p_\theta(x_{0,k} \mid x_t) \right].$$ ## Group Relative Policy Optimization {#sec:prelim_grpo} Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that avoids value function estimation by using group level reward normalization. It is typically applied to autoregressive policies $\pi_\theta(o_k \mid o_{`{=html}5 tokens/step). This suggests that instead of overfitting to a specific trajectory, the AR formulation may serve as an effective training \`\`scaffold" to refine the underlying model distribution. Such a refined distribution appears to be more resilient to the approximations inherent in parallel sampling compared to the original model. ## Training Efficiency {#subsec:efficiency} ```{=latex} \begin{wrapfigure}{r}{0.4\textwidth} \includegraphics[width=\linewidth]{figures/gsm8k_walltime.pdf} \caption{\textbf{Training efficiency on GSM8K.} Despite its larger per-iteration overhead, JustGRPO achieves a better accuracy/wall-time trade-off than the approximation-based baseline (ESPO), which saturates early. JustGRPO-Fast, which computes the probability ratio $\rho_{i,k}$ only at the top-25\% highest-entropy positions, accelerates training further. Wall time is measured on 16$\times$H100 GPUs.} \label{fig:walltime_efficiency} \end{wrapfigure} ``` Applying exact GRPO to dLLMs entails a structural trade-off: unlike autoregressive models, dLLMs must evaluate each position independently rather than in a single causal forward pass, so computing exact likelihoods incurs extra per-iteration overhead. Prior methods [@zhao2025d1; @ou2025principled] sidestep this cost by relying on approximations, which might suggest that exact optimization is impractical by comparison. Yet Figure `\ref{fig:walltime_efficiency}`{=latex} suggests otherwise: even with this overhead, JustGRPO remains competitive in its accuracy/wall-clock trade-off, matching the peak accuracy of a representative approximation-based baseline (ESPO) within comparable time and continues to improve. Moreover, this overhead is far from fundamental. Its dominant source is computing the per-position probability ratios $\rho_{i,k}$ in the GRPO objective (Eq. `\ref{eq:justgrpo-loss}`{=latex}). Motivated by our finding that reasoning is steered by a sparse set of high-entropy forking tokens (Section `\ref{sec:rethink_value_of_AO}`{=latex}), we introduce JustGRPO-Fast, which evaluates $\rho_{i,k}$ only at the top-25% highest-entropy positions, eliminating 75% of these forward evaluations. Importantly, since per-token entropy can be directly recorded during the rollout stage, identifying the high-entropy positions incurs negligible overhead. Empirically, JustGRPO-Fast improves the accuracy/wall-time trade-off even over the default JustGRPO, which itself already compares favorably with the approximation-based baseline (ESPO), reaching comparable accuracy in less wall-clock time. # Related Work {#sec:related} #### Diffusion language models. Motivated by the success of diffusion models in continuous domains [@ho2020denoising; @rombach2022high], recent work has extended diffusion to discrete text generation. Early embedding-space approaches [@diffusionlm; @diffuseq; @ssdlm] faced optimization and discretization challenges, leading to the adoption of masked diffusion models [@sedd; @mdlm; @shi2024simplified; @ou2024your], which operate directly in the discrete token space via random masking. Recent large-scale models like LLaDA [@nie2025large] and Dream [@ye2025dream] achieve performance competitive with autoregressive (AR) models. Among various advantages of dLLMs discussed in the literature, two aspects have received particular attention. First, dLLMs naturally support parallel decoding, enabling significant inference acceleration [@fastdllm; @ben2025accelerated; @labs2025mercury; @geminidiffusion; @song2025seed]. Second, their non-autoregressive formulation allows for arbitrary-order token generation, which has been hypothesized to benefit complex reasoning by relaxing strict left-to-right constraints [@ye2024beyond; @kim2025train]. #### The value of order arbitrariness. While early studies validated arbitrary order generation in constrained tasks [@ye2024beyond; @nie2025large; @kim2025train], recent research extends this to general reasoning. Some works demonstrate that dLLMs can naturally exhibit non-standard decoding behaviors (e.g., sketch-first) [@xie2025dream; @gong2025diffucoder], while others explicitly optimize decoding schedules to boost performance [@Peng2025PathPF; @huang2025reinforcing]. Moreover, some studies [@gong2025diffucoder; @hong2025improving; @shen2026improvingdiffusionlanguagemodel] also employ Pass@$k$ to measure the reasoning potential of dLLMs. Recognizing this potential, these works predominantly focus on enhancing performance within the arbitrary-order paradigm, leaving less-examined whether the benefit is uniquely tied to the order arbitrariness. Recently, [@du2025understanding] challenged the necessity of arbitrary order, arguing that forcing models to learn uniform permutations causes a significantly looser approximation of the underlying data distribution. While their analysis focuses on pre-training, we observe a parallel failure mode in reasoning and RL, suggesting that the flexibility can degrade exploration essential for RL training. This is also related to the broader view [@varshney2006toward] in which the ordering of a sequence carries useful structural information beyond its content, which arbitrary-order generation tends to overlook. #### Reinforcement learning for diffusion language models. Reinforcement learning for dLLMs faces structural optimization hurdles distinct from the autoregressive paradigm, primarily stemming from the combinatorial explosion of denoising trajectories. While early attempts [@zhao2025d1; @yang2025mmada; @gong2025diffucoder] sought to adapt token-level formulations directly, they were often limited by ill-defined state transitions, necessitating reliance on mean-field approximations. Consequently, the field has shifted toward sequence-level perspectives [@wang2025spg; @rojas2025improving; @ou2025principled], employing various surrogates to approximate the intractable marginal likelihood. Yet, an off-policy misalignment often persists across these methods: the heuristic-guided sampling required for efficient exploration diverges from the underlying diffusion prior, which can bias the gradients without principled correction [@schulman2015trust]. Two notable exceptions partially address these issues. LLaDOU [@huang2025reinforcing] introduces an auxiliary policy to model token position selection, enabling direct trajectory likelihood estimation, while TraceRL [@wang2025revolutionizing] aligns optimization with inference traces via a shrinkage-based step-wise MDP. Nevertheless, both retain the full arbitrary-order mechanism, implicitly assuming its necessity, whereas we question whether effective RL training can be achieved more simply by reconsidering the need for arbitrary order itself. # Conclusion {#sec:conclusion} The intuitive appeal of diffusion language models lies in their order arbitrariness, often perceived as a promising mechanism for navigating complex reasoning paths. However, our study suggests that for general reasoning tasks such as mathematics and coding, this unrestricted flexibility may inadvertently constrain the reasoning potential. Specifically, arbitrary-order generation appears to prioritize the refinement of a single trajectory over broader solution coverage. This observation points towards a simpler solution for eliciting reasoning capabilities of dLLMs. By aligning the model using a standard autoregressive objective, we enable the direct application of Group Relative Policy Optimization (GRPO) without complex adaptations tailored for order arbitrariness. This approach, JustGRPO, suggests that intentionally constraining the exploration order during training, despite its simplicity, can yield notable improvements in reasoning performance, while fully preserving the parallel decoding capabilities of dLLMs at inference. By returning to the basic left-to-right order for alignment, this work invites a reconsideration of the trade-offs between arbitrary *vs.* AR order in the development of diffusion language models. # Impact Statement {#impact-statement .unnumbered} This paper presents work whose goal is to advance the field of Machine Learning, specifically in the area of diffusion language models and reinforcement learning for reasoning tasks. Our findings suggest that simpler training approaches can be surprisingly effective, which may reduce computational costs and make these techniques more accessible. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here, as the primary contribution is methodological and aimed at improving the understanding and training of language models. # Acknowledgements {#acknowledgements .unnumbered} This work is supported in part by the National Key R&D Program of China under Grant 2024YFB4708200, the National Natural Science Foundation of China under Grants U24B20173, U2541227 and 625B2107, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYGXQNJSKYCXNLZCXM-I20. ```{=latex} \bibliographystyle{icml2025} ``` ```{=latex} \newpage ``` ```{=latex} \appendix ``` # Appendix {#sec:appendix .unnumbered} # Experimental Details {#app:experimental_setups} ## Data Preparation For mathematical reasoning tasks, we train on the official training split of each dataset, following the standard protocol in [@zhao2025d1; @ou2025principled]. For code generation tasks, we adopt the AceCoder-87K dataset [@zeng2025acecoder]. Following the data processing pipeline of DiffuCoder [@gong2025diffucoder], we further select 21K challenging samples from AceCoder-87K that are equipped with verifiable unit tests. ## Training Configuration Our training setup largely follows [@huang2025reinforcing]. Unlike [@huang2025reinforcing], we perform reinforcement learning directly on the dLLM without introducing additional trainable modules. During the rollout phase, we adopt exact autoregressive sampling, which allows direct log-probability computation under the standard GRPO formulation. We also reduce the total number of training steps, as JustGRPO exhibits fast and stable convergence. All experiments are conducted on 16$\times$ NVIDIA H100 GPUs. Detailed hyperparameters are reported in Table `\ref{tab:hyperparams}`{=latex}. ::: {#tab:hyperparams} **Hyperparameter** **Value** -------------------------------------- -------------------- Base Model LLaDA 8B Instruct RL Algorithm GRPO Optimizer AdamW Learning Rate $5 \times 10^{-6}$ LR Scheduler Constant Weight Decay 0.0 Optimizer Betas $(\beta_1, \beta_2)$ $(0.9, 0.999)$ Global Batch Size 64 Group Size ($G$) 16 Policy Update Steps 1 Training Steps 125 Max Completion Length 256 Sampling Temperature 1.0 KL Penalty Coefficient 0.0 : **Training hyperparameters** for JustGRPO`\protect`{=latex}`\footnotemark{}`{=latex}. ::: ```{=latex} \footnotetext{Empirically, GSM8K converges much faster (around 50 steps already reaches $\sim$89\% accuracy), while coding tasks (HumanEval, MBPP) are slower and benefit from the full 125 steps. We therefore used 125 steps uniformly across all datasets for consistency. } ``` ## Reward Function {#app:reward_function} #### Mathematical Reasoning Tasks. We employ a binary reward scheme. Each completion receives a reward of 1 if and only if the final answer is mathematically equivalent to the ground-truth solution following [@huang2025reinforcing]. #### Code Generation Tasks. For code generation, the total reward $r$ is defined as the sum of a correctness reward $r_{\text{code}}$ and a format reward $r_{\text{format}}$: $$r = r_{\text{code}} + r_{\text{format}}.$$ - **Correctness reward ($r_{\text{code}}$)**: Defined as the pass rate (ranging from 0 to 1) of the generated code on the provided unit tests. This term is only evaluated when $r_{\text{format}} = 1$. - **Format reward ($r_{\text{format}}$)**: A heuristic reward designed to encourage syntactically valid outputs. - $1.0$: A valid Markdown code block with correct Python syntax. - $0.5$: A valid Markdown code block that contains syntax errors. - $0.0$: Failure to generate a valid Markdown code block. # More Analysis Results on Reasoning Boundary {#app:boundary_ablations} To further validate the robustness of our findings about reasoning boundary in Section `\ref{sec:rethink_value_of_AO}`{=latex}, we conduct extended analyses on the HumanEval benchmark using the LLaDA-Instruct model [@nie2025large]. We investigate the impact of temperature and sampling strategies on reasoning performance. ## Temperature Analysis {#app:temperature_analysis} ![ **Pass@$k$ comparison with different temperatures`\label{fig:passk_comparison_temperature}`{=latex}**. ](figures/humaneval_temp.png){width=".75\\linewidth"} On HumanEval, we extend Pass@$k$ plot to $k=256$: at temperatures near $1.0$, the crossover between the two modes occurs at relatively large $k$, and the larger budget allows us to examine the comparison beyond it. As shown in `\cref{fig:passk_comparison_temperature}`{=latex}, the AR mode exhibits a standard pattern: performance peaks at moderate temperatures ($T \approx 0.6$) and degrades when $T > 1.0$, whereas the Arbitrary Order mode attains its peak performance at higher temperatures. This observation aligns with the *entropy degradation* mechanism discussed in Section `\ref{sec:rethink_value_of_AO}`{=latex}: the diffusion sampler inherently suppresses uncertainty at critical branching points, thereby requiring higher temperatures to induce sufficient exploration. Notably, even under these optimized settings, the arbitrary order mode does not match the reasoning potential of the AR mode in our experiments. As illustrated in the \`\`Optimal" comparison setting (bottom right of `\cref{fig:passk_comparison_temperature}`{=latex}), the best-performing AR configuration still shows better scaling behavior than the optimal arbitrary order baseline. A plausible explanation is that excessively high temperatures in arbitrary order decoding inject noise into tokens that require high determinism (e.g., code syntax or mathematical suffixes), leading to nonsensical outputs and degraded results [@wang2025beyond; @huang2025low]. ## Different Sampling Algorithms We also experiment with different sampling algorithms, such as negative entropy sampling (Neg-Entropy)  [@ye2025dream] and top-$k$ margin sampling (Margin)  [@kim2025train] in Figure `\ref{fig:alg_comparison}`{=latex}(a). ![ **(a) Sampling algorithm comparison**. **(b) Correlation between different sampling algorithms and AR in per-problem accuracy**. Algorithms with higher Pass@$k$ performance tend to show *higher correlation with AR* in problem-level performance. `\label{fig:alg_comparison}`{=latex} ](figures/alg_compare.png){width="\\linewidth"} The results show that although more sophisticated sampling algorithms can achieve better Pass@$k$ performance than default confidence-based sampling, they still cannot catch up with AR order. Meanwhile, these better Pass@$k$ algorithms also show slightly worse Pass@$1$ performance compared to confidence-based sampling, making the overall Pass@$k$ curve closer to AR order's Pass@$k$ curve. To investigate this similarity, we calculate the per-problem accuracy correlation between different sampling algorithms and the AR mode (Figure `\ref{fig:alg_comparison}`{=latex}(b)). We observe that algorithms with higher scaling potential (higher Pass@$128$) consistently show stronger correlation with AR, with the most effective method (Neg-Entropy) achieving a correlation of $0.970$. This suggests that sampling algorithms with better Pass@$k$ tend to behave more like AR in performance characteristics. ![ **Entropy comparison results on more forking tokens`\label{fig:entropy_bar_compare_more}`{=latex}**. ](figures/entropy_bar_compare_more.png){width="\\linewidth"} # More Results on Entropy Degradation {#app:more_results_entropy_degradation} To validate the robustness of the \`\`entropy degradation" phenomenon observed in Section `\ref{sec:rethink_value_of_AO}`{=latex}, we extended our analysis to a wider range of logical connectors. We conducted experiments on a comprehensive set of common connectives that typically serve as forking tokens in reasoning chains. The results, shown in `\cref{fig:entropy_bar_compare_more}`{=latex}, show that the phenomenon is consistent across these diverse tokens. Similar to the primary findings, the AR order maintains higher average entropy at these forks, indicating active reasoning and decision-making. Conversely, the Arbitrary Order consistently results in lower entropy, consistent with the premature collapse of branching possibilities. The specific forking tokens evaluated in this extended experiment include: \`\`Therefore", \`\`Thus", \`\`So", \`\`Since", \`\`When", \`\`Given", \`\`However", \`\`Let", \`\`First", \`\`Then", \`\`Next", \`\`Finally", \`\`Now", \`\`Similarly", \`\`Calculate", \`\`Solving", \`\`Notice", \`\`Specifically", \`\`Follows", \`\`Because", \`\`But", \`\`Or", \`\`Consider", \`\`Also", \`\`Express", and \`\`Write". # Is Random Order an Alternative? {#app:random_order} Our main analysis in Section `\ref{sec:rethink_value_of_AO}`{=latex} concerns how dLLMs are used in practice, where decoding is driven by confidence-based sampling and semi-autoregressive sampling [@nie2025large; @zhao2025d1]. This is the setting in which bypassing occurs and entropy degrades, and it is the setting our analysis targets. A dLLM, however, is not required to use such a sampler. A natural question is whether a random decoding order, which connects to the original dLLM formulation [@mdlm; @shi2024simplified], would behave differently. We find that it does not help, either. #### Random order does not help coverage, and it breaks generation. Under Pass@$128$, random order is at best comparable to confidence-based sampling, and it still falls well short of AR order (Table `\ref{tab:random_passk}`{=latex}, top). Its Pass@$1$ is far worse, well below both baselines (Table `\ref{tab:random_passk}`{=latex}, bottom). The reason is straightforward. In a dLLM, each token is predicted from its surrounding context, but a random order often requires the model to decide a token before that context exists, for example predicting token 30 when only token 1 is known. The resulting output is not merely incorrect, but structurally broken, as the following LLaDA rollout illustrates: > `\small`{=latex}`\ttfamily > 3`{=latex}. \*\*Third Month:\*\*\ >  - The number of downloads reduced by 30% compared to the second month.\ >  \\\[ 180 - (   \\times 180 - (   - 180) = 180 - 0.30 \\times 180 = 54 ...\\\] The LaTeX is corrupted, with dangling operators and missing operands. Outputs of this kind rarely earn a reward, which limits what RL can learn from. #### Random order does not survive RL either. We then bring random order into our own training setup. To apply an exact likelihood factorization akin to JustGRPO, we adopt a single random permutation and keep it fixed throughout training. We call this variant JustGRPO-Random. Keeping the rest of the pipeline unchanged, JustGRPO-Random only reaches 82.2% on GSM8K, compared to 89.1% for AR order (Table `\ref{tab:random_order_ablation}`{=latex}). A random order thus does not appear to be a remedy: in our experiments, it neither widens coverage as a sampler nor delivers strong reasoning after RL. ::: {#tab:random_passk} **Method** **GSM8K** **MATH-500** **MBPP** **HumanEval** -------------------- ----------- -------------- ---------- --------------- *Pass@$128(\%)$* Confidence-based 97.0 71.4 67.1 67.1 Fully Random 97.5 70.6 71.8 64.9 AR (Left-to-right) **99.0** **75.6** **78.7** **83.0** *Pass@$1(\%)$* Confidence-based **78.6** **30.4** **40.7** **40.5** Fully Random 43.3 14.1 15.0 12.4 AR (Left-to-right) 78.0 27.9 34.9 34.5 : **Random order does not improve coverage and collapses single-shot accuracy.** Pass@$128$ (top) and Pass@$1$ (bottom) for confidence-based arbitrary order, fully random order, and AR order, measured on LLaDA-Instruct. ::: ::: {#tab:random_order_ablation} **Method** **GSM8K Acc.** $\Delta$ --------------------- ---------------- ---------- LLaDA-Instruct 78.6% --- JustGRPO-Random 82.2% +3.6% JustGRPO (AR order) **89.1%** +10.5% : **Random order also fails in RL post-training.** *JustGRPO-Random* is identical to JustGRPO except that it decodes in a fixed random order rather than left-to-right. Results are measured on GSM8K with LLaDA-Instruct. ::: # General Capability Preservation {#app:general_capability} Beyond reasoning, we verify that JustGRPO preserves the model's general capabilities. We evaluate the previously trained JustGRPO checkpoint against the original LLaDA-Instruct model on four standard non-reasoning benchmarks: MMLU, MMLU-Pro, HellaSwag, and ARC-C. As reported in Table `\ref{tab:general_capability}`{=latex}, general capabilities are largely preserved, with only small task-dependent fluctuations. This is expected: the AR formulation in JustGRPO acts purely as a training-time scaffold for RL exploration rather than a structural constraint on the model. The architecture, attention pattern, and pretrained knowledge are largely preserved, so JustGRPO improves how the model explores reasoning trajectories during RL without interfering with how it encodes and expresses general knowledge. ::: {#tab:general_capability} **Model** **MMLU** **MMLU-Pro** **HellaSwag** **ARC-C** ---------------- ---------- -------------- --------------- ----------- LLaDA-Instruct 65.5 **37.0** 74.6 **88.5** JustGRPO **65.8** 36.7 **74.8** 87.5 : **General (non-reasoning) capabilities are preserved** after JustGRPO. Accuracy (%) on four standard benchmarks for the base model and the JustGRPO checkpoint. ::: #### Note on bidirectional refinement. Because JustGRPO imposes the AR factorization only as a training-time scaffold rather than a constraint on inference, we expect it to largely preserve the bidirectional behavior characteristic of dLLMs. This expectation is consistent with the preserved parallel decoding capability reported in Section `\ref{subsec:parallel_decoding}`{=latex}. A natural extension is to ask whether dLLMs can iteratively refine and revise their already-decoded outputs. This connects to corrective approaches such as CDLM [@zhang2025corrective] and to the bidirectional-context settings studied in ParallelBench [@kang2025parallelbench], and we view it as a promising direction that could complement JustGRPO. [^1]: We focus on these domains as the primary representatives of *general reasoning* due to their broad practical utility and central role in benchmarking the core intelligence of reasoning models.