Skip to content
mixing-floor
MOSS Workshop COLM 2026

The Compute Floor of Data Mixing

When loss-based reweighters start to drain easy synthetic text

Shayan Salehi1 Iman Mohammadi2
1University of Waterloo 2University of California, Irvine
Paper Code Results DOI BibTeX
Synthetic share per roundpythia-160m · 3 seeds
Every method starts at 0.50. The loss-monotone family (warm) drains the synthetic text; the interaction and gradient estimators (cool) stay near uniform.
0.50→0.20
Group-DRO, final synthetic share
0.50→0.37
DoReMi-style, final synthetic share
≈ 1016
FLOPs per run: the compute floor above which the split reads cleanly
1 GPU
2.30 × 1017 FLOPs per Full-tier run, a few GPU-hours per seed
§ 01

Abstract

Automatic data-mixing methods are usually evaluated with pretraining sweeps from 70M to billions of parameters, which leaves a basic question open: how much compute does it take to tell these methods apart?

We study this on a pool split by provenance, 9 real Pile subsets and 9 synthetic instruction corpora, where every method receives the same per-domain validation-loss signal and we record how it shifts the synthetic share of the mixture. At 160M parameters (2.30 × 1017 FLOPs per run), the adaptive methods separate into two families across three seeds: loss-monotone reweighters move mass toward the harder real text (Group-DRO to a synthetic share of 0.20, DoReMi-style to 0.37), while interaction and gradient estimators stay near the 0.50 start. The split follows a training-induced loss gap that makes real text the high-loss provenance.

Two cheaper tiers locate a compute floor: the split is already clear at 1.2 × 1016 FLOPs, but at 2.5 × 1014 FLOPs it becomes seed-dominated, with Group-DRO showing no net direction and DoReMi-style reversing. Held-out test loss gives the least-moving method, Aioli, the lowest mean, a result we hedge given n = 3 and the confound of online training. The protocol diagnoses which mixture a policy selects, not final model quality; all code and artifacts are released at github.com/mixing-floor/compute-floor.

  1. Two families, read off one number.

    Given the identical loss signal, loss-monotone reweighters cut the synthetic share; Aioli, DoGE and Skill-It hold it near 0.50. The split is tight across all three seeds.

  2. The trajectory decides, not the optimizer.

    At round 0 the provenances are tied (2.94 vs 2.90 nats). Online training opens a gap of about one nat, real 4.09 vs synthetic 3.06, and a score that rises with own loss must chase it.

  3. A floor, three orders below a frontier run.

    The split already holds at 1.2 × 1016 FLOPs on pythia-70m and is seed-dominated at 2.5 × 1014. Below the floor, report a run as inconclusive, not as a null.

§ 02

Protocol: freeze the signal, vary the rule

Every method runs the identical protocol, so the only variable across runs is the method itself. Mixing is sampling: each step draws a batch by multinomial(batch, p) over 18 domains, and every online method moves p off one signal, per-domain validation loss. We log p each round and read off the mass on the synthetic domains.

Real 9 Pile subsets

raw text, no template

Synthetic 9 instruction corpora

### Instruction: {prompt} ### Response: {response}

The synthetic class is defined by instruction format rather than strictly by model generation: the slot labelled Self-Instruct holds the human-written Dolly-15k, and ShareGPT pairs human prompts with model responses.

01 · pool
Split by provenance
Disjoint train / val / test token blocks per domain, fixed split seed 1234.
02 · signal
One shared loss
Per-domain validation loss is the only input every online method consumes.
03 · update
Multiplicative weights
Each method differs only in the score si it exponentiates.
04 · read-off
Synthetic share
Σsynth pi per round, plus the L2 distance from uniform.
Family 1

Loss-monotone

The score rises with a domain's own (excess) loss, so the update is forced toward the higher-loss provenance.

Family 2

Interaction & gradient

The score asks how much training on a domain lowers loss elsewhere: Aioli's mixing-law column sums, DoGE's gradient alignment.

Hybrid & control

Skill-It, Stratified

Skill-It routes loss through a learned skills graph and reshuffles within each provenance. Stratified (p = 1/k) never moves.

§ 03

Turn the compute dial

The same six methods, the same pool, at three budgets. At the Full and Intermediate tiers the family split is clean. At the Minimal tier it falls apart: Group-DRO points nowhere on net and DoReMi-style climbs to 0.75.

Metric
Fig. 1Mean over seeds 0, 1, 2 with ±1σ bands; hover for values, click a method to hide it. Synthetic share is Σ p over the 9 synthetic domains (uniform start 0.50). L2 from uniform counts how much mass moves regardless of direction: Skill-It ranks second at the Full tier (0.111) while its share barely moves, because it reshuffles within each provenance class.
§ 04

Where the floor sits

The floor is the lowest budget at which the loss-monotone methods agree in direction across seeds. Group-DRO's three Intermediate-tier seeds land in [0.234, 0.239]; at the Minimal tier they scatter to 0.996, 0.253 and 0.262. Both cheaper tiers run pythia-70m, so the floor is a budget effect at fixed model size, not a size effect.

(a) Final synthetic share vs. compute
Mean ± 1σ; small marks are individual seeds
(b) Seed spread collapses with budget
Std of the final synthetic share over seeds
Fig. 2The hatched band marks the interval in which the floor lies: we run no budget between 2.5 × 1014 and 1.2 × 1016, so we place it there rather than at a point. Below it the loss-monotone direction is unreliable, through incoherent scatter (Group-DRO) or a coherent reversal (DoReMi-style, all three seeds up).
Vetting a new reweighter: run it beside Stratified and Group-DRO on a provenance-split pool at the Intermediate budget or above, over three seeds, about 1.1 × 1017 FLOPs in all. A share that falls with Group-DRO's, as tightly, marks a loss-monotone policy.
§ 05

Loss is not value

What separates the families is how much the update's score leans on a domain's own loss. All six methods share one exponentiated-gradient update on the simplex and differ only in the score they feed it:

Shared update

A loss-monotone score is forced toward the highest-loss provenance; an interaction or gradient score is not, because its move depends on cross-domain transfer, not on ℓ. Which provenance is higher-loss is an empirical input, and it fixes the direction of mass flow:

A-priori prediction
Per-provenance validation loss, Aioli run
Fig. 3Line is the class mean over 9 domains and 3 seeds, band the per-class min to max. At the base checkpoint the classes are tied; the gap opens inside the first adaptation round and settles near one nat, consistent with forgetting on the small fixed real shards. The split is systematic, not strict: Github (real, 2.90) sits below Self-Instruct (synthetic, 4.03).

Decompose a domain's loss as , irreducible entropy plus a reducible gap. A raw-loss method cannot separate the two terms, so it chases whichever provenance carries the higher loss whether or not that loss is reducible. DoReMi's excess-loss centering is the standard fix, which is why it moves less than raw Group-DRO. DoGE's inner product carries a self term , so its near-uniform behaviour holds because cross terms dominate on this pool, not by construction.

§ 06

Where the mass goes

It is not provenance the loss-movers chase but loss. Group-DRO piles onto high-loss real domains (Pile-CC 0.164, OpenWebText2 0.150) and starves easy synthetic ones (Code-Alpaca 0.004), yet also trims low-loss real domains such as Github and ArXiv. Aioli stays in a tight band around uniform.

Final per-domain weight relative to uniform (1/18), Full tier
Fig. 4Mean over seeds. Cell colour is log2(w / uniform), clipped to ¼× … 4×; grey is uniform. Hover a cell for the weight ± std.
§ 07

Held-out test loss, hedged

Aggregate test loss over 18 domains, Full tier
Mean ± 1σ (bar), seeds 0, 1, 2 (rings); lower is better

Aioli has the lowest mean, 3.60 nats, and beats DoGE and Skill-It on every seed. Against Stratified, DoReMi-style and Group-DRO it lands within seed scatter, and no aggressive loss-mover pulls ahead of the uniform control by more than that scatter.

With n = 3 this describes average behaviour, not an ordering: models are scored after online training rather than a clean static retrain, so mixture choice is confounded with online dynamics and with Aioli's probe-driven trajectory. The runner's --static-eval path removes that confound.

§ 08

Budget and faithfulness

FLOPs per run ≈ 6ND, N total parameters, D = max_steps · batch · ctx. Counting total rather than non-embedding parameters overshoots, so read each budget as a ceiling.
TierModelctxbatchmax stepstokens / runFLOPs / runGroup-DRO shareDoReMi share

DoGE and Skill-It add one-time estimation surcharges over the Full-tier base: 2.37 × 1017 (+3%) and 2.87 × 1017 (+25%). The full headline study, 6 methods × 3 seeds, is about 4.3 × 1018 FLOPs, each run roughly 435× under a 1020-per-run ceiling. All runs used one NVIDIA A100 (80 GB).

Honest labels. DoGE and Skill-It refer only to faithful implementations; the older validation-loss heuristics live behind a flag and are never given those names.
MethodStatusDeviations from the original
Aiolifaithful portthe Full tier uses a stateful EMA (0.1) path instead of the reference's stateless reset
DoGEfaithful, Alg. 1online single model, per-round estimation, mean-abs score scaling
Skill-Itfaithful, Alg. 1continual-pretraining regime, uniform init, mean-abs scaling
DoReMi-styleapproximatevalidation-shard loss; no max(0,·) clip or c = 10−3 smoothing; frozen pretrained reference
Group-DROapproximatehigh-loss upweighting kept; joint min-max model step omitted
Stratifiedexactuniform control, p = 1/k
§ 09

Reproduce it

One self-contained runner regenerates every reported number. Data streams from the Hugging Face Hub; models continue from pretrained Pythia checkpoints.

Setup
git clone https://github.com/mixing-floor/compute-floor.git
cd compute-floor
pip install -r requirements.txt
Smoke test · Minimal tier (--config SUPER_LIGHT), minutes
python Code/run_experiment.py --config SUPER_LIGHT --seeds 0 --scenario S2_9R_9S --eval-test --out-dir runs/main
Reported run · Full tier (--config HEAVY), 3 seeds
python Code/run_experiment.py --config HEAVY --seeds 0 1 2 --scenario S2_9R_9S --eval-test --out-dir runs/main
python Code/visualize_results.py --out-dir runs/main --fig-dir runs/figures

Compare your runs/main against the released Results/main. Signal-level diagnostics that log DoGE's gradient inner products, Skill-It's influence graph and Aioli's interaction blocks are in Results/diagnostics.

§ 10

What this does not show

Behaviour, not quality

The protocol records which mixture each policy picks. No downstream task metric is reported, and no method is claimed to train a better model.

Online, not static

Test loss is measured after online training, which tangles mixture choice with training dynamics. A static retrain on each final mixture would settle it.

One scenario, small n

Scenario S2 at three seeds. The split and L2 ordering are tight apart from DoGE, which wanders from 0.453 to 0.557 between seeds.

Provenance ⇄ template

The synthetic class is defined by instruction format, so provenance and template are confounded. Whether the split survives past 160M, or vanishes on an all-real pool as predicted, is open.

§ 11

Citation

@inproceedings{salehi2026computefloor,
  title     = {The Compute Floor of Data Mixing: When Loss-Based
               Reweighters Start to Drain Easy Synthetic Text},
  author    = {Salehi, Shayan and Mohammadi, Iman},
  booktitle = {MOSS Workshop at the Conference on Language Modeling (COLM)},
  year      = {2026},
  url       = {https://mixing-floor.github.io}
}
Code and released results · Zenodo
@software{salehi2026computefloor_code,
  title     = {compute-floor: code and results for The Compute
               Floor of Data Mixing},
  author    = {Salehi, Shayan and Mohammadi, Iman},
  year      = {2026},
  version   = {v1.0-camera-ready},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23618257},
  url       = {https://doi.org/10.5281/zenodo.23618257}
}

10.5281/zenodo.23618257 pins the exact release used in the paper; 10.5281/zenodo.23618256 always resolves to the latest version.