Implicit Curriculum Hypothesis
Summary
The Implicit Curriculum Hypothesis is the claim that language-model pretraining acquires skills in a stable, compositional, and predictable order across model families and data mixtures. The linked paper operationalizes the hypothesis with ElementalTask: a suite of simple and composite exact-match tasks plus checkpoint trajectories and function-vector analyses.
Interface
- Main object: a diagnostic hypothesis about ordered skill emergence during pretraining.
- Task suite: 91 simple and composite tasks covering string operations, morphology, translation, retrieval, coreference, logic, reading-comprehension-style tasks, and arithmetic.
- Emergence measurement: first checkpoint where a task exceeds an absolute accuracy threshold.
- Representation measurement: residual-stream function vectors extracted from correctly answered prompts.
- Prediction interface: use task-representation similarity and kernel ridge regression to predict held-out composite task trajectories.
- Released artifacts: arXiv preprint, official ElementalTask GitHub repository, and HF
elemental-tasks/model-trajectoriesdataset.
Role In The Wiki
Use this entity when a page needs the caveat that smooth aggregate loss curves can hide ordered capability emergence. It is also the current local handle for capability-emergence diagnostics based on internal representation geometry.
For time-series and world-model work, the transfer is a measurement pattern rather than direct evidence: define interpretable latent-state, rare-regime, context-use, channel-coupling, event-stream, and action-conditioned rollout probes; log absolute-threshold emergence over checkpoints; and test whether internal representations predict future probe trajectories.