How much can language models memorize?

Source

• Raw Markdown: paper_language-model-memorization-capacity-2025.md
• PDF: paper_language-model-memorization-capacity-2025.pdf
• Preprint: arXiv 2505.24832v3
• ICML: ICML 2026 oral
• OpenReview: ICML 2026 forum
• Award: ICML 2026 Outstanding Paper Honorable Mention
• Author blog: John X. Morris, 2025-06-03
• Gonzo ML discussion: Telegram post 5721 (local extract at papers/language-model-memorization-capacity-2025/telegram-post-gonzo-ml-5721.md)
• Gonzo-linked review: ArXivIQ ICML 2026 review
• Reference checkpoint listed by Gonzo: rhysjones/gpt2-774M-fineweb-150B
• Local artifact audit: papers/language-model-memorization-capacity-2025/official_artifacts_metadata.json

The arXiv version is titled How much do language models memorize? The accepted ICML 2026 version changes do to can; this page uses the accepted title. The Hugging Face checkpoint linked by the Gonzo post is a third-party conversion of Karpathy’s llm.c GPT-2 774M FineWeb run, not an official checkpoint release by the paper authors. No verified official code repository was found at ingest time.

Status And Credibility

The paper was first submitted to arXiv on 2025-05-30 and last revised there as v3 on 2025-06-18. The official ICML site lists the updated paper as an ICML 2026 oral, and the ICML program chairs list it as an Outstanding Paper Honorable Mention. The author team spans FAIR at Meta, Google DeepMind, Cornell University, and NVIDIA.

This is high-credibility, tier-1 evidence for an information-theoretic and empirical account of GPT-style model capacity. Its strongest quantitative conclusions remain scoped to the tested architectures, data construction, compression estimator, reference models, training recipes, and a standard loss-based membership-inference attack.

Core Claim

The paper separates information stored about a particular training sample from information that captures the underlying data-generating process:

• unintended memorization is sample-specific information that the trained model adds beyond a reference model;
• generalization is information about reusable structure in the data distribution.

For a sample $x$, reference model $\theta$, and trained target model $\hat{\theta }$, the algorithmic definition is

$${\mathrm{mem}}_U^K(x,\theta ,\hat{\theta })=H^K(x\mid \theta )-H^K(x\mid \theta ,\hat{\theta }).$$

Exact Kolmogorov complexity is uncomputable, so the experiments approximate conditional description lengths with likelihood-based arithmetic coding. On uniformly random synthetic sequences, where reusable structure is deliberately removed, the measured plateau estimates how much sample-specific information the training procedure stores.

The main empirical result is an approximately linear capacity law for the tested GPT-style Transformers:

$$\mathrm{Capacity}(\hat{\theta })\approx \alpha |\hat{\theta }|,\alpha \approx 3.6\text{bits per parameter}.$$

This is an empirical lower bound under the tested training procedure, not a universal physical maximum for every architecture or optimizer.

Evidence

Question	Paper evidence	Calibrated reading
How much can the tested GPT-style models store?	Synthetic random-sequence experiments show a capacity plateau and fit about 3.64 bits per parameter in half precision.	A lower-bound estimate for the tested dense GPT family and saturated training setup.
Does parameter precision double usable storage?	The reported mean rises from 3.51 bits per parameter in bfloat16 to 3.83 in fp32.	Extra numerical bits are not converted one-for-one into measured sample storage.
What happens on real text?	Fully deduplicated 64-token FineWeb sequences show more unintended memorization in larger models and smaller datasets; rare-token and non-English sequences form a vulnerable tail.	Average behavior does not remove rare-sample risk.
Where does double descent appear?	Evaluation loss worsens near the data-to-capacity boundary and improves after dataset information exceeds measured model capacity.	Supports a capacity-sharing explanation in this controlled setting, not a universal proof of every double-descent mechanism.
Can capacity predict membership inference?	A sigmoid fit relates capacity-to-dataset ratio to F1 for a standard loss-based attack; targeted validation on GPT-2 125M and 1.5B is generally within about 1.5 F1 points.	Useful average-case scaling evidence for one attack family, not a privacy certificate.
Is extraction the same as membership inference?	Membership inference can remain high when exact suffix extraction is zero; on large deduplicated text datasets, train extraction approaches test extraction.	Extraction, membership inference, unintended memorization, and generalization MUST be reported separately.

Gonzo Claim Audit

The Telegram post is valuable discovery context, but several phrases are stronger than the paper supports.

Gonzo statement	Audit against the paper
Capacity is about 3.6 bits per parameter.	Supported with scope. The estimate is an empirical lower bound for tested GPT-style models, not an architecture-independent ceiling.
fp32 gives 3.83 versus 3.51 bits per parameter for bfloat16.	Supported. These are the paper’s reported means in the precision ablation.
The result confirms Tishby’s Information Bottleneck two-phase theory.	Not directly established. The paper studies sample-specific memorization, generalization, capacity saturation, and data-size regimes. It does not present its result as a direct test of Tishby’s representation-level Information Bottleneck dynamics.
The capacity boundary is where double descent appears.	Supported in the controlled experiments. The broader universal-mechanism wording should remain a hypothesis beyond these architectures and datasets.
More than 100 tokens per parameter statistically guarantees that membership inference is impossible.	Overstated. The fitted law predicts F1 near 0.5 for the paper’s standard loss-based average-case attack at contemporary token-to-parameter ratios. This is not a guarantee against all membership-inference attacks, rare-sample attacks, extraction, or future stronger attacks.
Capacity accounting can guarantee no confidential training examples leak.	Unsupported. The paper does not provide differential privacy or a worst-case leakage guarantee. Rare sequences remain more memorized, and average F1 at chance is not proof that every sample is protected.

Relation To Nearby Sources

Learning is Forgetting and this paper both use information and compression language, but they measure different objects:

• Learning is Forgetting probes checkpoint-time information in internal representations and argues for an expansion-to-compression trajectory.
• How much can language models memorize? estimates sample-specific information stored in final model parameters and varies dataset size relative to measured capacity.

They are complementary, not interchangeable evidence for one universal two-phase law.

Deep Learning is Not So Mysterious or Different adds another boundary: representational capacity is not the same as the complexity of the function selected by optimization. The current paper estimates how much random information a training procedure can store; it does not show that structured-data solutions use that whole capacity or that a larger model must generalize worse.

Why Diffusion Models Don’t Memorize studies a different axis: checkpoint time. Its generalization and memorization timescales should not be collapsed into this paper’s dataset-to-capacity ratio.

Diffusion Consistency RMT adds another distinct axis: dataset size relative to data dimension and covariance spectrum at a fixed diffusion-training recipe. Its memorization-to-renormalization transition and effective-noise mapping are not the same as this paper’s algorithmic-information capacity estimate, even though both use the language of memorization and a data-size boundary.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Scaling and efficiency	adjacent	Introduces a measured information-capacity axis in addition to parameters and tokens.	A time-series analogue needs a stable information unit across sampling rates, channels, tokenizers, event streams, and compression schemes.
Training dynamics	adjacent	Shows capacity saturation and a memorization-to-generalization tradeoff across model/data regimes.	Needs checkpoint-time multivariate time-series evidence and separation from representation-level Information Bottleneck claims.
Benchmark hygiene	warning	Shows that extraction, membership inference, and unintended memorization can disagree.	Time-series generators need attack-specific privacy, subsequence extraction, duplicate, rare-regime, and average-versus-worst-case reporting.
Latent-state preservation	warning	Finite capacity forces selective storage once a dataset no longer fits sample by sample.	The paper does not test whether compression preserves rare regimes, exogenous variables, event timing, control inputs, interventions, or decision-relevant latent state.
Control and counterfactuals	insufficient evidence	Capacity accounting could become one systems diagnostic.	No time series, trajectories, actions, control inputs, interventions, counterfactual prediction, or planning are evaluated.

Because its direct agenda contribution is adjacent, warning, or insufficient evidence, this source should not be added to the central Foundation TSFM coverage matrix.

Time-Series Transfer

The useful transfer is not 3.6 bits per parameter as a new TSFM constant. It is a measurement program:

1. define what information in a multivariate time series is sample-specific versus reusable system structure;
2. choose a reference model that captures the background dynamics without the target trajectory;
3. estimate conditional description length in a unit stable across channel count, sampling rate, tokenization, and missingness;
4. vary dataset information relative to model capacity;
5. separately test average forecast quality, rare-regime retention, event extraction, membership inference, and action-conditioned state prediction.

A time-series model can look well-generalized on average while memorizing unusual trajectories or erasing rare decision-critical states. Both failure modes matter.

Limitations And Gotchas

• Exact Kolmogorov complexity is uncomputable; likelihood-based compression is an estimator whose result depends on the coding family and reference model.
• The 3.6 bits-per-parameter figure is a lower-bound estimate from dense GPT-style Transformers trained to saturation, not a theorem for MoE, recurrent, state-space, diffusion, vision, or time-series models.
• The paper’s own appendix says the environment may not generalize to other datasets, architectures, or training setups.
• The membership result covers a standard loss-based attack and average F1. It is not differential privacy, a worst-case bound, or a guarantee against extraction and stronger attacks.
• Rare-token sequences remain disproportionately memorized even when average attacks weaken.
• The token-to-parameter threshold depends on tokenization and data entropy; raw token count is not a stable cross-domain information unit.
• The double-descent result is strong controlled evidence for a capacity-boundary interpretation, not proof that every observed double-descent curve has the same cause.
• No verified official code or author-released checkpoint was found at ingest time.

Links Into The Wiki

• Training Dynamics
• Time-Series Scaling And Efficiency
• Time-Series Benchmark Hygiene
• Learning is Forgetting
• LLMs as Noisy Channels
• Deep Learning is Not So Mysterious or Different
• Why Diffusion Models Don’t Memorize
• Diffusion Consistency RMT

Open Questions

• Can multivariate time-series model capacity be estimated in bits without making the answer an artifact of quantization, patching, channel count, or sampling rate?
• Which reference-model design separates reusable dynamics from sample-specific trajectory memory under non-stationarity?
• Do rare events, exogenous shocks, interventions, and unusual channel combinations remain membership-inference or extraction outliers after average attacks reach chance?
• Does a TSFM show a data-to-capacity double-descent boundary, and does that boundary align with improved latent-state probes or only average forecasting loss?
• Can model-capacity estimates predict when extra parameters improve rare-state preservation versus merely increase sample-specific memorization?
• How should privacy evaluation combine average loss-based membership inference with worst-case attacks, subsequence extraction, duplicates, and formal privacy guarantees?