Alex Open Research Wiki

Action100M: A Large-scale Video Action Dataset

6 min read

  • Important
  • Skimmed

Action100M: A Large-scale Video Action Dataset

Source

Status And Credibility

Action100M was released in January 2026 by an overlapping Meta FAIR/HKUST/Sorbonne team and appears in the CVPR 2026 Workshops proceedings for EgoVis, pages 8832-8842. The official artifacts include a Meta GitHub repository, a public Hugging Face preview, a paper, schema documentation, and a sample annotation tree. This is credible current dataset evidence, although the labels are fully automated pseudo-labels and the complete dataset is not publicly exposed: the released Hugging Face repository is explicitly a 10% preview.

Core Contribution

Action100M scales hierarchical open-vocabulary action annotation over instructional videos. Its paper reports:

  • 1,199,096 HowTo100M videos covering about 14.6 years;
  • 147,092,653 temporally localized hierarchical segments;
  • approximately 21.27 billion words across four generated text fields;
  • approximately 205 GB for all annotations, metadata, and Trees of Captions.

The dataset is an annotation and metadata layer, not a video mirror. The public schema identifies source videos by YouTube ID and contains no video-binary column.

Public Release

The official Hugging Face repository publishes a 10% preview by source-video rows:

PropertyPublic value
Rows120,000
Splittrain only
Parquet shards120
Compressed/download bytes39,173,609,923
Decoded dataset bytes86,153,267,321
AccessPublic, ungated
LicenseFAIR Noncommercial Research License

One row represents one source video and contains video_uid, metadata, and nodes.

Annotation Schema

Each node has temporal and hierarchy fields; model-output fields are nullable. In particular, the final gpt structure may be absent:

video_uid
metadata
  title, description, view_count, like_count
  duration, upload_date
  transcript[{time, text}]
nodes[]
  node_id, parent_id, level
  start, end
  plm_action, plm_caption
  llama3_caption
  gpt?.summary.{brief,detailed}
  gpt?.action.{brief,detailed,actor}

The hierarchy makes temporal scale explicit: lower levels correspond to finer, shorter actions; higher levels summarize coarser procedures. Every node has temporal bounds and parent linkage. The paper says nodes shorter than four seconds are excluded from final GPT-OSS aggregation, but they remain as tree nodes. In the repository’s official one-video sample, all 639 nodes shorter than four seconds have gpt: null; among 278 longer nodes, 262 have non-empty GPT fields and 16 are null. That sample is useful for understanding nullability, not for estimating corpus-wide missingness.

Automated Annotation Pipeline

flowchart LR
  Video[HowTo100M video] --> Emb[V-JEPA 2 frame embeddings]
  Emb --> Cluster[hierarchical agglomerative clustering]
  Cluster --> Nodes[temporal segment tree]
  Nodes --> PLM[PerceptionLM-3B segment captions]
  Nodes --> Llama[Llama-3.2-Vision leaf-frame captions]
  PLM --> Tree[Tree of Captions]
  Llama --> Tree
  Tree --> GPT[GPT-OSS-120B, three Self-Refine rounds]
  GPT --> Labels[brief/detailed action + actor + brief/detailed summary]

The paper reports that 3.23% of segments receive an N/A action label, usually for intros, ads, subscription reminders, or non-action content. It also reports strong duration imbalance: 64% of segments are 0-3 seconds, 23.8% are 3-10 seconds, 10.2% are 10-60 seconds, and about 2% exceed one minute.

Relationship To VLWM

The Action100M paper explicitly says that the pipeline extends and improves the procedure introduced in VLWM. This is the annotation dataset most directly connected to the VLWM line, but it is not the missing VLWM training corpus.

PropertyVLWM data described in 2025 paperAction100M release
Source mixtureHowTo100M, COIN, CrossTask, YouCook2, EgoExo4D, EPIC-KITCHENS-100, NaturalReasoningHowTo100M only
Main unitGoal-plan trajectoryHierarchical video segment node
Structured targetsGoal, interpretation, action, state changeAction, actor, summary
Explicit world-state change after actionYesNo
Public availabilityEmpty HF placeholder10% HF preview

Action100M is therefore best understood as a later released hierarchical action/caption substrate from the same research line, not as proof that the exact VLWM examples or target format were released.

Evidence And Utility

The paper trains VL-JEPA on Action100M and reports consistent scaling on zero-shot action recognition plus text-to-video retrieval. Its controlled data ablation reports that LLM-aggregated brief action descriptions outperform direct PerceptionLM-3B pseudo-labels in the matched setup. These results support the usefulness of hierarchical aggregation for representation learning; they do not validate every individual annotation as correct.

The most relevant use for world models is multi-scale procedural structure: one can learn that an observed video segment corresponds to a fine action, a coarser subtask, and a broader activity. The dataset does not by itself provide explicit before/after physical state, a controllable action channel, reward, outcome, or counterfactual transition.

Data And License Caveats

  • The public artifact covers 120,000 source-video rows, about 10% of 1.2M videos; it is not established as exactly 10% of all hierarchical segments or decoded bytes.
  • The reported 147,092,653 count covers hierarchical segments/nodes; the final GPT fields are nullable, and nodes shorter than four seconds are retained without GPT aggregation in the official sample.
  • Labels are generated automatically by V-JEPA 2, PerceptionLM-3B, Llama-3.2-Vision-11B, and GPT-OSS-120B; they are not human ground truth.
  • The authors use global captions, local captions, metadata, and ASR to reduce hallucination, but source-model errors can still propagate coherently through Self-Refine.
  • The data is long-tailed and redundant; common actions such as speaking to camera are overrepresented.
  • Only 72% of full-dataset videos have retrieved ASR transcripts.
  • The source videos are external YouTube/HowTo100M assets; deletion, availability, copyright, and platform terms remain separate from the annotation release.
  • The FAIR Noncommercial Research License prohibits commercial use and applies redistribution conditions.
  • video_uid is an external identifier and potential provenance handle; it is not an immutable video snapshot.

Foundation TSFM Relevance

Agenda slotVerdictEvidenceMissing pieces
Data diversity and long tailadjacentProvides broad procedural-video coverage with open-vocabulary hierarchical action labels at very large scale.It is video/text rather than numeric time series; action frequency is highly imbalanced and only 10% is public.
Event streams and hierarchyadjacentTree nodes encode temporal intervals, parents, and abstraction levels from atomic motions to procedures.No irregular numeric observations, typed event ontology, or channel/state schema.
Control and counterfactualswarningAction labels describe observed physical activity and may help learn action semantics.Observed labels are not logged control inputs or interventions; no next-state outcomes or counterfactual trajectories are provided.
Benchmark levelinsufficient evidenceVL-JEPA transfer experiments show representation utility.Action100M is a training corpus, not a closed-loop world-model benchmark; individual-label accuracy and intervention utility remain unverified.

Open Questions

  • Will the full 147M-segment annotation corpus be released, and under what exact access mechanism?
  • How accurate are action and actor fields under human audits stratified by duration, domain, hierarchy level, and rare concepts?
  • Can the hierarchy support action anticipation or next-state prediction rather than only action recognition and retrieval?
  • Which fields remain useful after removing title, description, or ASR context to test visual grounding and leakage?
  • Can the same pipeline generate typed before/action/after transitions suitable for action-conditioned world models?

Graph View

Backlinks