AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild

Source

• Raw Markdown: paper_anymo-2026.md
• PDF: paper_anymo-2026.pdf
• Official project-page and code-status snapshots: preserved as project_page_snapshot.md and official_code_readme.md in the raw paper directory; these auxiliary artifacts are not part of the public wiki artifact allowlist.
• arXiv v2: https://arxiv.org/abs/2605.22715v2
• DOI: https://doi.org/10.48550/arXiv.2605.22715
• Project: https://baiyuchen.com/project/AnyMo
• Official repository: https://github.com/Breezelled/AnyMo
• AnyMo Bench: https://huggingface.co/datasets/CRUISEResearchGroup/AnyMo-Bench

Status and credibility

AnyMo is a recent arXiv preprint by Baiyu Chen, Zechen Li, Wilson Wongso, Lihuan Li, Xiachong Lin, Hao Xue, Benjamin Tag, and Flora Salim. Version 1 was submitted on 2026-05-21 and version 2 on 2026-05-23. The paper is licensed CC BY 4.0 and lists UNSW Sydney plus HKUST/HKUST Guangzhou affiliations. No peer-reviewed venue was identified on arXiv or the official project page as of 2026-08-10.

The paper is credible as a current research candidate from an established wearable-sensing group and includes extensive methods, ablations, 14 zero-shot downstream datasets, retrieval and captioning tests, and a public benchmark release. Reproducibility remains incomplete: the official GitHub repository at commit 0f0d989f59f517658feb8c651a2bbc48e88eaa9c contained only a README saying the code was being prepared, with no code, weights, license file, tags, or releases.

Core claim

Wearable setup shift is structured by body geometry. Instead of treating sensor location, orientation, and hardware as arbitrary nuisance variation, AnyMo synthesizes plausible IMU signals over a dense body surface, trains a body-graph encoder to recover full-motion latents from sparse setup views, discretizes those latents into compact motion tokens, and aligns the tokens with language.

The resulting interface supports open-vocabulary motion understanding from sparse wearable sensors: zero-shot activity recognition, bidirectional IMU-text retrieval, and motion captioning.

Method

flowchart LR
  N[Nymeria mesh, skeleton, motion] --> S[2,374 candidate surface placements over 23 body segments]
  S --> I[physics-grounded acceleration + angular velocity with device-noise priors]
  I --> V[paired full setup views]
  V --> M[mask all but 1-5 visible sensor nodes]
  M --> G[ST-GCN encoder + Transformer predictor]
  G --> P[time-preserving setup-stable latent]
  P --> Q[PQ-VAE: 2 x 2,048-entry codebooks]
  Q --> T[interleaved full-body IMU tokens]
  T --> L[Qwen2.5-0.5B motion-language model]
  L --> O[recognition / retrieval / captioning]

Geometry-aware IMU simulation

The paper uses Nymeria’s mesh and skeleton motion to construct local sensor frames at selected body-surface vertices. Acceleration and angular velocity are computed in the simulated sensor frame, with random mounting rotation and two hardware-style noise priors estimated from quiet real Nymeria IMU windows. The exported synthetic archive contains 831 eligible samples and 2,374 candidate placements across 23 body segments.

The simulation is intended to generate many plausible views of the same underlying body motion while preserving a known kinematic topology. Real head and wrist IMU streams are not used as representation-pretraining targets; they define noise priors and the held-out sim-to-real protocol.

Masked cross-view predictive contrastive learning

For each motion window, AnyMo independently samples two full body-graph views. Sparse observations retain one to five segment nodes and replace the rest with a learned mask. A shared ten-block ST-GCN encodes time-preserving sequence latents, and a six-layer Transformer predicts the opposite full-view latent.

For predicted and target latent sequences, the paper averages cosine similarity over time and uses a symmetric stop-gradient InfoNCE objective:

$${\mathcal{L}}_{\mathrm{MCVPCL}}={\mathcal{L}}_{A\to B}+{\mathcal{L}}_{B\to A}.$$

This objective combines sparse-to-full recovery with instance discrimination without collapsing each window into one clip embedding.

Full-body IMU tokenization

The frozen ST-GCN latent is discretized by a product-quantized VAE with two codebooks of 2,048 entries, a 128-dimensional bottleneck, and 64-dimensional vectors per codebook. Two code indices are interleaved at each latent timestep. A typical five-second, 300-frame input produces 75 latent steps and 150 IMU tokens plus boundary tokens.

The paper reports 99.27% and 99.90% code usage, codebook perplexities of 1,285.7 and 1,098.1, and a 0.61% exact sequence-collision rate. These are tokenizer-utilization diagnostics; they do not by themselves establish preservation of dense body state.

Motion-language alignment

AnyMo extends Qwen2.5-0.5B with 4,096 IMU code tokens. Each code token is mapped through its learned quantizer vector and projected into the LLM hidden space rather than initialized as an unrelated vocabulary row. Training combines:

• causal IMU-token language modeling;
• IMU-text contrastive alignment for narration retrieval;
• supervised activity-label contrastive learning;
• instruction generation for captions and multiple-choice recognition.

The text-aligned Nymeria export is subject-disjoint from five held-out subjects. The paper reports 164,387 pretraining windows, 159,098 labeled instruction-tuning windows expanded to 986,322 language-model rows, and another 986,322 paired IMU-text contrastive rows. Narration paraphrases and label construction use GPT-OSS-120B and GPT-5.4-family classifiers with expert review.

Evidence and results

Zero-shot activity recognition

AnyMo is evaluated on 14 downstream datasets that are not used for training. The reported averages are:

Metric	AnyMo	Author-marked improvement
Accuracy	35.7	+11.7%
Macro-F1	29.5	+11.6%
R@2	57.5	+22.6%

AnyMo is strongest on average, but not on every dataset or metric. The evaluation combines datasets with different class counts, body placements, sampling regimes, and upstream split conventions, so the average should not be read as one homogeneous leaderboard.

IMU-text retrieval and captioning

On held-out real Nymeria IMU streams, the paper reports relative MRR improvements of 15.9% for IMU-to-text retrieval and 28.6% for text-to-IMU retrieval. On wearable motion captioning, it reports an 18.8% relative improvement in BERT-F1. The captioning system also transfers to out-of-domain EgoExo4D head-mounted IMU according to author-reported qualitative and quantitative tests.

Ablations

The paper reports that geometry-aware surface simulation, cross-view predictive contrastive pretraining, product-quantized tokens, narration contrastive tuning, label contrastive tuning, and the multiple-choice instruction branch each contribute to the final results. These ablations support the composite recipe, but executable reproduction is blocked until code and weights are released.

AnyMo Bench

The paper also introduces AnyMo Bench, a public Nymeria-derived benchmark with Fine150 and Core50 label spaces. It separates unseen-subject generalization from same-position cross-device transfer. The AnyMo model itself is not the main benchmark baseline table; the table trains DeepConvLSTM, MantisV2, and COMODO on the benchmark split.

The benchmark card, split tables, and artifact metadata contain a row-count tension: 154,695 windows are claimed, while 123,874 train plus 30,965 test rows and dataset_info.json imply 154,839 exported rows per configuration.

Interpretation for the wiki

AnyMo is most relevant as a setup-robust multivariate representation and multimodal interface, not as a forecasting or world-model result. It makes three useful ideas concrete:

1. Known nuisance structure can be simulated. Body-surface geometry provides a stronger augmentation prior than arbitrary channel transforms.
2. Sparse observations can target a structured latent state. The encoder predicts a full-body latent from partial sensor graphs while preserving temporal order.
3. Numeric tokens can interface with language. Product-quantized IMU representations support retrieval and generation without serializing every raw value as text.

The world-model boundary remains clear: no action, control input, intervention, reward, counterfactual, or closed-loop rollout is modeled.

Limitations

• Preprint evidence with no identified peer-reviewed venue.
• Public benchmark but no released code, weights, executable environment, or repository license at the pinned audit date.
• Real supervision, annotation, and central evaluation depend heavily on Nymeria.
• Geometry and noise simulation assumptions may not cover unseen sensor physics, loose mounting, drift, failure, or body-model mismatch.
• Sparse-to-full latent recovery is tested mainly through recognition, retrieval, and captioning rather than dense state reconstruction or calibrated uncertainty.
• Automated narration and label pipelines can introduce semantic bias despite verification and expert review.
• Aggregate zero-shot averages mix heterogeneous datasets and protocols.
• AnyMo Bench’s headline and exported row counts disagree.
• No passive forecasting, intervention modeling, action-conditioned transition prediction, or control evaluation.

Foundation TSFM relevance

Agenda slot	Verdict	Evidence	Missing pieces
Data diversity and long tail	partially closes	Dense body-surface simulation creates many setup views; AnyMo-180 and Fine150/Core50 expand the activity vocabulary.	Need broader devices, placements, populations, failure modes, and non-Nymeria real supervision.
Self-supervised objective	partially closes	Masked cross-view predictive contrastive learning targets full-view time-preserving latents from one to five sensors.	Need executable artifacts and dense-state probes beyond task transfer.
Representation quality	partially closes	Strong author-reported recognition/retrieval/captioning transfer and non-collapsed token usage.	Need reconstruction/state accessibility, calibration, layer/readout audits, and independent reproduction.
Tokenization	partially closes	Compact product-quantized IMU tokens preserve sequence order and connect to an LLM.	Need rate-distortion, robustness, rare-motion retention, and serving-cost comparisons.
Native multivariate modeling	partially closes	Fixed 23-node body graph encodes cross-segment structure and sparse sensor visibility.	Not arbitrary-schema or high-dimensional multivariate modeling; topology is body-specific.
Language/context interface	partially closes	Body-location prompts, motion narrations, label text, retrieval, and caption generation share one model.	Need tests of corrupted/missing text, semantic grounding, and dense numeric fidelity.
Control and counterfactuals	insufficient evidence	None.	Add actions or interventions and evaluate transition prediction or closed-loop decision utility.

Links into the wiki

• AnyMo
• AnyMo Bench
• AnyMo Bench source
• Time-Series Classification Foundation Models
• Self-Supervised Representation Learning
• Synthetic Data For Time Series
• Unified Multimodal Models
• Latent Tokenization
• Time-Series Benchmark Hygiene
• Foundation Time-Series Model Research Agenda

Open questions

• Does geometry-aware synthetic setup coverage predict transfer to genuinely new device families, mounting looseness, and placement drift?
• Which dense body-state variables remain recoverable after product quantization?
• How much of the gain comes from geometry-aware simulation versus Nymeria scale, label curation, and motion-language instruction tuning?
• Do retrieval and captioning improvements survive incorrect body-location prompts or semantically ambiguous activities?
• When code and weights are released, do the reported cross-dataset results reproduce under pinned preprocessing and sampling protocols?