# FactoryBench Dataset Metadata Snapshot

Fetched: 2026-07-21

## Canonical Sources

- Paper: https://arxiv.org/abs/2605.07675v1
- Official project page: https://www.forgis.com/papers/factorybench.html
- Official evaluation client: https://github.com/Forgis-Labs/FactoryBench
- Official generation/evaluation repository: https://github.com/Forgis-Labs/FactoryBench_Generation
- Official Hugging Face dataset: https://huggingface.co/datasets/Forgis/FactoryBench

## Dataset Type

FactoryBench is a question-answering benchmark over industrial robotic multivariate telemetry. Its 21 template families instantiate five answer formats across four levels:

1. **State:** interpret the current machine observation/state.
2. **Intervention:** predict the effect of a current event or control intervention.
3. **Counterfactual:** reason about an alternative past.
4. **Decision:** diagnose root cause and recommend remediation or optimization steps.

The benchmark is grounded in approximately 15k normalized episodes from FactoryWave, AURSAD, and voraus-AD.

## Public Release Snapshot

The official Hugging Face repository at revision `5c0c7e4c55ac6df61e6d5aad8c406d4f38439372` contains 1.05 GB of JSONL Q&A files and reports 70,918 items:

| Level | Train | Validation | Test | Total |
|---|---:|---:|---:|---:|
| L1 State | 12,674 | 1,338 | 1,309 | 15,321 |
| L2 Intervention | 33,311 | 3,428 | 3,487 | 40,226 |
| L3 Counterfactual | 2,353 | 265 | 321 | 2,939 |
| L4 Decision | 9,949 | 1,251 | 1,232 | 12,432 |
| **Total** | **58,287** | **6,282** | **6,349** | **70,918** |

At fetch time, the Hugging Face Dataset Viewer failed to generate the dataset because one field was inferred as `null` in one shard and `string` in another. The JSONL files remain present, and the official README demonstrates direct `data_files=` loading. The viewer failure SHOULD be treated as an artifact-health issue, not evidence that the release is absent.

The paper additionally says that a private 15% subset is held out at the physical-episode level for contamination checks and future evaluation. The public 70,918-item split and any private evaluation results therefore MUST be reported separately; the current public artifact does not expose the private episode data, gold answers, or rubric annotations.

## Underlying Telemetry

- FactoryWave: 8,983 UR3/KUKA KR10 episodes at 125/83 Hz across pick-and-place, screwdriving, and peg-in-hole, with 27 injected fault types.
- AURSAD: 4,094 episodes in the FactoryBench paper's relabelled/normalized accounting.
- voraus-AD: 2,122 episodes.

FactoryWave records more than 100 channels spanning setpoints/control inputs, effort, feedback observations, task phase, context, and fault metadata.

## Causal And Decision Semantics

FactoryBench is closer to the foundation-TSFM target than ordinary forecasting benchmarks because it explicitly separates state interpretation, intervention effects, counterfactual reasoning, and decisions. The evidence boundary is important:

- L2 mixes events, disturbances, and control interventions; an item SHOULD NOT be interpreted as a clean causal intervention unless the manipulated variable and timing are explicit.
- L3 physical counterfactual ground truth is approximate. The paper repeats runs with controlled conditions, selects a pre-injection match by signature-kernel MMD, and treats the post-injection divergence as a proxy for a do-operation. It is not an exact shared-state counterfactual.
- L4 free-form items are scored by a three-judge LLM ensemble. Results depend on judge models, rubric version, prompt, cost, and agreement; they are not deterministic in the same sense as L1-L3.

## Paper-Reported Baseline Results

In zero-shot evaluation of six frontier LLMs, no model exceeds 50% chance-corrected accuracy on L1-L3 or 18% on L4. Claude Sonnet 4.6 leads L1-L3 at 46.8%, 47.1%, and 45.9%; GPT-5.1 leads L4 at 17.7% while scoring 30.9%, 30.0%, and 31.7% on L1-L3.

These results measure direct LLM processing of serialized telemetry and benchmark prompts. They do not establish the ceiling for specialized time-series encoders, tools, retrieval, fine-tuning, or action-conditioned world models.

## Metadata And Release Caveats

- The Hugging Face card still contains an anonymous double-blind citation and lists arXiv IDs for AURSAD and voraus-AD rather than the FactoryBench paper ID. The named arXiv v1 and official GitHub repositories supersede that stale citation metadata.
- License statements conflict: the paper says FactoryBench and FactoryWave are MIT and available for commercial use, the Hugging Face card says CC BY-NC 4.0, and the official GitHub repositories say CC BY-NC-SA 4.0. No whole-release reuse claim is safe without maintainer clarification and a pinned artifact.
- The dataset is imbalanced across levels, especially L2 versus L3.
- Faults are atomic, fast, and drawn from a closed catalogue; production faults are often compound, gradual, and partially observed.
- License terms differ between the dataset and code repositories; see `license.md`.