FactoryBench
Summary
FactoryBench is a 70,918-item Q&A benchmark over industrial robotic multivariate telemetry. It separates four capability levels: state interpretation, intervention reasoning, counterfactual reasoning, and engineering decisions.
Benchmark Contract
- Data: approximately 15k normalized episodes from FactoryWave, AURSAD, and voraus-AD.
- Levels: L1 state, L2 intervention, L3 counterfactual, L4 decision.
- Formats: single-choice, multi-choice, ranking, numerical/tensor, and free-form.
- Scoring: deterministic for L1-L3; three-judge LLM rubric for L4.
- Public release: train/validation/test JSONL files and evaluation/generation code.
Official Artifacts
- Preprint: https://arxiv.org/abs/2605.07675v1
- Official project page: https://www.forgis.com/papers/factorybench.html
- Official evaluation client: https://github.com/Forgis-Labs/FactoryBench
- Official generation repository: https://github.com/Forgis-Labs/FactoryBench_Generation
- Official Hugging Face: https://huggingface.co/datasets/Forgis/FactoryBench
- Raw dataset metadata: source.md
- Raw paper Markdown: paper_factorybench-2026.md
Role In The Wiki
FactoryBench is a benchmark specification for the gap between dense signal comprehension and operational decision support. It is especially useful for evaluating an integrated FactoryNet data substrate and HEPA-style latent event model.
The benchmark should not be treated as direct evidence that a Q&A model has learned an action-conditioned world model. A model can retrieve protocols without understanding telemetry, and a dynamics model can understand transitions without emitting free-form remediation text.
Relation To Foundation TSFM Agenda
Use the source-level agenda mapping in FactoryBench rather than duplicating verdict rows here.
At the entity level, FactoryBench partially closes the benchmark taxonomy gap by separating state, intervention, counterfactual, and decision questions. It still needs persistent-state, specialized-encoder, leakage, exact-intervention, and closed-loop outcome protocols.
Caveats
- No verified ICML acceptance status; current status is arXiv preprint.
- L3 uses approximate MMD-matched physical reruns rather than exact counterfactual twins.
- L4 depends on LLM judges and curated manufacturer/expert protocol coverage.
- Faults are atomic and closed-catalogue rather than compound, gradual production failures.
- The Hugging Face viewer currently fails and the card contains stale anonymous citation metadata.
- The paper claims MIT/commercial use, while current data/code artifacts state incompatible non-commercial licenses.
- The paper retains a private 15% episode-level evaluation subset.