FactoryBench: Evaluating Industrial Machine Understanding

Source

• Raw Markdown: paper_factorybench-2026.md
• PDF: paper_factorybench-2026.pdf
• Preprint: https://arxiv.org/abs/2605.07675v1
• DOI: https://doi.org/10.48550/arXiv.2605.07675
• Official project page: https://www.forgis.com/papers/factorybench.html
• Official evaluation client: https://github.com/Forgis-Labs/FactoryBench
• Official generation repository: https://github.com/Forgis-Labs/FactoryBench_Generation
• Official Hugging Face: https://huggingface.co/datasets/Forgis/FactoryBench
• Raw dataset metadata snapshot: source.md

Status And Credibility

FactoryBench is an arXiv v1 preprint submitted on 2026-05-08 with public dataset and code artifacts. Unlike HEPA and FactoryNet, neither the canonical arXiv record nor the official project page currently claims an ICML workshop or main-conference acceptance for FactoryBench. It is included because Alex explicitly selected it and because the source is tightly aligned with the knowledge base’s benchmark agenda.

The paper is current and produced by the Forgis/ETH-led team behind the other two sources. Public release improves auditability, but there is no verified peer-reviewed venue status, no independent replication, and the Hugging Face card still carries stale double-blind metadata.

Core Claim

FactoryBench evaluates whether a model can progress from reading dense industrial telemetry to reasoning about interventions, counterfactuals, and remediation. It organizes approximately 71k Q&A items into four levels:

Level	Canonical question	Evaluation target
L1 State	What is the machine doing now?	Observation/state interpretation
L2 Intervention	What follows from an event or control intervention now?	Immediate consequence reasoning
L3 Counterfactual	What would have happened under a different past?	Alternative-history reasoning
L4 Decision	What should the operator do?	Root-cause diagnosis and remediation

This hierarchy is closer to the foundation-TSFM target than a forecasting-only leaderboard because it separates state, consequences, alternative futures, and decisions.

Data And Generation Pipeline

flowchart LR
  FW[FactoryWave telemetry] --> N[normalized episodes]
  A[AURSAD] --> N
  V[voraus-AD] --> N
  N --> T[21 typed template families]
  KG[error-code and remediation knowledge graph] --> T
  T --> Q[70,918 QA items]
  Q --> D[deterministic scoring for L1-L3]
  Q --> J[three-judge LLM scoring for L4]

FactoryBench is grounded in approximately 15k normalized episodes:

• 8,983 FactoryWave episodes from UR3 and KUKA KR10 robots;
• 4,094 AURSAD episodes in the paper’s normalized/relabelled accounting;
• 2,122 voraus-AD episodes.

FactoryWave records more than 100 telemetry channels at 125 Hz for UR3 and 83 Hz for KUKA KR10, including setpoints/control inputs, effort, feedback observations, context, task phase, and fault metadata. The Q&A generator uses 21 manually authored template families and five answer formats: single-choice, multi-choice, ranking, numerical/tensor, and free-form.

Public Release

The Hugging Face release contains 70,918 items:

Level	Train	Validation	Test	Total
L1	12,674	1,338	1,309	15,321
L2	33,311	3,428	3,487	40,226
L3	2,353	265	321	2,939
L4	9,949	1,251	1,232	12,432
Total	58,287	6,282	6,349	70,918

At ingest time, the Hugging Face Dataset Viewer failed with a string-to-null schema cast, although the JSONL files were present and direct data_files= loading was documented. The card also retained an anonymous double-blind citation and tagged the two upstream dataset arXiv IDs rather than FactoryBench’s own paper ID. Those are release-hygiene issues, not evidence that the benchmark is unavailable.

The paper also says that a private 15% subset is held out at the physical-episode level for contamination checks and future evaluation. The public 70,918-item split and any private evaluation score must therefore remain separate; the current public repository does not expose the private episode data, answers, or rubric annotations.

Causal Semantics Audit

The four level names should not be accepted uncritically as proof of causal validity.

L1 state

Many questions test extraction, phase localization, numerical prediction, or fault recognition from serialized telemetry. Good performance establishes signal comprehension under the benchmark representation, not a persistent latent-state model.

L2 intervention

L2 includes control interventions, disturbances, and events. A collision or payload change is not the same object as an action chosen by an agent. Each item should identify whether the manipulated variable is a control input, intervention, exogenous event, or observed fault.

L3 counterfactual

Physical counterfactual ground truth is approximate. The authors repeat tasks with controlled conditions, inject a fault at a fixed time, and select the run whose pre-injection segment minimizes signature-kernel MMD against the baseline. The post-injection divergence is a useful empirical proxy, but two physical runs do not share an identical pre-intervention state. L3 therefore measures reasoning plus proxy quality.

L4 decision

L4 ground truth combines manufacturer error documentation, expert-authored protocols, and a knowledge graph. Free-form answers are scored by the median of three LLM judges. Results depend on judge models, prompt/rubric version, agreement, cost, and the benchmark’s protocol coverage.

Evidence And Results

The paper evaluates six frontier LLMs zero-shot and reports chance-corrected accuracy:

Model	L1	L2	L3	L4
Claude Sonnet 4.6	46.8	47.1	45.9	4.3
Qwen3-235B	36.0	35.6	43.6	4.4
Mistral Large 3	34.6	31.7	36.3	5.4
GPT-5.1	30.9	30.0	31.7	17.7
DeepSeek V3.2	25.0	29.1	28.5	7.6
Qwen3-4B	21.8	27.5	28.8	2.9

No model exceeds 50% on L1-L3 or 18% on L4. The rank reversal is informative: Claude leads structured signal reasoning but collapses on decisions, while GPT-5.1 leads L4 despite weaker L1-L3 scores. The authors interpret this as a separation between signal comprehension and protocol retrieval.

This is a baseline, not a ceiling. The evaluated systems process serialized telemetry directly. Specialized time-series encoders, tools, calculators, retrieval over manuals, supervised adaptation, and action-conditioned world models are not ruled out by the result.

Limitations

• L3 counterfactuals are approximate matched physical reruns, not exact do-operator samples.
• Faults are atomic, fast, and selected from a closed catalogue; production failures are often compound, gradual, and weakly labeled.
• L2 mixes events and interventions unless each item is audited by variable semantics.
• L4 depends on LLM-as-judge scoring and curated protocol coverage.
• Template-generated questions can share surface structure; split leakage and episode/template overlap need explicit audits.
• The level distribution is highly imbalanced, especially L2 versus L3.
• The release’s viewer and citation metadata are stale or broken.
• The paper’s MIT/commercial-use claim conflicts with CC BY-NC 4.0 on the Hugging Face card and CC BY-NC-SA 4.0 in the official code repositories.
• A private 15% episode-level evaluation slice is not publicly auditable.
• The benchmark does not test a model that continuously maintains state and evaluates candidate control sequences in closed loop.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Benchmark level	partially closes	Direct industrial telemetry benchmark separates state, intervention, counterfactual, and decision questions.	Need episode/template leakage audits, stronger tools/specialized baselines, hidden tests, and longitudinal state evaluation.
Causal structure and counterfactuals	partially closes	Explicit L2/L3 tasks and physically repeated fault interventions expose alternative-outcome reasoning.	L3 is an approximate MMD-matched proxy; L2 must separate actions, interventions, events, and disturbances.
Context and reasoning interface	partially closes	Questions combine numeric telemetry, task/machine context, typed prompts, and protocol knowledge.	Need corrupted-context controls and evidence that models use telemetry rather than template or document shortcuts.
Control utility	adjacent	L4 asks for remediation and optimization recommendations.	No closed-loop execution, candidate-control rollout, safety shield, reward/cost, or intervention outcome scoring.
Native multivariate encoding	warning	More than 100 telemetry channels expose the failure of direct LLM processing on dense numeric signals.	Evaluate specialized multivariate encoders and hybrid encoder-LLM systems under matched input information.
Public reproducibility	partially closes	Dataset, client, generation code, scoring, and counts are public.	Fix viewer schema, stale citation metadata, pin revisions, and publish independent reproductions.

Why It Matters For This Wiki

FactoryBench operationalizes the question at the center of the knowledge base: does a model only forecast observations, or does it maintain enough state to explain the system, reason about interventions, and support decisions? It is especially valuable as a benchmark specification for an integrated FactoryNet + HEPA model.

The important caveat is that Q&A competence is not itself a world model. A language model can retrieve a remediation protocol without understanding the signal, while a numerical dynamics model can predict consequences without producing a readable answer. A credible system should separate and then combine telemetry encoding, latent-state maintenance, action-conditioned rollout, protocol retrieval, and decision scoring.

Links Into The Wiki

• FactoryBench entity
• FactoryNet
• HEPA
• Time-Series Benchmark Hygiene
• Causal Time Series
• Robotics Time-Series Modeling
• Time-Series Foundation Models
• Foundation Time-Series Model Research Agenda

Open Questions

• How much of L1-L3 can be solved from template form, labels, or local numerical extraction without learning persistent machine state?
• Are train/validation/test splits disjoint by physical episode, intervention pair, fault family, and template instantiation?
• What specialized encoder + LLM architecture closes the gap between numeric signal comprehension and protocol retrieval?
• Can a FactoryNet-pretrained latent model answer L2/L3 by simulating alternative control inputs instead of generating text directly?
• How should L4 score decision quality, safety, cost, and executability after the proposed action is applied?
• Can the benchmark add closed-loop episodes where the model’s chosen remediation changes the future telemetry and is scored against outcome?