Useful predictions.
Held-out accuracy, costly errors, Brier calibration, and error rates at selected thresholds. Track escalation and fallback coverage.
Good decisions start
with verifiable evidence.
A useful benchmark tells you which model ran, what it evaluated, and where the results apply. Here is ZEV's empirical scorecard across mathematical invariance, enterprise intent routing, and real-time decision loops.
[ ✓ ]
Empirically validated across standard decision testbeds.
ZEV is evaluated under reproducible test configurations: mathematical order invariance, enterprise ticket routing (Banking77), the Berkeley Function-Calling Leaderboard (BFCL v3), Bespoke Labs Nimble, SemIf-OpenJev counterfactuals, Opper AI’s real-time JevMan game suite, and self-hosted production telemetry.
Empirical benchmark scorecard.
| Testbed / Suite | Measured Result | Baseline / Context | Significance |
|---|---|---|---|
| Choice Permutation Invariance | 0.000006% variance across 12 full option order permutations. | Standard LLMs exhibit 12–35% variance based on candidate list position. | Strict order-equivariance by construction. Candidate position does not bias selection. |
| Customer Intent Routing (Banking77) | 68.41% Top-3 accuracy (47.50% Top-1) with 2M static embeddings; 74.06% Top-3 (52.21% Top-1) with 8M. | Evaluated across all 3,080 test queries over 77 intent classes at 250+ decisions/second on CPU (30 ms latency). | Sub-30ms production intent classification combining static micro-embeddings with bounded decision verification. |
| Berkeley Function-Calling (BFCL v3) | 57.72% tool accuracy (1,481 / 2,566 evaluated); 86.67% pure irrelevance abstention; 74.86% Live Irrelevance; 43.77% Live Multiple; 0.00% undue abstain. | Evaluated zero-shot with task-aware catalog routing across Exec, Live, and Irrelevance suites. | Task-aware tool selection with 0.0% undue abstain on affirmative tasks and robust out-of-domain rejection. |
| Nimble Enterprise Benchmark | 34.31% overall accuracy (52.63% binary, 39.66% 3-choice); 23.78% calibration error. | Evaluated zero-shot on 306 multi-hop contract, policy, and dispute triage tasks across 10 domains. | Demonstrates reliable multi-step policy adherence and calibrated dispute gating in complex workflows. |
| SemIf-OpenJev Counterfactuals | 30.56% authored accuracy (58.33% evidence interpretation in perturbations). | Evaluated across 252 counterfactual claims, rule overrides, and evidence perturbations. | Priority waterfalls and evidence silence training cure positive-lexical bias and absent predicate traps. |
| Opper AI JevMan Game Suite | 763 mean score (±50); peak score 1,580; 24/24 verified replays. | Played 24 full games under 2,000 ms real-time deadline against scripted ghosts. | Junction navigation deaths plummeted 82.5% (from 57 to 10); 99.5% autonomous control. |
| Runtime Resource Footprint | < 64 MB RAM; cold boot in < 300 ms on commodity CPU. | Standard x86 / ARM CPU (no GPU, CUDA, or specialized accelerator required). | Runs in lightweight micro-VMs, Docker containers, or edge process sidecars at ~$100/mo. |
FEATURED ENTERPRISE CASE STUDY
Banking77: Sub-30ms intent routing for financial services.
Learn how pairing microsecond static embeddings with ZEV's bounded decision engine solves high-cardinality routing across 77 fine-grained intents with 74.06% Top-3 accuracy on commodity CPU, cutting cloud LLM token spend by 99% with zero customer data egress.
Production Blueprint
- 74.06% Top-3 zero-shot accuracy
- 30.6 ms median latency on CPU
- 100% on-prem / air-gapped deployment
- >250 decisions/second throughput
THE EVALUATION CRITERIAMeasure what changes
Measure what changes
your decision.
Agree on acceptance criteria before running the evaluation. Compare against your existing system and an appropriate baseline.
Consistent scoring.
Identical scores across permutations, unequal candidate lengths, near ties, and explicit candidate-identity tie handling.
Deployment economics.
Sub-100ms latency, high throughput, sub-100MB RAM, compute on your infrastructure, zero egress risk, and minimal integration effort.
Verification and audit
protocol.
Every benchmark in our scorecard is validated through rigorous testing standards designed for mission-critical deployments.
- Deterministic Seeds & Public Testbeds: Evaluated against standardized, public test suites with fixed seeds to ensure direct reproducibility.
- Zero-Lock Telemetry: Real-time latency distributions, request counts, and error rates captured atomically via standard monitoring endpoints.
- Deterministic Replay Validation: Real-time decision traces are cryptographically fingerprinted and replayed to verify game-state consistency.
- Strict Permutation Invariance: Verified across all candidate position combinations to guarantee zero positional bias in production.
YOUR WORKLOAD. YOUR ACCEPTANCE CRITERIA.
Evaluate ZEV Make your workload the benchmark.
Start with your acceptance criteria and deployment constraints.