SGBS-01: Deterministic Governance Boundary Benchmark
Results are now available: benchmarking deterministic governance in SETT vs. LangGraph
A few days ago, after sharing SETT here, I received a very useful question from the community:
What does SETT’s deterministic governance boundary actually provide beyond adding a deterministic policy layer to an existing agent framework?
Rather than trying to answer that only conceptually, I decided to turn the question into an experiment.
I have now completed and published SGBS-01: Deterministic Governance Boundary Benchmark, the first experiment in the SETT Governance Benchmark Series.
The benchmark deliberately removes the LLM from the equation so that only authorization, execution integrity, traceability, fail-closed behavior, implementation footprint, and runtime overhead are measured.
Three configurations were compared:
● SETT native
● LangGraph baseline, with no added governance, used only as a negative control
● LangGraph + ad-hoc deterministic policy, implementing the same frozen policy semantics manually in the graph
The most interesting result is that the comparison did not produce a simple winner.
Across the frozen 21-scenario correctness matrix:
● SETT: 0/10 BLOCK/REJECT escapes
● LangGraph baseline: 9/10 escapes
● LangGraph + policy: 0/10 escapes
SETT and the manually governed LangGraph implementation both achieved:
● 100% Boundary Integrity Rate
● 100% fail-closed behavior under the injected policy failure
● 100% normalized policy consistency
So an ad-hoc deterministic layer built correctly on LangGraph can reproduce the same boundary behavior tested here.
That matters, because one of the goals of this experiment was specifically not to assume that this architectural idea can only exist inside SETT.
The differences appeared elsewhere.
SETT enforced the tested routes through a structural governed submission mechanism, while the LangGraph + policy version required each relevant execution path to be intercepted explicitly.
SETT was also the only tested configuration with a native hash-chained action-audit facility.
There was a cost: governance was measurably slower.
In the official v0.12.0 performance run:
● SETT mean latency: 784.8 µs
● LangGraph + policy: 575.7 µs
● LangGraph baseline: 394.4 µs
A follow-up single-patch ablation on SETT reduced its mean latency to 673.0 µs (-14.2%), reduced p99 by 25.5%, and increased throughput by 18.8%, without changing decision or routing semantics.
The first chart shows the overall Stage 13 ablation. The patch reduced SETT’s latency while both LangGraph controls remained within normal run-to-run variation.
I also broke the SETT result down by scenario family to check whether the overall improvement was being driven by one particular path. It wasn’t: every measured family improved, although the magnitude varied depending on the execution path.
The benchmark also found one real SETT v0.12.0 observability defect: an orphaned handler.failed trace event on the unknown-action path. It did not affect authorization or produce a boundary escape, but it was reproducible and is reported explicitly in the paper. The issue has since been corrected in SETT v0.13.1 and later.
For me, the most useful conclusion from SGBS-01 is not that deterministic governance is impossible elsewhere, it clearly isn’t.
The result is narrower:
The tested boundary can be reproduced manually, while SETT attempts to make that boundary a structural property of the framework and couples it with native audit evidence. That structural convenience currently comes with measurable implementation overhead.
That is a much more useful result than simply claiming architectural superiority.
The complete methodology, limitations, raw-data references, results, and ablation analysis are available here:
SGBS-01 — Deterministic Governance Boundary Benchmark
Benchmark repository and frozen experimental artifacts:
This is only the first experiment in the series. SGBS-01 intentionally contains no LLM, so no claim here extends to model behavior, natural-language ambiguity, or production security.
Thank you again to the people who took the time to question the original architecture. That feedback directly influenced what I decided to measure next.