# SGBS: SETT Governance Benchmark Series **URL:** \ **Category:** Projects & Demos\ **Tags:** ai, performance, developer-program, open-source, benchmark\ **Created:** [August 24, 2026, 6:28am UTC](https://devcommunity.amd.com/t/sgbs-sett-governance-benchmark-series/895 "2026-08-24T06:28:17Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Eduardo\_Daniel\_Vinal](https://avatars.discourse-cdn.com/v4/letter/e/b782af/32.png) [@Eduardo\_Daniel\_Vinal](https://devcommunity.amd.com/u/Eduardo_Daniel_Vinal)\ **Post date:** [August 24, 2026, 6:28am UTC](https://devcommunity.amd.com/t/sgbs-sett-governance-benchmark-series/895/1 "2026-08-24T06:28:17Z")
**SGBS-01: Deterministic Governance Boundary Benchmark** _ **Results are now available: benchmarking deterministic governance in SETT vs. LangGraph** _ A few days ago, after sharing SETT here, I received a very useful question from the community: What does SETT’s deterministic governance boundary actually provide beyond adding a deterministic policy layer to an existing agent framework? Rather than trying to answer that only conceptually, I decided to turn the question into an experiment. I have now completed and published **SGBS-01: Deterministic Governance Boundary Benchmark** , the first experiment in the SETT Governance Benchmark Series. The benchmark deliberately removes the LLM from the equation so that only authorization, execution integrity, traceability, fail-closed behavior, implementation footprint, and runtime overhead are measured. Three configurations were compared: ● **SETT native** ● **LangGraph baseline** , with no added governance, used only as a negative control ● **LangGraph + ad-hoc deterministic policy** , implementing the same frozen policy semantics manually in the graph The most interesting result is that the comparison did **not** produce a simple winner. Across the frozen 21-scenario correctness matrix: ● SETT: **0/10 BLOCK/REJECT escapes** ● LangGraph baseline: **9/10 escapes** ● LangGraph + policy: **0/10 escapes** SETT and the manually governed LangGraph implementation both achieved: ● **100% Boundary Integrity Rate** ● **100% fail-closed behavior under the injected policy failure** ● **100% normalized policy consistency** So an ad-hoc deterministic layer built correctly on LangGraph **can reproduce the same boundary behavior tested here**. That matters, because one of the goals of this experiment was specifically _not_ to assume that this architectural idea can only exist inside SETT. The differences appeared elsewhere. SETT enforced the tested routes through a structural governed submission mechanism, while the LangGraph + policy version required each relevant execution path to be intercepted explicitly. SETT was also the only tested configuration with a native hash-chained action-audit facility. There was a cost: governance was measurably slower. In the official v0.12.0 performance run: ● SETT mean latency: **784.8 µs** ● LangGraph + policy: **575.7 µs** ● LangGraph baseline: **394.4 µs** A follow-up single-patch ablation on SETT reduced its mean latency to **673.0 µs (-14.2%)**, reduced p99 by **25.5%** , and increased throughput by **18.8%** , without changing decision or routing semantics. The first chart shows the overall Stage 13 ablation. The patch reduced SETT’s latency while both LangGraph controls remained within normal run-to-run variation. ![Stage 13 ablation overview. SETT native mean steady-state action latency decreased from 784.8 µs to 673.0 µs (-14.2%) after the targeted patch, while the LangGraph baseline and LangGraph + policy controls remained within normal run-to-run variation (+1.0% and +1.5%).](https://global.discourse-cdn.com/amd/original/2X/e/e10cc21691b121b46cdf34f06fbc9770fd5061c1.png) I also broke the SETT result down by scenario family to check whether the overall improvement was being driven by one particular path. It wasn’t: every measured family improved, although the magnitude varied depending on the execution path. ![SETT native latency by scenario family. The Stage 13 patch improved mean latency across every measured scenario family, with reductions ranging from -6.9% to -33.5%. Values are the family-level aggregates reported in the SGBS-01 paper.](https://global.discourse-cdn.com/amd/original/2X/e/e7c5b2b96d9ad8fc1ce16015b842abfc5951cd53.png) The benchmark also found one real SETT v0.12.0 observability defect: an orphaned handler.failed trace event on the unknown-action path. It did not affect authorization or produce a boundary escape, but it was reproducible and is reported explicitly in the paper. The issue has since been corrected in SETT v0.13.1 and later. For me, the most useful conclusion from SGBS-01 is not that deterministic governance is impossible elsewhere, it clearly isn’t. The result is narrower: **The tested boundary can be reproduced manually, while SETT attempts to make that boundary a structural property of the framework and couples it with native audit evidence. That structural convenience currently comes with measurable implementation overhead.** That is a much more useful result than simply claiming architectural superiority. The complete methodology, limitations, raw-data references, results, and ablation analysis are available here: **SGBS-01 — Deterministic Governance Boundary Benchmark** > **[SGBS-01: Deterministic Governance Boundary Benchmark](https://zenodo.org/records/22065880)** > > This benchmark removes language-model inference and measures deterministic authorization, execution integrity, traceability, fail-closed behavior, implementation footprint, and runtime overhead across three configurations. This edition reports... Benchmark repository and frozen experimental artifacts: > **[GitHub - EduDanielV/sett-governance-benchmark-series at sgbs-01-v1.0.0](https://github.com/EduDanielV/sett-governance-benchmark-series/tree/sgbs-01-v1.0.0)** > > sgbs-01-v1.0.0 This is only the first experiment in the series. SGBS-01 intentionally contains no LLM, so no claim here extends to model behavior, natural-language ambiguity, or production security. Thank you again to the people who took the time to question the original architecture. That feedback directly influenced what I decided to measure next.