The original SDSIE proposal treated inference like a bicycle transmission: watch the model’s confidence, and shift between cheap and expensive compute depending on how hard the current token actually is.
It’s a nice idea. It’s also, as of right now, not the idea that’s actually delivering the numbers. This post is about what I found when I finally stopped trusting the shifting mechanism and started testing it properly — and about the much simpler thing that turned out to work instead.
A Gear That Computes But Never Shifts
Here’s the uncomfortable finding, stated plainly: for most of this project’s life, the “smart gear” was computing a perfectly correct decision — and then nobody was listening to it.
Across the codebase, the entropy-gated controller was, without exception, correct. Given a token’s output distribution, it computed Shannon entropy, ran it through Schmitt-trigger hysteresis, and returned the right active/inactive decision every time. That part checked out under direct review, matched the governing equations, and reproduced identically across independent implementations.
What it was wired to was the problem. The decision got logged. It did not get executed on.
This pattern showed up in five separate places, written across different sessions and different tools:
- sdsie_server.py — computes
k_proposal, writes it to telemetry, then runs the identical fp16 forward pass regardless of what the gate said. - harness_step1.py — the original precision-gating prototype. Same shape: gear computed and logged every step, actual compute unaffected.
- Four hand-rolled speculative engines in the root folder (
sdsie_cuda_graph_engine.py,sdsie_speculative_fast.py,sdsie_static_speculative.py,sdsie_unified_engine.py) — real scout/target speculation exists in these, but each carries its own scout-cache or target-cache correctness bug, independent of the gating question.
The clearest single piece of evidence for this: I ran a threshold sweep across nine different clutch configurations, from one that almost never activates to one that’s active a third of the time. Throughput across all nine sat in a flat 51.1–51.8 tok/s band. If the gate were actually changing what got computed, that number should have moved. It didn’t move at all. That’s not a subtle statistical result — that’s a smoking gun.
The problem: Every part was tested in isolation, and mathematically sound, but failed in real testing.
Before I could say anything about whether adaptive gearing works, I had to actually re-build it — script by script — and then test it properly.
INT4/FP16 precision-gating (Resolution Gear). First real clutch-to-kernel wiring — GatedLinear switches computation between calibrated INT4 and FP16 per-layer, driven by the live entropy signal.
The result: On-the-fly quantization kept token fidelity on 2 of 3 prompts at a single layer quantization, but already on the second layer the degradation was just as bad as a full 32 layer quantization.
This was when I made the call to not waste more of my time on this specific system, for now. This does not mean that adaptive mechanism is dead as a concept, both AdaEDL (Qualcomm) and SGLang’s own adaptive-length mechanism both report real gains from this general approach.
Saved By a Sidenote
As I was sifting through the debris of the project I initially had full confidence in, I sorted out the scripts that actually worked from those that worked on paper only. I created the base repository for the SDSIE – Fixed k=5 Speculative Decoding. What initially was meant as a possible side project, saved the project as a whole.
No gate, no threshold, no hysteresis. Just draft, verify, repeat. Instead of the full 8B model getting dragged through memorybusses for each token, the smaller 1B model proposes 5 tokens to be used in the output. The 8B model will check the tokens and either verify or dismiss them.
If the tokens are verified, they will be used, and the energy consumption and speed is fully based off the 1B model. If the proposed tokens are dismissed, the 8B model will create new tokens.
Initial testing showed very positive results, but I was not going to make the same mistake as with the original SDSIE. This is the part of the project that survived every round of scrutiny I threw at it, and I threw a lot at it, because I’d already been burned once by trusting numbers that turned out not to reproduce.
None of the initial numbers were correct. Getting there required finding and fixing several measurement bugs that would otherwise have quietly biased every result.
The part I actually want to talk about: how many ways this can go wrong quietly
A partial list of what was actually wrong, underneath results that looked totally plausible at the time:
- The baseline and the treatment weren’t warmed up the same way. One script ran five throwaway generations before timing anything; its “matched” sibling ran zero. Two scripts claiming to measure the same thing, under different thermal conditions, the whole time.
- A genuine GPU thermal transient, not a bug in the code at all — power draw climbs for the first 40 seconds or so of sustained load and doesn’t fully settle for close to two minutes. Any benchmark shorter than that is measuring a GPU that hasn’t finished warming up, and whichever condition happens to run first in the loop eats the cold part for free.
- Tokenization mismatch — one script fed the raw prompt string to the model, its sibling used the chat template. Not a huge effect, but not zero, and not something you’d catch by staring at the output.
- A convergence check that could never converge. Power-stability logic waited for consecutive readings to agree — impossible when the workload alternates between two genuinely different power levels by design. Fixed by checking convergence per-workload-label instead of pooled.
- A temperature check pooled across two workloads that settle at different steady-state temperatures. Same shape of bug as above, in a different variable — I fixed it for power and missed that I needed to fix it for temperature too. I found a solution and rewrote the convergence logic properly.
- A tolerance tighter than the sensor’s own noise floor. 1.5W absolute tolerance, on a signal with 2.9–6.8W of real sample-to-sample noise. No amount of waiting fixes a threshold that was never physically achievable. The fix was to set the tolerance to a more reasonable 1.5%.
None of these are exotic. Every one of them is the kind of thing that produces a number that looks completely reasonable, gets written into a README, and sits there being wrong until someone goes looking specifically for the thing that would prove it wrong. That’s the actual work of this kind of research — not the clever mechanism, the unglamorous business of making sure the ruler you’re measuring with isn’t warped.
Every Fix Moved The Number Up, Never Down
The boring, cynical explanation — that a project fishing for good numbers just keeps finding upward errors because that’s what it’s looking for.
What actually happened is closer to the opposite: every fix removed a source of penalty that had been dragging both baseline and speculative measurements down together, and the speculative condition — which has more headroom to lose thermally, given it’s driving two resident models instead of one — had more to gain back. The numbers went up because the earlier numbers were conservative by accident, not generous by design.
Does Fixed k=5 Speculative Decoding Work?
Fixed K=5 speculative decoding is the shippable result: lossless, reproducible across independently-written scripts down to the decimal on accept rate, cross-validated against a completely separate benchmark harness, with a converged, drift-checked measurement methodology behind every number.
And the numbers? Full testing once all bugs were fixed shows 100% token-for-token fidelity against the FP16 baseline on all three prompts, every run.
| Prompt | Speedup vs FP16 | Energy reduction | Accept rate |
| Poem (low determinism) | +8.4% | −32.7% | 42.5% |
| Physics explanation | +24.0% | −39.2% | 51.7% |
| Code (high determinism) | +83.5% | −60.8% | 85.4% |
Does the above results feels counterintuitive?
The three prompts span a range of token-level predictability for the model, not difficulty for a person — and that distinction matters, because the results above can look backwards at first glance. Code is often considered a more cognitively demanding task than free-form poetry, yet it gets the largest speedup (85.4% accept rate) while poetry gets the smallest (42.5%).
Speculative decoding’s accept rate depends on how sharply peaked the model’s next-token probability distribution is, which is a different axis from how hard a task is for a person.
Code has strict, learned syntactic structure — matching brackets, indentation rules, a constrained vocabulary of keywords and common idioms — so at most token positions there is essentially one syntactically valid continuation, or a very small set of them. The scout’s greedy guess is usually right, and draft windows survive largely intact regardless of how logically demanding the underlying code is to write.
Open-ended creative writing has close to the opposite property at the token level: at nearly every position there are many equally plausible word choices (synonyms, alternate phrasings, meter- and rhyme-driven word selection for the Chant Royal form used here), so the model’s distribution is flatter and the scout is wrong more often — again, independent of how hard the poem actually is to compose.
Physics explanation falls in between: more lexical variety than code, but far less than open verse, and its 51.7% accept rate lands squarely between the other two.
This is precisely the axis speculative decoding’s speedup is sensitive to (see the accept-rate figure above), which is why the prompts were chosen to span it deliberately — not to span perceived task difficulty.
Fixed K=5 speculative decoding is: lossless, reproducible across independently-written scripts down to the decimal on accept rate, cross-validated against a completely separate benchmark harness, with a converged, drift-checked measurement methodology behind every number in the table above.
The real and verifiable truth is that Fixed k=5 Speculative Decoding is up to 83.5% faster, uses up to 60.8% less energy and is 100% accurate compared to a full fp16 model.
What does this mean in real production then?
At SDSIE project website is a calculator, which use the lower end of the results above. This is the reality for a small, 512 GPU datacenter.
This is how it will scale, if testing in real datacenters holds up.
| Scale | Energy Saved | OPEX Saved | CO2 Avoided |
|---|---|---|---|
| 10,000 GPUs | 19.0 GWh/yr | $2.28M/yr | 4,921 metric tons/yr |
| 50,000 GPUs | 94.9 GWh/yr | $11.4M/yr | 24,604 metric tons/yr |
| 100,000 GPUs | 189.7 GWh/yr | $22.8M/yr | 49,209 metric tons/yr |
Nvidia alone has shipped roughly 22 million H100-equivalents of AI compute since it started tracking — about 73% of the global total, more than Google, AMD, Amazon, and Huawei combined (Epoch AI, Aug 2026).
The clearer number, though, is electricity. The IEA reports that data centers worldwide consumed roughly 415 TWh in 2024 — about 1.5% of global electricity use — and projects that figure to nearly double by 2030. That 415 TWh works out to an average continuous draw of about 47 GW, running 24/7, every day of the year — equivalent to roughly 47 average-sized (1 GW) nuclear reactors operating continuously. By 2030, on the IEA’s own trajectory, that’s closer to 95 reactors’ worth. (Note: this is all data centers, not narrowly AI GPU clusters — but AI buildout is the driver IEA cites for the projected doubling).
Sign up for our newsletter, and follow the Fixed K=5 Speculative Decoding project!