At Edgescale, we’re engineering the infrastructure that brings artificial intelligence into the real world. Our work powers AI in the places that keep society running — manufacturing floors, hospitals, utilities, transportation networks, and more. By bridging the gap between the cloud and the physical edge, we enable real-time intelligence where humans and machines work together.
Part 2 of a four-part series on the sovereign AI factory we built and run on our own hardware. Part 1 covered the architecture: independent agents, no orchestrator, every artifact hash-chained back to the idea it came from. This part is about the harder problem of testing and verifying.
Letting a model that writes code then be responsible for reporting whether or not the code works is kind of like letting a kid grade their own homework.
This is the part of agentic tooling I trust least, and I don’t think that’s a controversial position among people who have actually shipped code with it. Don’t get me wrong – code generation is genuinely good now. But verification is where the whole thing can fall apart, because the standard pattern is to ask the same class of system that produced the artifact whether the artifact is any good. You may get a confident answer, but that confidence may not correlate with whether or not the code actually ran as planned.
So when we built our Sovereign AI Factory, the verification piece got most of the design attention. We made three key decisions here, all focused on keeping the thing that builds separate from the thing that judges what’s built.
#1 – The tests are written from intent, never from the code
The test author agent gets exactly two inputs: the original idea, in the operator’s own words, and the command-line signature of the prototype as actually deployed. It never sees the source.
That constraint is critical. A test generated from the source it’s testing will pass. It has to. It was written by reading what the code does and then asserting that the code does that thing. You end up with a green suite that has verified a tautology, and the failure mode is invisible, because nothing looks wrong. Green is green.
Written from intent, the test asks a different question: did the thing successfully accomplish what the operator actually wanted to happen? Those two questions deviated more often than I expected going in. A prototype can be internally coherent, well structured, and a faithful implementation of a misreading of the request. Tests derived from the code will never catch that. Tests derived from the intent of the request catch it on the first run.
The second input matters as much as the first. The test author reads the deployed prototype’s real interface, not an interface someone intended it to have. So the tests are black-box in the strictest sense – they exercise the thing that exists, through the surface it actually exposes, against the outcome someone actually asked for.
# 2 – The sandbox has no GPU and no network
Here we made the decision that admitted prototypes get deployed into a hardened, isolated pod with zero GPU allocation and no network egress. That means prototypes can only do one thing, which is run.
The no-egress part is the obvious security posture. The zero-GPU part is the more interesting one, because of what it rules out: a prototype under test cannot call a model. Whatever the tests measure, they are measuring the prototype’s own behavior, not a model’s ability to talk its way through a test at runtime. If the prototype needed a model to look like it worked, then it doesn’t work.
#3 – Evidence is a precondition, not an input
The next decision we made is that test conductors must be deterministic. So the system takes each test suite, runs it against the live prototype, and records what happened. That record is what we treat as ground truth. Not a summary of what happened, not an assessment of whether it went well, but what actually happened.
Then the quality gate scores the prototype across four dimensions: clarity (can a human read and maintain it), correctness (does it do the right thing, as evidenced by the run), fit (is it self-contained and appropriate to the constrained box it has to run in), and usefulness (does it actually serve what the operator asked for, not just something adjacent to it). There’s a threshold and a floor for the overall score under every individual dimension. Then, the gate only returns an “accept” if the execution evidence also passed. The evidence is a precondition, sitting outside the scoring entirely.
If evidence were just another weighted input, a sufficiently strong showing on the other dimensions could outvote a failing test run. Elegant, well-documented, clearly-reasoned code that does not work could score its way through. Making evidence a gate rather than a factor means no amount of quality can compensate for not running. A model’s opinion about the code never gets to overrule the record of what the code did or did not do.
The security gate sits after that and automatically blocks anything scoring high/critical, which is a separate concern and mostly uninteresting until it fires. When it does fire, or when quality fails, the prototype goes to remediation with the actual failures attached – not a description of the problem, the ground-truth record of it – and gets three attempts before we stop.
What broke
Quite a bit. A few things are worth noting here, though, because each lesson is now baked into the design.
The first is the one most on the nose for a piece about testing: for a stretch, the factory was grading its own test harness instead of the prototypes. The execution pass rate sat under one percent, and it looked like the prototypes were mostly junk. They weren’t. When we pulled a sample of the failures apart by hand, essentially every one was the harness’s fault, not the code’s. The test author – kept strictly black-box – was calling prototypes through interfaces they never exposed: no required input flag, a flag name that didn’t exist, one command reused for three tests that each expected something different. And the runner was scoring broken plumbing as a failing prototype – a read-only working directory, a missing binary in the sandbox, a module that wasn’t installed, a pod that never even reached running.
None of that was the prototype. The fix had two halves. We gave the test author the prototype’s real command-line signature, so it exercised the interface that actually exists instead of one it guessed at – that’s why the CLI signature is the second input in decision #1, and it’s the whole reason black-box doesn’t mean blind. And we taught the runner to recognize an infrastructure fault and pull it out of the verdict: a missing binary is a broken sandbox, not a broken prototype.Pass rates stopped being fiction the moment we stopped blaming the code for the harness’s own mistakes.
The second was in the build-and-repair loop, and it was embarrassingly mundane. Certified prototypes stopped accumulating – the count sat flat for days while the pipeline looked busy the entire time. The build side was fine. The remediation agent was the problem: it was running on a model that truncated its own output on longer repair orders, so the fix instructions came back malformed, the repair silently did nothing, and the prototype burned through its three attempts and died – not because it couldn’t be fixed, but because the instructions to fix it never arrived intact. Nothing errored loudly. The queue just quietly failed to make progress. Moving that one agent to a model that held its output together got the count climbing again. The lesson wasn’t about that particular model. It was that a silent malformed-output failure buried deep in the loop looks exactly like “the work is just hard,” and for a while we read it as the latter.
The third: a handful of prototypes were poison pills. One specific malformed shape crashed the remediation agent outright, and because the loop was built to retry, it retried forever – the same prototype wedging a worker in an infinite loop while everything behind it waited. We added a dead-letter path, so a prototype that crashes the fixer gets quarantined instead of retried and the loop moves on. This is obvious in hindsight (actually, most of these are).
And one that was less a bug than a bad guess: we set the quality bar too high to start. The accept threshold was 8.0, and prototypes that passed 100% of their tests were getting turned away on score alone – working code, rejected for not being polished enough. We dropped the threshold to 7.0, kept the per-dimension floor, and re-scored the backlog against the new bar. The floor matters more than the ceiling here: a thing that runs and does what was asked shouldn’t die because a model thought the variable names could be better.
None of these were failures of the models writing code. They were failures in the machinery around the models – a verifier blaming the code for its own broken plumbing, a silent serialization bug, a missing escape hatch in a retry loop, a quality bar tuned by taste instead of by evidence. Which is the point of the whole series. The hard part was never generation – it was building something around a generation whose output you could actually trust.
Why any of this needs to be at the edge
Everything above could run in a data center. What we have today is deterministic execution evidence from a hardened sandbox running on the appliance itself. That’s the reason the factory can certify anything without a person in the loop. But, that’s not the hard part.
The harder part of this equation is that a model in a data center can review your code, reason about your architecture, and tell you what it expects to happen. But it cannot tell you what happened when that code ran against a specific piece of machinery, on a specific line, in a specific plant, with the sensor drift and the timing quirks and the twelve years of accumulated modifications that make that installation completely unique. That evidence only exists in the building, and any verification loop that needs it must run locally, in the same building.
So that’s why we built a Sovereign AI Factory that certifies against the record of execution rather than a judgment about the code. As our CTO, Marc, always says – this is AI bounded by physical reality. That boundary is critical, and testing against what actually happened is the only verification that survives contact with an operating plant.
In Part 3, I’ll talk about the artifact trail that this system produces. And I’ll also talk about why having a record that outlives any particular model is essential to any physical plant or operation.