The AI systems lab for unpredictable inference demand, from kernels to cloud

AI demand has become unpredictable.

A chatbot request has a predictable shape: one prompt, one answer. An agent’s doesn’t. It plans, branches into sub-tasks, retries when it fails, calls tools and waits on them, and decides its next step only after it sees the result. A single request can become one model call or a hundred, and no one knows which until it runs.

Today’s inference infrastructure was built for predictable load. It sizes capacity for the peak and pins models to GPUs. Under uncertain demand, that design fails both ways. Provision for the worst case and most of the silicon sits idle. Provision for the average and requests queue, stall, or get dropped. Between every call, accelerators fill with high-frequency, millisecond-scale bubbles of underutilized silicon and compute. We call this dark compute. This isn’t a flaw in one chip. It’s structural, and it will exist in every chip design to come.

And this is only the beginning. As AI moves toward more general intelligence, agents will reason longer, spawn other agents, and act more on their own. Uncertainty won’t be an edge case of AI compute. It will be its foundation.

OpenInfer is the systems lab building inference infrastructure for unpredictable compute demand. We go after idle bubbles at every layer of the stack, on any silicon, and turn that dark compute into AI goodput.

AI that runs everywhere, for everyone.

AI will be a central part of our lives, from drug discovery to assistants that operate on our behalf. For that vision to come true, AI has to be accessible to everyone at a fraction of today’s cost, and it should run where it needs to, not just in the homogeneous cloud.

We see a world where inference runs on any system, from cloud to neoclouds to on-prem and devices, with maximum efficiency.

Built for high-frequency workload change.

The industry has focused on incremental fixes that optimize along a single dimension: making inference faster while it’s running. We look at an additional dimension: utilizing all the bubbles of compute in between those inferences.

Inference is becoming unpredictable. At OpenInfer, we systematically look at every layer of the stack to use every compute cycle for high-quality reasoning.

We’re pushing system boundaries to handle non-deterministic workloads, much as hypervisors and operating systems were built to do for general-purpose computing. We make heterogeneous silicon work together to drive higher goodput, and we use compute cycles on any surface, from cloud to on-prem to edge.

Loom

The first result from our systems lab is the Inference OS, which turns hardware overprovisioning into what we call oversubscription. We built a product around it and launched it: Loom. Here are the numbers from live, real-time operation.