Kog thinks GPUs still have a lot of inference headroom

French startup Kog is making a blunt bet: standard datacenter GPUs still leave too much inference performance on the table, and software can win some of it back.

That’s the idea behind KIE, or Kog Inference Engine. The company says it can deliver up to 30x faster LLM inference by digging into GPU behavior instead of waiting for new silicon. It’s a big claim, and the market seems willing to listen. Kog says its May tech preview brought in about 200 tangible business leads.

The timing makes sense. Inference, not training, is where a lot of the money is headed. People want models that answer quickly enough to fit into coding workflows, customer support, product generation, and agent pipelines where latency turns straight into cost.

Why speed matters

The clearest early pull is software engineering. That tracks.

Anyone who’s used coding agents knows the pain. You hand off a task, walk away, and wait. Sometimes for a long time. Anthropic already charges a premium for that with Claude Fast Mode, which costs more for lower latency. That’s a useful signal. If one of the biggest model vendors can charge directly for speed, the demand is real.

Kog is aiming at exactly that kind of customer. Not hobbyists. Teams running models inside production workflows, where waiting 30 seconds instead of five minutes changes the economics. The same logic applies to prompt-to-app tools, game generation, and other systems where every extra minute cuts throughput.

There’s still a limit here. Kog’s founder, Gaël Delalleau, says many prospective customers aren’t ready to fine-tune smaller models. They want the bigger systems, and they want them faster. That helps explain the opportunity. It also means Kog can’t live on toy benchmarks for long.

The strong demo, and why it’s not enough

Kog’s public demo showed 3,000 tokens per second per request, which is a serious number. The catch is that the demo used Laneformer 2B, a small model with roughly 2 billion parameters that’s now open sourced.

That matters.

A 2B model is much easier to push hard than the large frontier models most buyers care about. Smaller models have less memory pressure, simpler execution paths, and fewer bottlenecks when you’re trying to squeeze throughput from a GPU. If you want to show that your system can beat standard inference stacks, this is a sensible place to start. If you want to show it works on the larger models customers actually deploy, the bar gets much higher.

Kog knows that. Delalleau says the company has shifted toward accelerating larger models because that’s where demand is. That’s the right move, and the hard one.

Large-model inference is not just about raw compute. Memory bandwidth, kernel efficiency, scheduling, cache behavior, and decoding all matter. The bigger the model, the more each of those problems shows up. An optimization that looks excellent on a 2B model can fall apart once the model is 10x or 50x larger.

Kog’s bet on deep GPU work

Kog isn’t pretending that part is easy.

Delalleau talks about understanding the “laws of physics” and the “laws of the GPU,” which sounds theatrical until you hear the method. The team is working at a very low level, using assembly, binary analysis, and hardware-specific tuning. Each new GPU generation takes weeks or months of work.

That’s expensive. It’s also awkward for a startup that wants to scale quickly. With a team of 11, Kog can’t support every chip. It has to pick its battles.

That choice says a lot about the company’s thesis. Newer GPUs still have enough memory bandwidth and architectural headroom that software can matter a lot if you’re willing to get close to the metal. A lot of inference folks think GPUs are already near the practical limit for decoding. Kog is arguing the software stack on top of them is usually the real bottleneck.

There’s some history behind that view. CUDA performance has always depended on careful kernel work, memory layout, and scheduling. Kog is pushing that logic down into inference itself, not just training libraries or general-purpose runtimes.

How it stacks up against other inference plays

Kog isn’t the only company attacking inference from the software side.

Another French startup, ZML, recently released hardware-agnostic software that bypasses Nvidia’s CUDA to support faster inference across multiple chip types. That’s a broader compatibility play. Kog sounds narrower, more obsessive, and probably more brittle. It’s closer to the kind of systems work you’d expect from Stanford’s Hazy Research group than from a typical startup product team.

That comparison matters. Hazy-style work tends to produce elegant systems that outperform generic stacks in specific conditions. It also tends to come with sharp constraints. Generalizing those wins across a fragmented hardware market is the hard part.

And the market is fragmented. AMD MI300X, Nvidia H200, future Nvidia parts, whatever comes next. Each generation shifts the bottlenecks a little. A company trying to wring performance from each one has to keep re-learning the machine. That slows scale.

Europe’s sovereignty angle isn’t just noise

Kog also fits neatly into Europe’s current obsession with AI sovereignty.

It’s backed by France’s Bpifrance and the French Tech 2030 program, and it has support from Scaleway. That gives it the kind of institutional backing European AI startups often need if they’re going to compete with U.S. chip and model giants.

There’s a practical side to that, too. If Europe wants more control over where inference runs, what chips it runs on, and how much value stays local, software that makes existing hardware go further is attractive. Buying new accelerators is expensive. Getting more out of the ones already in the rack buys time.

Still, sovereignty doesn’t rescue weak software. The product has to work. Funding and policy support only go so far if the benchmark claims don’t turn into deployments.

The real test is larger models

Kog says it plans to show its first major model running at 10x speed in September, and it hopes that will help it show customer traction and raise a Series A. That’s the moment to watch.

If Kog can show meaningful gains on large models, on real enterprise workloads, with acceptable operational complexity, it’ll have something rare: a software layer that extends the life and utility of expensive GPUs. That matters for anyone running model-serving infrastructure at scale.

If it can’t, it risks becoming another deep-tech story with an impressive demo and a hard road to production. There are plenty of those in inference infrastructure. The gap between a striking benchmark and a system customers can trust is usually ugly and technical. It lives in tokenizer behavior, batch shaping, memory pressure, and the other unglamorous details that never make it into a launch post.

Kog’s pitch makes sense because inference is a systems problem again. The easy gains are mostly gone. What’s left is work close to the hardware, where one good kernel can change the economics and one bad assumption can sink the plan.

That’s where Kog has planted itself. Now it has to prove the GPU still has room to breathe.