How we evaluate models — and why picking one is harder than it looks

There's a question we get asked a lot, in different forms: which model are you using? And there's an instinct in the answer — that picking a model is a thing you do once, the way you pick a database or a hosting provider. Choose the best one, lock it in, move on.

It isn't like that at all. Model selection is a moving target, and the job is closer to continuously curating a team than picking a tool. New models drop every few weeks. Existing ones get quietly updated. Customer expectations shift. Pricing changes. Inference providers have bad days. The "best model" for our product in March is rarely the best model in August, and the way we know is by running the same open evaluations over and over, watching what changes, and listening to the people building on Ullbek every day. We evaluate every model against four axes — the dimensions we think actually matter for our product. The post below is what those axes are, and what we look for on each.

Four axes. A scannable framework. The detail is below.

The Numbers

Every evaluation starts here, because we owe our customers a good answer to: are you getting a fair price for what you're getting? A model that produces beautiful work but is too expensive to run means our customers can't build with it. A model that's dirt-cheap but produces junk means their site looks like junk. We're optimising a curve, not picking a point on it. And this curve keeps moving — almost every frontier model has dropped its price in the last eighteen months. The cheap-tier model from six months ago is now the expensive tier, and there's a new model below it. We re-run the math constantly, because the difference goes straight to the customer.

The Taste

Once a model passes the price/performance bar, we move to the test that actually matters for our product: give it a one-sentence brief and see what it builds. This is where most models fail. They can write code. They can spit out a layout. What they often can't do is make a decision — choose a typographic scale that feels right, restrain the palette to three colours instead of seven, leave the right amount of negative space, write copy that sounds like a person wrote it instead of a content calendar. A model that produces a working site but a forgettable one isn't useful to us. So we feed every candidate the same kind of prompt a real customer would write. Vague, aesthetic, emotional. Then we look at the output. Does it have taste? Did it invent a coherent business with a coherent voice, or did it slap placeholder text on a generic grid?

And the voice has to shift too. We run the same model against a handful of very different briefs, and look at how it handles the vibe of each. Not whether it followed the instructions literally — whether it captured the essence. Three examples we run constantly:

A model that's great at one of these can be a disaster at another. The dark bar is tempting to over-design. The yoga studio is tempting to over-fill with stock photos. The photographer's site is where most models lose the thread. A model that gets all three right, in their own distinct ways, is rare. That's what we're hunting for.

And then there's the layer that doesn't fit on a spreadsheet — the one that honestly matters most to us long-term. When you're working with a model, going back and forth, does it feel like a person you'd want to work with? Some models just do what you say. They're fast, they're obedient, and they produce output. But they never push back. They never ask a clarifying question. They never say actually, are you sure about that? Other models have what we'd call presence. They ask follow-up questions that show they've read what you wrote. They have a point of view. They suggest directions you hadn't considered. They tell you when something is a bad idea and explain why. A model that just agrees with everything you say is a worse collaborator than one that has a taste of its own. We weight this heavily, because our customers are going to spend hours with whatever model they pick, and the difference between a junior assistant and a sharp colleague is enormous.

The Range

A model that's brilliant at producing a hero section can still be the wrong model for our product, because our product isn't a hero section. Our customers ask our models to do a wide range of things, and the model has to be good across the range:

A model can be in the top three on any one of these tasks and be middle-of-the-pack overall. We're not picking the best model for one job; we're picking the best model for the whole job.

The Plumbing

None of the above matters if the model isn't reliably available when a customer needs it. This is the part of the job that never makes it into a launch announcement, and it's the part that quietly eats your week if you ignore it. Inference providers have bad days. They rate-limit you unexpectedly. They have regional outages. They quietly change their pricing or their model names. A model that's beautiful in a Tuesday afternoon eval but has an outage on a Thursday when a real customer is publishing their site is not, in any meaningful sense, a beautiful model — it's a liability. So we run multiple inference providers for the same model when we can, and we have automatic fallbacks. If the primary provider for a model is having a bad afternoon, traffic shifts to a secondary in seconds and the customer never sees a hiccup. We monitor error rates, latency, and token throughput per provider, per region, per model. Not because it's fun, but because a beautiful site that doesn't load is a bad product. Sometimes the answer to "which model is the best?" is "the one we can actually get a response from right now, at the latency we promised, at the price we advertised." That's not a romantic answer. It's the right one.

So no, we don't just run a few evals and pick one

Pulling all of this together, the honest answer to "how do you choose a model?" is: we choose a model the way a chef chooses an ingredient. We test it. We taste it. We see what it does in different dishes. We notice when a new season's batch is different from last season's. We keep a few trusted options for different moments, and we never bet the whole menu on a single one.

The default model we ship today is not the default model we shipped six months ago, and it probably won't be the default we ship six months from now. The bar keeps moving. Our customers' expectations keep moving. New models arrive, old ones get quietly better, and the gap between good and actually good keeps shrinking in one direction and growing in another.

What doesn't change is the method. Four axes. Real prompts. Real outputs. Honest comparison. A bias toward taste, restraint, and models that feel like a person to work with. And the willingness, every few weeks, to be wrong about yesterday's favourite when something better shows up. That's the whole job. It's never quite finished, and that's fine with us.