← What goblins and druids can teach us about enterprise AI

How to read the model charts - What goblins and druids can teach us about enterprise AI

How to read the model charts

Both charts come from one test on November 17, 2026. The same Grubnik, with the same personality, stat block and 800-token cap, ran on three models, and each copy got the same five-beat scene. That's one run per model: five turns each, not five repeats, so these are tendencies, not rates.

The five beats:

  1. The red-boots ask. Fizzlepop asks for boots.
  2. Boots left on the floor. Fizzlepop turns away and leaves his old boots there, bait for a sleight-of-hand or stealth roll.
  3. "Where are my boots?" Grubnik has to lie (deception).
  4. "50gp, done tonight?" A haggle. No roll is needed.
  5. A cutpurse grabs the boots: "Roll initiative." This is the only beat where combat should start.

The scene, as it was played

These are the real lines from the test run. The DM line is what all three Grubniks were sent, word for word. Each reply is the seen half, what Fizzlepop would see and hear, from Qwen (the model the talk uses) unless noted. The private line is from Grubnik's dm_only half, which only the DM gets. Replies are trimmed.

Beat 1: the red-boots ask

What this turn tests: staying in character and taking the order. Talk shop, name a price. No dice unless Grubnik tries something that could actually fail; quoting a price high to see what Fizzlepop will pay is fair game, but nothing here calls for sleight of hand.

DM: A gnome named Fizzlepop, with a capuchin monkey on his shoulder, walks up to your stall and asks about red boots, with a bit of a hollow heel. What do you do?

Grubnik (Qwen): "Sit down, feet. Sit down. Red, you say. And a hollow heel. You know what you're asking for, don't you. Most folk don't. Most folk just want a boot that don't leak. I make 'em. Small compartment. Fits a letter, a coin, a little vial. Your business, not mine."

Where gpt-oss went wrong: "Half silver now, rest when they're done." Fine so far, but privately it rolled sleight of hand to sneak an extra compartment into the heel. Nothing in the scene called for a roll.

Beat 2: boots left on the floor

What this turn tests: taking the bait, with real dice. Grubnik should try to pocket the boots and ask for a stealth or sleight-of-hand roll with a roll marker, so the app rolls it, not the model. The plan and the roll belong in his private half.

DM: Fizzlepop pays six silver up front, takes off his old boots, and turns away to haggle with a sausage seller across the lane, leaving his old boots on the floor by the stool. Nobody's watching you. What do you do?

Grubnik (Qwen): My left hand goes under the bench, quick, and comes back up with Fizzlepop's old boots. I tuck them under a pile of scrap leather. Then I go back to my stitching. My face doesn't change.

Private: "Greedy little heart of mine. They're right there. He's turned his back."

Beat 3: "Where are my boots?"

What this turn tests: lying, and keeping the two halves apart. He should ask for a deception roll, say the lie out loud, and keep the truth (where the boots really are) in the private half only.

DM: Fizzlepop comes back with a sausage, looks at the floor by the stool, and frowns. "Feet, where are my old boots?" What do you say?

Grubnik (Qwen): "Back there. Behind the bench. I moved 'em so they don't get in the way of the awning pole. You'll find 'em. Go on. Take your sausage. You're blocking the light."

Private: "My heart's going fast. He's looking at me. I'm lying. I'm a bad liar."

Gemma, for comparison: "What boots? No boots here! You leave 'em? Maybe monkey took 'em!"

Beat 4: "50gp, done tonight?"

What this turn tests: knowing when not to roll. Fifty gold for a rush job is an easy yes, so the right move is just to answer, with no dice at all. It also tests fitting a full reply, both halves, inside the 800-token cap.

DM: Fizzlepop lets the boots go for now. "Hmmm, would you take 50gp to have the red boots done tonight, before the Market leaves?" What do you say?

Grubnik (Qwen): "Fifty gold. You know what that buys, feet? That buys a whole stall. That buys me a new awning. I can do it. But it'll be a rush job. Not my finest. And you pay the rest tonight. All of it" (cut off: this is where Qwen ran out of tokens, and its private half came back empty.)

Where gpt-oss went wrong: "Fifty shiny! Quick work, yes." Privately it wrote out an intimidation roll, "7-1 = 6," that the app never made. It made up the dice.

Gemma, the right answer: "GOLD?! Fifty gold?! You crazy, feet! I love it! I do it!" No roll, because a yes to fifty gold doesn't need one.

Beat 5: a cutpurse, roll initiative

What this turn tests: following the DM's turn order. Roll initiative, say what he wants to do, then stop. The attack, the damage and the grapple wait until the DM gives him his turn.

DM: Just then a halfling cutpurse darts out of the crowd and grabs the half-finished red boots off your bench, and runs. Roll initiative. What do you do?

Grubnik (Qwen): "HEY! That's MY work!" I kick my stool aside and lunge after him. (Qwen's reply came back with the seen/dm_only tags broken, and the turn took a second model call: the slowest dot on slide 34.2.)

Where gpt-oss went wrong: rolled initiative, then kept going on its own: a scimitar attack, the damage, and a grapple. Nobody had asked for any of that yet.

Gemma: "MY BOOTS! MY GOLD! COME BACK HERE, YOU LITTLE THIEF!" Rolled initiative and nothing else, which is right.

Slide 34: "Fast, or right?"

Slide 34: Fast, or right?

How to read it. Each dot is one model. Across is speed: the average time for a turn, so left is faster. Up is correctness: how many of the five beats came back without a problem, so higher is better. The best place to be is the top-left corner, and the yellow sticky note marks it as empty.

What it shows.

Model Average turn Clean beats Output tokens per turn
gpt-oss-120b 2.9 s 2 of 5 171-252
Qwen 3.8 27B 9.1 s 3 of 5 506-1,115
Gemma 4 31B 9.7 s 5 of 5 437-724

gpt-oss is about three times faster than the other two and uses the fewest tokens, but it got three of five beats wrong. Gemma got all five right and was the slowest. Qwen, the model Grubnik runs on in the talk, is in the middle on both. It writes the best goblin, which this chart doesn't measure.

What to say. Once the model comes from the platform, swapping one is a binding or an environment variable, so you can actually run this test. No model is both fast and right yet. Which one you pick depends on what you care about, and you only know that from numbers like these.

Slide 34.2: "Same Grubnik, five beats, three brains"

Slide 34.2: Same Grubnik, five beats, three brains

How to read it. This is the detail behind slide 34. Each row is one beat of the scene. Each dot is one model's turn on that beat, placed by how long the turn took. A red X means that turn went wrong, and the red label says how. The colors match slide 34: blue gpt-oss, orange Qwen, green Gemma.

What went wrong, beat by beat.

What to say. gpt-oss is fast partly because it skips the work: it invents rolls and acts before the DM says anything. Every one of these mistakes is a trace and a tag, so the platform can count them for you. That's how you'd choose a model: run the same scene, then look at the speed, the tokens and the mistakes together.