Project 01 · last run published Oct 19, 3:14 AM
Terraria Agent Bench
A local open model (gpt-oss:20b) plays real Terraria through a bridge mod. It writes programs over skills written in C#, the game runs them, and a benchmark checks every claim against the live game. Every run publishes itself here when it finishes.
- 17/18fights survived
- 4/4 vs 8/12A* vs greedy
- 12/19plan steps, best run
- 7/8 vs 7/8empty vs learned library
105 runs · 13 marked as harness bugs · press 1–9 to move around
Watch it play
The newest recorded run: the model on a learned skill library, Oct 18, 6:35 AM. One clip per task, failures included.
- Work benchpassed0:46
- Stonepassed0:44
- Doorpassed0:55
- Copper barsfailed1:13
- Chair and tablepassed0:57
- Woodpassed0:45
- Hammerpassed1:00
- Furnacepassed2:00
Real game footage from the sim. The game runs fast-forwarded while the model's programs play out, and freezes while the model thinks. The run these come from.
The starter plan, run by run
Nineteen steps: wood, a house, stone, copper, iron and on. Each column is one run. Grass-topped columns fought enemies, stone columns are the build track with enemies off. Hearts are deaths.
Survival suites
Share of scripted fights the character survived and won. Each block is one suite.
A local model writing its own skills
Macros are off: gpt-oss composes mining, crafting and placing itself. A program that passes is saved and offered to later tasks. Left mark: empty library. Right mark: learned library.
- Wood✓✓
- Work bench✓✓
- Chair and table✓✓
- Door✓✓
- Stone✓✓
- Furnace✓✓
- Copper bars✗✗
- Hammer✓✓
Found by measuring
Most progress came from bugs a run or a suite made visible, including bugs in the benchmark itself. Hover one for the fix and the commit.
Attack refused to swing below 30% health: 277 refusals in one run, 0 swings
the character fights back when cornered (run 4)
The copper shortsword was treated as a 14-tile ranged weapon, so the character stabbed air: 0 kills in runs 1 to 5
first kills; survival suite 3/6 to 11/12
Scenarios started at spawn, which the Mac run had turned into a house and a mine shaft
every attempt fights on the same ground
The escape started at 30% health with a zombie already on top: 6 and 7 deaths in 4 minutes
low-health scenarios pass (zombie_half_hp 6/6 across S2 and S3)
The greedy walker cannot take a route that starts by moving away from the target
nav suite 8/12 to 12/12; tunnel course 0/3 to 3/3
The first detour course was unsolvable by any walker (stairs faced the wrong way)
the A/B measures the walker, not the course
Two benchmark processes drove one game and cancelled each other's skills: a 0/8 run
a second driver exits with a clear message
The check wanted the item named Wood; the model held 31 BorealWood and, when told it lacked Wood, crafted a BorealWoodWorkBench on its own
the model's correct adaptation now counts as a pass
A task started on a floating island at height 66, above a 29-tile drop
tasks start on real ground
The model saved a skill with hard-coded coordinates; reused elsewhere, it walked to trees that were gone
saved skills carry over to new places
Scenarios used the Mac character (120 max life, extra gear), which flatters the numbers
results don't depend on what earlier runs left in the save
The floating-island fix (an air-gap test) skipped real ground over caves and started tasks underground, invalidating a whole A/B pass
the A/B restarted on verified ground
One furnace attempt ran 6918 s of wall clock inside a 180 s skill budget: timeouts count game ticks, and the sim was running far below 60 ticks/s
a slow game can no longer hide a stuck skill
Mine_nearest tried ~100 trees one by one behind the same Ebonstone wall it could not dig
the same spot fails in 2-4 s with 'Ebonstone needs 65% pickaxe power' instead of hours
The containers spent their CPU drawing frames nobody sees: ~21 ticks/s with the agent's own code at 0.4 s of CPU per 78 s skill
60 ticks/s headless, ~500 at speed 8; the same 7-tree task takes the same 1639 ticks and ends on the same tile at 1x and 4x, in 3.5 s instead of 78 s
One of three game instances crashed and its worker failed 27 jobs in a second as unreachable, which published a 3/30 suite
a crash costs time, not a wrong number on the page
In a snow start the model asked for a WorkBench three times; craft only said 'Wood 0/10' while it held 35 BorealWood
workbench passes; on a fresh world per task the fresh pass got 5/8 and the learned pass 6/7, wood to door all on the first attempt
A reused library skill aborted a furnace program for being 3 stone short of the check from the task it was learned on (the furnace needs 20, it had 27)
reused skills chain: hammer reused three saved skills in one program
Fast-forward made fights harder: with the world running between calls, every second the agent spends deciding costs 4 seconds of game, and the flier scenario fell from 3/3 at 1x to 1/3 at 4x
headless 1x survival suite 16/18, the best so far (15/18 rendered)
Task patches 120 tiles apart while tree search reaches 180 each way, and the model cut 30 trees a call: the fifth task found no tree in reach
chair_table, which failed in every earlier run, passed first try in both passes
On a steep hillside no flat 3-tile floor was in reach, so the work bench could not be placed anywhere the character walked
both failing hillside spots now craft the door
A tunnel stopped at grass with a tree on it ('did not break: something on top'); Terraria won't break a block under a tree
the same tunnel digs through (15 tiles)
At 35 health with zombies 3 tiles away on both sides, dig_in (about 6 s of digging) died every time; the scenario sat at 1/3
first live run (Mac showcase): flee escaped, 25 tiles, nothing within 18; then the zombies caught up during the 6 s dig_in and it died. Next: run further before digging
Flee stopped at 18 tiles clear; a zombie walks ~5 tiles/s and dig_in takes ~7 s, so they reached the hole mid-dig (0/3, same as the Mac showcase). Once a bunker healed to its target, the work step mined out with zombies still on the roof
two_zombies_low_night 0/3 -> 3/3, then 2/3 on the next build (5/6); whole suite 14/18 -> 15/18 -> 16/18
Craft reported ingredients for one batch ('CopperOre 0/3') after making 1 of 3 bars, so nothing said 9 ore were needed
the hint shows up in the night3 log, and gpt-oss still mined 2. It keeps reading the ore count as a bar count: wording alone doesn't fix copper_bars
The model gave up: three attempts of 'cannot_smelt_bars: wait 0', which the bench scored like any other failed program
no give-up programs in the night3 or night4 passes
No Furnace could go down at the bottom of a 2-wide shaft. Three bugs stacked: levelling started 2 columns out (the column beside you is the wall), it first walked toward flat ground up the shaft and levelled mid-jump 25 rows up, and every pickup hop made the next mine check reach in mid-air
station suite 9/12 -> 12/12 (shaft 0/3 -> 3/3); nav 4/4 and survival 16/18 on the same build
Every run
Including the ones a harness bug spoiled. They stay in the log, greyed out, and are left out of the charts.
- Oct 15, 11:24 AMPlan run (survival track) reached 4/19 plan steps in 10 min. The agent was slain 37 times. (deaths counted once per poll (fixed before run 3))
- Oct 15, 11:39 AMPlan run (survival track) reached 4/19 plan steps in 23 min. The agent was slain 5 times.
- Oct 15, 12:03 PMPlan run (survival track) reached 4/19 plan steps in 6 min. The agent was slain 4 times.
- Oct 15, 1:34 PMPlan run (survival track) reached 5/19 plan steps in 4 min. The agent was slain 6 times.
- Oct 15, 1:38 PMPlan run (survival track) reached 5/19 plan steps in 4 min. The agent was slain 7 times.
- Oct 15, 1:43 PMPlan run (build track) reached 7/19 plan steps in 11 min.
- Oct 15, 1:45 PMPlan run (build track) reached 7/19 plan steps in 18 min.
- Oct 15, 2:04 PMSurvival suite: survived 3 of 6 fights. (harness bug: spawn location and the Mac character (before the arena and fresh_player))
- Oct 15, 2:13 PMSurvival suite: survived 11 of 12 fights. The agent was slain once.
- Oct 15, 2:19 PMSurvival suite: survived 15 of 18 fights. The agent was slain 3 times.
- Oct 15, 2:28 PMNavigation, A*: 9 of 12 courses finished. (harness bug: the detour course was unsolvable (replaced by the tunnel course))
- Oct 15, 2:29 PMNavigation, greedy walker: 9 of 12 courses finished. (harness bug: the detour course was unsolvable (replaced by the tunnel course))
- Oct 15, 2:30 PMNavigation, A*: 12 of 12 courses finished.
- Oct 15, 2:31 PMNavigation, greedy walker: 8 of 12 courses finished.
- Oct 15, 2:39 PMEmpty skill library: the model passed 0 of 1 tasks. (harness bug: planner returned empty programs (fixed: programs need at least one step))
- Oct 15, 2:40 PMEmpty skill library: the model passed 1 of 1 tasks.
- Oct 15, 2:41 PMEmpty skill library: the model passed 0 of 7 tasks. (harness bug: two benchmark processes drove one game (fixed: per-instance lock))
- Oct 15, 2:41 PMEmpty skill library: the model passed 0 of 7 tasks. (harness bug: two benchmark processes drove one game (fixed: per-instance lock))
- Oct 15, 2:46 PMEmpty skill library: the model passed 0 of 2 tasks. (harness bug: the check wanted the item named Wood while the model held BorealWood (fixed: wood variants count))
- Oct 15, 2:50 PMEmpty skill library: the model passed 2 of 7 tasks. (harness bug: tasks shared one cut-down forest and one start sat on a floating island)
- Oct 15, 3:00 PMEmpty skill library: the model passed 1 of 3 tasks. (harness bug: start finder put tasks underground (fixed the same afternoon))
- Oct 15, 3:11 PMEmpty skill library: the model passed 5 of 8 tasks.
- Oct 15, 5:29 PMLearned skill library: the model passed 1 of 8 tasks. (harness bug: the game container was killed mid-run (Docker ran out of memory with four worlds starting at once))
- Oct 15, 5:44 PMSurvival suite: survived 3 of 30 fights. (harness bug: one of three game instances crashed and its worker failed 27 jobs as unreachable (fixed: jobs go back to the queue))
- Oct 15, 5:47 PMSurvival suite: survived 26 of 30 fights. The agent was slain 4 times.
- Oct 15, 5:49 PMEmpty skill library: the model passed 4 of 6 tasks. (harness bug: tasks shared forests (patches 120 tiles apart, tree search 180 each way): the fifth task found none; stopped and re-placed)
- Oct 15, 5:52 PMSurvival suite: survived 16 of 18 fights. The agent was slain 2 times.
- Oct 15, 6:00 PMEmpty skill library: the model passed 5 of 8 tasks.
- Oct 15, 6:01 PMSurvival suite: survived 16 of 18 fights. The agent was slain 2 times.
- Oct 15, 6:10 PMLearned skill library: the model passed 6 of 7 tasks. (harness bug: incomplete: stopped before copper_bars when the host ran low on memory (6/7 run, wood to door all first-attempt))
- Oct 15, 6:15 PMNavigation, A*: 12 of 12 courses finished.
- Oct 16, 12:00 AMEmpty skill library: the model passed 6 of 8 tasks.
- Oct 16, 12:19 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 16, 1:10 AMSurvival suite: survived 14 of 18 fights. The agent was slain 3 times.
- Oct 16, 1:19 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 16, 1:36 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 16, 1:51 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 16, 2:32 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 16, 3:00 AMPlan run (survival track) reached 10/19 plan steps in 62 min. The agent was slain 26 times.
- Oct 16, 4:04 AMSurvival suite: survived 15 of 18 fights. The agent was slain 3 times.
- Oct 16, 4:14 AMSurvival suite: survived 9 of 12 fights.
- Oct 16, 4:16 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 16, 4:18 AMSurvival suite: survived 0 of 1 fights.
- Oct 16, 4:22 AMSurvival suite: survived 9 of 12 fights.
- Oct 16, 4:26 AMSurvival suite: survived 0 of 1 fights.
- Oct 16, 4:30 AMSurvival suite: survived 0 of 1 fights.
- Oct 16, 4:34 AMSurvival suite: survived 9 of 12 fights.
- Oct 16, 4:38 AMSurvival suite: survived 9 of 12 fights.
- Oct 16, 4:41 AMSurvival suite: survived 0 of 1 fights.
- Oct 16, 4:44 AMSurvival suite: survived 9 of 12 fights.
- Oct 16, 4:48 AMSurvival suite: survived 0 of 1 fights.
- Oct 16, 4:51 AMSurvival suite: survived 12 of 12 fights.
- Oct 16, 5:06 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 16, 5:44 AMNavigation, A*: 4 of 4 courses finished.
- Oct 16, 5:46 AMSurvival suite: survived 16 of 18 fights. The agent was slain once.
- Oct 16, 5:55 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 16, 6:23 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 16, 5:34 PMSurvival suite: survived 15 of 18 fights. The agent was slain 3 times.
- Oct 16, 5:50 PMSurvival suite: survived 12 of 12 fights.
- Oct 16, 5:51 PMLearned skill library: the model passed 0 of 1 tasks.
- Oct 16, 5:54 PMLearned skill library: the model passed 0 of 1 tasks.
- Oct 16, 5:55 PMLearned skill library: the model passed 0 of 1 tasks.
- Oct 16, 6:33 PMNavigation, A*: 4 of 4 courses finished.
- Oct 16, 6:33 PMSurvival suite: survived 4 of 4 fights.
- Oct 16, 6:34 PMSurvival suite: survived 4 of 6 fights. The agent was slain once.
- Oct 18, 2:10 AMPlan run (survival track) reached 6/19 plan steps in 61 min. The agent was slain 31 times.
- Oct 18, 2:11 AMEmpty skill library: the model passed 6 of 8 tasks.
- Oct 18, 2:20 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 18, 2:32 AMSurvival suite: survived 17 of 18 fights. The agent was slain once.
- Oct 18, 2:47 AMSurvival suite: survived 12 of 12 fights.
- Oct 18, 2:49 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 18, 3:05 AMLearned skill library: the model passed 6 of 8 tasks.
- Oct 18, 3:11 AMPlan run (survival track) reached 10/19 plan steps in 61 min. The agent was slain 42 times.
- Oct 18, 3:21 AMEmpty skill library: the model passed 6 of 8 tasks.
- Oct 18, 3:32 AMLearned skill library: the model passed 6 of 8 tasks.
- Oct 18, 3:45 AMEmpty skill library: the model passed 6 of 8 tasks.
- Oct 18, 3:57 AMLearned skill library: the model passed 5 of 8 tasks.
- Oct 18, 4:10 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 18, 4:13 AMPlan run (survival track) reached 8/19 plan steps in 62 min. The agent was slain 26 times.
- Oct 18, 4:26 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 18, 4:44 AMEmpty skill library: the model passed 6 of 8 tasks.
- Oct 18, 4:59 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 18, 5:14 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 18, 5:16 AMPlan run (survival track) reached 7/19 plan steps in 60 min. The agent was slain 31 times.
- Oct 18, 5:24 AMLearned skill library: the model passed 6 of 8 tasks.
- Oct 18, 5:35 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 18, 5:47 AMLearned skill library: the model passed 6 of 8 tasks.
- Oct 18, 6:00 AMEmpty skill library: the model passed 6 of 8 tasks.
- Oct 18, 6:11 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 18, 6:20 AMEmpty skill library: the model passed 7 of 8 tasks.
- Oct 18, 6:35 AMLearned skill library: the model passed 7 of 8 tasks.
- Oct 18, 1:23 PMSurvival suite: survived 17 of 18 fights. The agent was slain once.
- Oct 18, 1:38 PMPlan run (survival track) reached 9/19 plan steps in 60 min. The agent was slain 21 times.
- Oct 18, 2:39 PMPlan run (survival track) reached 9/19 plan steps in 60 min. The agent was slain 27 times.
- Oct 18, 3:40 PMPlan run (survival track) reached 11/19 plan steps in 60 min. The agent was slain 34 times.
- Oct 18, 5:02 PMPlan run (survival track) reached 10/19 plan steps in 60 min. The agent was slain 22 times.
- Oct 18, 6:04 PMPlan run (survival track) reached 8/19 plan steps in 65 min. The agent was slain 19 times.
- Oct 19, 12:00 AMPlan run (survival track) reached 8/19 plan steps in 60 min. The agent was slain 68 times.
- Oct 19, 12:01 AMPlan run (survival track) reached 8/19 plan steps in 60 min. The agent was slain 52 times.
- Oct 19, 1:02 AMPlan run (survival track) reached 8/19 plan steps in 61 min. The agent was slain 27 times.
- Oct 19, 1:04 AMPlan run (survival track) reached 10/19 plan steps in 60 min. The agent was slain 14 times.
- Oct 19, 2:04 AMPlan run (survival track) reached 9/19 plan steps in 61 min. The agent was slain 28 times.
- Oct 19, 2:07 AMPlan run (survival track) reached 11/19 plan steps in 65 min. The agent was slain 36 times.
- Oct 19, 3:05 AMPlan run (survival track) reached 7/19 plan steps in 60 min. The agent was slain 65 times.
- Oct 19, 3:14 AMPlan run (survival track) reached 12/19 plan steps in 61 min. The agent was slain 29 times.
What is and isn't measured
Real game, real inputs. Movement, fighting and item use go through the player's controls each tick, the way a keyboard would. The navigator from TerraBlind (Reisenbug, MIT) runs with its free-item helpers switched off.
Our shortcuts. Mining waits a simulated swing time, then breaks the tile directly; the dig-in bunker places its roof blocks instantly from the inventory. Both are next to move to real item use.
Test setup, not skills. Scenarios teleport, spawn enemies and reset the character to set a known start. The agent never calls these.
The model's job. gpt-oss:20b, running locally, writes programs over skills written in C#. It plans and recovers; it does not press keys itself.
Small numbers. Suites run each scenario 1 to 5 times, so one attempt moves a pass rate a lot. Runs made on a harness later found broken stay listed, marked, and out of the charts.
Game time, not wall time. From 2026-10-15 evening, scenario limits and timings count the game's own ticks. Earlier suites ran at ~20 ticks/s against wall-clock limits, so a "40 s" scenario gave the agent about 14 s of game. Suites run headless, often fast-forwarded (several game ticks per frame); the same task takes the same number of ticks at any speed.
Navigation option from TerraBlind by Reisenbug (MIT), with its free-item helpers switched off.











