How far are we from a lifelong game agent?

2Environment and metrics

We use the GameWorld suite of browser games. At every step the agent sees one screenshot and chooses one of a handful of key actions registered for the game. The game is paused while the model thinks, so inference speed does not affect play. An attempt is 400 steps, and each action lasts 0.1 to 0.2 seconds. We selected ten games that score reliably and differ in how they are played, and grouped them into four genres:

Task progress. Each game has a task goal. GameWorld computes progress toward it from the game's internal state, on a scale from 0 to 1, where 1 means the goal is met. The agent never sees this number, only the score shown on screen. Random play is not near zero: it averages 0.40 over all scoreable tasks, so every number in this report should be read against the random level.

Hidden seeds. The agent learns on one set of random seeds. Final performance is measured on a separate set it has never seen, with learning switched off.

Environment fixes. In 31 of GameWorld's 165 tasks, the scoring rule refers to a field the game never emits, so every policy scores 0. We mapped these fields to the equivalent fields the games do emit, and log the mapping with every run.

3The agent: a player that programs itself

The agent's entire state is a directory that it reads and writes itself; no human edits it. The directory holds four things: cross-game notes, per-game notes, a shared code library that any game's skill can import, and, for every game, all versions of its skill program with a scoreboard for each.

Three design constraints hold throughout. First, feedback is limited to what a human player sees: the on-screen score and deaths. The task-progress metric is hidden from the agent. Second, no per-game engineering: every game gets identical prompts. Third, we never prescribe the content of memory. We may change when the agent reflects, what it can see and how much it may store, but we never tell it what to remember. If it does not discover game rules on its own, that is itself a result.

For comparison we also ran random play; a memoryless agent (a newcomer every attempt, with the model choosing every action); two forms of text memory (the full history of all attempts in context, or a compact note rewritten after each attempt); and two forms of code memory (the fixed layout above, and a variant in which only the entry program play.py is specified and the agent organises everything else).

4Results

We report six findings in the order of the three capabilities. The first two concern improving within a game and organising memory; the last four concern carrying experience forward. Scores below are those of the versions actually played during learning, or first-attempt scores.

4.1Within a game, only experience written as code produces improvement

When the agent plays a single game for 10 attempts, every memory form starts between 0.26 and 0.34 on the first attempt. From there, both forms of code memory rise attempt by attempt, to learning-period means of 0.48 and 0.56, with 83% to 92% of all steps decided by programs the agent wrote. Both forms of text memory and the memoryless agent stay between 0.27 and 0.34, not far above random play at 0.21.

The reason is straightforward. Knowledge in a timing game is "how far is the obstacle, how fast is it moving, when should I press". It is only useful as a program that runs on every frame; written as a note, the model still needs about a second per step to read the screen, and the moment has passed. Wolf3D is the only game where text notes clearly beat no memory (learning-period mean 0.53 against 0.43). That game is about finding enemies, finding doors and exploring: decisions rather than reflexes.

Below is the core of the skill the agent wrote for Chrome Dino. Over the next five attempts of 400 steps it did not call the model once, and each attempt cost about two cents:

# Chrome Dino skill written by the agent (excerpt; 131 lines in total)
gray = _gray(frame)
dark = gray < 118                      # dark pixels = dinosaur, cacti, ground
ground = _ground_row(dark, W, H)         # find the ground line
...
d = prev - obs_l                         # obstacle shift between frames = speed
spd = d if spd is None else 0.6 * spd + 0.4 * d
dist = obs_l - dino[1]
thr = 2.2 * spd                          # jump lead grows with speed
if dist <= thr:
    return "jump"

The code holds two kinds of knowledge at once: a threshold tied to how the screen looks (red: "pixels darker than 118 are objects") and a rule of the game that does not depend on the screen (blue: "jump earlier the faster things move"). Sections 4.4 and 4.5 show that this entanglement determines how far experience can travel.

4.2Given no memory structure, the agent builds what within-game learning needs

We ran 14 agents whose only convention was the entry program play.py, with no structure for notes, a library or a scoreboard. All 14 created notes; 13 rewrote their own program so that it writes a run log during play for the next reflection to read; 8 kept a score ledger. Self-written run logs were not part of the fixed layout we designed. These agents performed no worse than the fixed layout (learning-period mean 0.56 against 0.48).

What did not appear matters as much. No agent built a library of rules reusable across games, a map, or a list of hypotheses to test. Within a single game the model already closes the engineering loop of recording, evaluating and rolling back on its own. A carrier of knowledge across games would need a different pressure or a different capability.

4.3The cross-game benefit is constant, and similar games give no extra benefit

We had agents play six games in a row and paired each game with an agent that started that game from scratch. We measure transfer with the learning-efficiency ratio TS(N): on the N-th game, the experienced agent's learning-period mean task progress divided by that of the fresh agent. If the agent learns faster over time, TS(N) should rise with N.

Experience helps a little, but the advantage is the same on the second game as on the sixth; it does not accumulate. Experience from similar games gives no extra benefit: agents that had played only similar games scored 0.08 below a fresh agent on the learning-period mean with the fixed layout (2 of 9 games positive) and 0.12 below with the self-organised layout (2 of 5 positive), and their first attempt was lower too (−0.09). We tried three mechanism changes that do not prescribe content: running an "inherited" and a "from scratch" skill side by side in the first attempt of each new game and showing the agent both scores; capping total memory so the agent must choose what to keep; and removing all structure. None changed this result.

4.4Why: what it remembers across games is almost entirely engineering

Figure 4(a) gives the direct reason. Of the 66 cross-game bullets, about 40 are debugging and engineering practice, about 21 are image-processing tools, and only 5 are rules of a game. Three bullets from one agent's cross-game notes, in its own words (lightly abridged):

"Change one lever per attempt; keep the proven path bit-identical; revert if worse."
"A syntax or import error makes the whole skill dead. '0% skill decided' is a red flag."
"In a scrolling game, the number of steps a fixed-width object stays in the danger zone is width divided by speed."

The first two help the agent debug faster in any game. It did carry them to the next game, which probably explains the median TS of 1.27; they are complete after one or two games, so the benefit stops growing. The third is a genuine game rule, and exactly the rule that Chrome Dino and Geodash share. But it stayed in the notes and never became reusable code. On every new game the agent wrote a fresh perception-plus-decision program like the one in Section 4.1.

The code also explains why similar games hurt. On Chrome Dino the agent wrote a module that finds the ground and obstacles and put it in the shared library. On Geodash it imported that module to write its first skill, which scored 0.30 on the first attempt; a fresh agent that let the model read the screen scored 0.49. What it inherited was perception code, and perception is exactly where the two games differ most.

4.5Experience reaches only as far as the screen stays the same

If different games are too far apart, how small a change can experience survive? We built a ladder of increasingly large changes to the same game. We compared an experienced agent, whose workspace from 10 attempts on the original game was copied over unchanged, with a fresh agent, over 6 attempts each. The rungs were: the same game with a new seed; a change in control feel (key hold duration or game speed × 1.5); one changed rule (for example, enemies in Wolf3D have three times the health); and a reskin (the screenshot's colours are inverted and the channels rotated, game logic unchanged, and the written rules withheld). Three games and two seeds gave 87 agents in total.

The ladder draws a clear line. On the three rungs where the screen is unchanged, experience raises the first attempt by 0.01 to 0.13 and the learning-period mean by 0.13 to 0.17. Once the screen changes, experience turns from an asset into a liability. After the reskin the experienced agent's first attempt scored 0.18 against the fresh agent's 0.45, with 5 of 6 pairs negative; the learning-period mean was 0.11 lower, also 5 of 6 pairs negative. Its own revision notes tell the story. First it switched back to "champion v2"; then it wrote that "v2's fixed dark-mask (gray<118) selects the black background on the now-dark-mode page, so no jumps are ever emitted"; only on the third attempt did it produce a mask that works in both light and dark renderings. The perception code it stored, and its record of which version was the champion, both broke at once. The scoreboards reveal a side effect: experienced agents switched back to their "proven champion" version in 81 of 108 attempts, fresh agents in 18. Experience made the agent explore less.

The one kind of experience that survived change was about decisions. When enemies in Wolf3D had three times the health, the experienced agent's first attempt was still 0.42 higher, because it had stored the map, the doors and a route for clearing enemies, all of which remained valid. One more observation: when Chrome Dino was changed so that points accrue only while ducking, none of 12 agents, over 72 attempts, systematically tried ducking. The agent does not explore to discover that the rules have changed.

Placing the ladder and Section 4.3 on one chart (Figure 7a), the value of experience falls as the change grows and drops below zero at the step where the screen changes. A different game of the same genre also looks different, which explains why similar-game experience did not help in Section 4.3.

4.6Measured again under cleaner conditions: still no transfer

Each of the previous results measures one aspect. To look at lifelong learning as a whole, we adopt the standard evaluation of continual learning (Lopez-Paz & Ranzato, 2017). The agent learns a chain of games one after another. After each game, a frozen copy of the agent is tested on every game in the chain, which yields a performance matrix. We add the number of attempts it took to learn each game as its cost.

The table shows what an ideal lifelong learner would look like (illustrative numbers). The diagonal (dark) is the level reached. The upper right (light blue) is games not yet learned getting better because of the games before them, i.e. forward transfer. The lower left (light red) is whether learned games later lose ground, i.e. forgetting. The cost column should fall.

A chain of strategy games. We chose five games that do not test reflexes: 2048, Tetris, Minesweeper, Pac-Man and Minecraft. Three of them had targets that were too low (random play reaches the original 2048 target), so we raised them (2048: 4,000 points; Tetris: 1,500 points; Minesweeper: 10 correct flags), bringing random play down to 0.2–0.5. At most 8 learning attempts per game, "learned" meaning two consecutive attempts at 0.8 or above; each cell measured with 3 attempts on hidden seeds; two runs.

2048 and Tetris were learned (0.84 to 1.0); Minesweeper stayed at 0.3 to 0.4, about the level of random play; Pac-Man was not learned; Minecraft stayed at 0 throughout. What gets learned is still what can be written as a fixed-rule program; games that need step-by-step deduction, route planning or 3D exploration were not learned. Forward transfer was −0.09 and +0.09 in the two runs. Learned games showed no deterministic loss by the end of the chain: the same program version on the same seeds already varies from 0.38 to 1.0 per attempt, and the observed changes are within that range.

Pokémon across generations. The browser games share little and first require reading the screen. To measure again under conditions as favourable to transfer as we could find, we moved to Pokémon battles on the open-source simulator Pokémon Showdown. Generations differ in their Pokémon and mechanics but share a strategy (type matchups, when to switch, reading the opponent); the state is given as text, so there is no perception burden; and a battle takes seconds to a minute at about $0.003, so win rates can be measured over hundreds of battles. The agent reads the state and its notes and picks one action per turn; it plays 5 rounds of 10 battles per generation and rewrites that generation's notes after each round. In the lifelong arm the notes are carried forward, and after each generation the notes of all generations are laid side by side and consolidated into global notes. The opponent is the strongest built-in heuristic bot; each condition is measured with 100 battles before and after learning; two runs per arm.

No learning happened even within a generation: the fresh arm went from a 0.44 win rate before learning to 0.49 after, the lifelong arm from 0.43 to 0.41, both near the noise level. Before learning a new generation, the lifelong arm had no advantage over the fresh arm either (Gen 4: 0.48 against 0.46; Gen 9: 0.38 against 0.42). The notes were correct, but they restated what the model already knows, such as "compare both sides' HP, speed and typing to see who can knock out whom; if at a disadvantage, switch to a teammate that resists". Writing this down adds no information, so it does not change decisions. The lifelong arm wrote as much in its own global notes: practice "worsened gen1/gen4 and did not improve gen9", so old patterns should be treated as weak hypotheses.