This describes work done as an intern on the tools team of a game studio, for a cross-platform game (mobile, PC and console) whose tests run on Android devices. Internals, names and identifiers are deliberately left out. Numbers are from my own measurements at the time and I have kept the caveats that came with them.
A QA team maintains thousands of regression test cases written for humans: "open the album, take a photo, check that it appears in the list." Turning one of those into an automated script is skilled, tedious work, and the goal of my internship was to make a model do it. Give the system a sentence, get back a .py file that a device farm can run, with assertions that mean something.
The naive version takes an afternoon. Paste the test case into a prompt with a few example scripts, get code back. I measured what that produces on thirty real cases: thirty scripts, zero usable. Of 584 UI selectors the model wrote, 503 did not exist in the game. It never once used a debug command, because it did not know any existed. It was fluent, confident, and useless.
Three kinds of fact a model cannot guess
Every failure fell into one of three buckets, and none of them is a reasoning problem.
- Control names. The model writes
poco("BackpackButton"). The real node has a terse, unguessable name inherited from a prefab, and in the source it is called something else again. - Command channels. The game exposes debug (GM) commands, but they travel down four different channels depending on what they do. Send one down the wrong channel and nothing happens. No error, no log line, nothing. The script runs on and asserts against a state that never changed.
- Configuration identifiers. "Place a wooden chair" needs a config id. "Go to the indoor test scene" needs a scene id. "Check the affection value" needs a save-field path. These are in spreadsheets and protobuf files, not in the UI, and guessing is always wrong.
Worse than any of those is the false pass: a script that runs green while checking nothing. In one early batch of 124 cases, 45 passed on device. I read them. Only 7 verified what the test case was actually about. The other 38 asserted things like "the HUD toolbar exists," which is true in every scene of the game, or compared two constants the script had defined itself. A false pass in a regression suite is worse than no test, because it dresses up "unverified" as "verified."
A false pass, reconstructed. Every line is green and none of them touched the thing the test case was about. The audit described below has a rule for each row on the right.
So the design question became: how do you make the model operate under facts it did not invent, and how do you catch a green that lies?
Build the knowledge base from source, not from the phone
The obvious approach is to scan the running game: open every page, dump the control tree, record what is clickable. We did that, and it produced the most trustworthy facts we had, because a control seen on a real device definitely exists at runtime. It also had a ceiling. Scanning is slow, it needs an account in the right state, and empty states hide things: the album list has no items if the album has no photos. One suite went from 4 of 12 passing to 7 of 12 after I took three photos on the test phone. Same generator, same code.
The breakthrough was realising that most of the facts already exist in the game's source tree, in full, for free. So the knowledge base became a layered thing where every row carries its provenance:
| Layer | Where it comes from | What it is good for |
|---|---|---|
| On-device dumps | Real phone, real control trees, recorded transitions | Most trustworthy. Proves a control is visible and clickable now. Slow and account-dependent. |
| Human-verified | QA confirmed a page-open command works | Most trustworthy. Very little of it. |
| Source-derived | Debug command registries, config spreadsheets, protobuf, scene files | Hard facts for ids, coordinates, command names. Says nothing about what is reachable in the UI. |
| Source-parsed | Lua UI handlers: which button opens which page | A lead, not a fact, until a device confirms it. |
| Static scan | Every Unity prefab, recursively expanded | Complete structure. No promise the node exists at runtime. |
The static scan alone yields about 123,000 nodes. The source-derived layer added roughly 1,600 debug commands across the four channels, 33,000 object placements with computable world coordinates, and 1,800 protocol messages. The prompt keeps the provenance tags so the model knows which facts are harder than others.
This one change moved the number most. On a single ticket with seven test points, the system produced a runnable script for 2 of 7 before the source-derived layer existed, and 6 of 7 after, five of them clean on every audit. That is seven data points, and the earlier run was not archived, so I treat it as a strong signal rather than a statistic. But it is the same ticket, same pipeline, and the cause is unambiguous: the model now had the ids and command names it needed.
No embeddings
People assume grounding means a vector store. This system has none, on purpose. Retrieval is a tokeniser that does greedy longest-match against a closed vocabulary of keywords the knowledge base actually contains, plus alias expansion. We explicitly rejected a proper Chinese segmenter, because we do not want the linguistically correct word, we want a word that exists in the table.
Page candidates are then scored, and the scoring is where the design thesis lives. A textual match gets a literal score. But the two largest bonuses have nothing to do with text: +0.40 if the page has a verified way to open it, and up to +0.25 scaled by how much real-device evidence exists for it. Executability outranks similarity. A page we know how to reach and have seen on a phone beats a page whose name merely matches better. Grounding runs in about 25 milliseconds.
Grounding mostly yields page names. The selectors the model actually needs come from the next step: for the top pages, a blueprint assembler runs a few hundred SQL queries across a dozen tables and packs 18 to 21 fact sections into the prompt. The prompt ends up around 25,000 tokens, half of it blueprint. That, not retrieval, is what determines output quality.
One call, and the model picks the exit
The normal path makes exactly one model call. Everything else is local and deterministic, so the cost lands only on the one step that needs a model. The model chooses its own exit: if the request is clear it writes code; if information is missing it emits a structured clarification block, which the UI turns into a form with candidate pages; if the input is not a test case at all it just replies. I tried a keyword router first and measured that 31 of 35 common action words appeared in none of our vocabularies. The clarifier module shrank by a quarter when it stopped deciding and started parsing the model's decision.
After the call, four rewrites run with no model involved, because each has exactly one right answer: a missing (Clone) suffix on a runtime node, a dialog placed in the wrong UI layer, a command sent down the wrong channel, a click written as a coordinate tap. These have to run after the gap assessment, because the AST rewrite erases comments, including the TODO markers the model leaves where the blueprint had no control for it. I learned that ordering the slow way.
Then static validation: a selector linter that collapses deep offspring chains that time out on device, a check for control names that appear nowhere in the knowledge base, and a syntax check.
The false-pass audit
The part I am proudest of is an auditor built from regular expressions and a little AST walking, with eleven hard P0 rules, one conditional, and six P1s. A P0 is turned into an error that drives up to two regeneration rounds; a P1 is recorded. The rules read like a list of the ways a green test has personally lied to me:
- A scene teleport with no check that the scene actually changed, or a scene id that does not exist in the project.
- Asserting that a permanent HUD element exists. True everywhere.
- Asserting two constants the script defined against each other.
- An assertion inside a
trywhoseexceptswallows it. - Asserting only that a debug command was sent, never reading back its effect.
- A
raiseguarding the preconditions but no assertion on the actual requirement.
Six of the rules, rebuilt for this page. Paste your own test script, or edit the samples. The production auditor has eighteen rules and knows the game's real control names and scene ids; this one only knows the patterns.
The most humbling finding: the HUD-toolbar false pass was caused by our own prompt. A rule I had written told the model to confirm arrival in a scene by checking the toolbar. In one 54-case batch, 17 scripts carried that exact false pass. The audit caught what the prompt had taught. The rules themselves have pitfalls too. Counting assertions is not a strength measure. A performance-test exemption has to match a real function call with a parenthesis, not a keyword, or a docstring mentioning "performance" switches off the whole audit. And a P0 should prefer to miss than to misfire, because a false P0 sends a correct script back for rework.
Measuring honestly
The one-line version I use elsewhere is "roughly 30% to 80%." Here is what that line compresses, with the sample sizes attached.
| Sample | Before | After | What changed |
|---|---|---|---|
| One ticket, 7 test points, runnable script produced | 2/7 | 6/7 | Source-derived fact layer added |
| Album suite, 12 intents, on-device pass | 4/12 | 10/12 | Grounding fixes, then three photos on the phone |
| 100 cases, three pipeline versions, executable by the runner | 0% | 100% | Framework compliance enforced |
| Same 100, debug commands that were fabricated | 80% | 12% | Command registry indexed |
| Same 100, scripts with a P0 assertion defect | 2, then 8 | 0 | Audit added. The middle version got worse before it got better. |
The first row is the one that matters and the one with the smallest n. The bottom rows come from the same hundred cases through three versions and are the cleanest before/after I have, with a paired test on effective assertions coming out at p = 0.008. I also retired a number I liked: "automatable rate 13.5% → 30%," because script-output rate turned out to be non-monotonic (97% → 10% → 31% across versions) as the system got stricter and then relaxed, so it measured caution, not quality.
The most valuable measurement was not a pass rate. I ran all 731 leaf cases of one regression module through the pipeline. 13.5% produced a script. 85% asked a clarifying question. That looks terrible until you classify the questions. 294 were missing a page or control name, which the platform can fix by mining more source. The other 330 were missing the object, the precondition state, or a checkable expectation, which no knowledge base can supply. Then the real finding: for all 731 cases, the steps, expected result and precondition fields were empty. They were titles. The best possible knowledge base gets you to 54%. The rest is a collaboration problem, not a model problem, and reframing it that way changed what the team asked QA for.
What I would tell the next person
- Silent degradation is the enemy. The three recurring root causes in three weeks of optimisation were: something degraded without an error, the prompt contradicted itself (rules in code, examples in the database, edited separately), and static inference was treated as device fact. A model obeys rules, so the cost of a contradiction is not gibberish. It is the model marking the one reachable path as no-match.
- Mine the source before scanning the device. Source is complete and free. The device is truth but expensive. Use the device to verify, not to discover.
- Make the model pick the exit. A form with candidate pages beats a script that downloads and fails.
- Audit the green. Every regression suite has false passes in it. Write down how you have been fooled, turn each one into a rule, and let the rules drive regeneration.
- Over-defence is a failure mode too. At one point the system was so careful about unverified selectors that it would not generate anything when the facts were right there in the knowledge base. A test case exists to be automated, not to prove the generator is cautious.
Loosening that last rule was the day the system started being useful.