Experiments · EXP-002
A language model with no prior context writes a complete project
Given only the authoring guide as context, a model wrote a project with four provider requests that resolved on the first try. It is a single run, with no protocol: it helps to frame the question, not to answer it.
Question
If an application really is AAA3, a model that has never seen it should be able to operate it by reading only its documentation. Is that so in practice?
Setup
A language model, in a fresh session, was given only Experience Creator's authoring guide (experience-creator.llm-guide.json, area aaa3 and its recipe). It was asked for a project using all four providers.
Result
The model wrote a JSON with four requests that validated and resolved on the first try, ending with exit code 0.
Why the evidence is partial
- A single run, with no record of the model or its version.
- The guide was written and tuned by the same team that ran the test, knowing the test would happen.
- There is no measure of the result's quality beyond it being valid and resolving.
What it would take
A protocol with several runs per model, several different models, tasks written by someone unfamiliar with the guide and a success criterion defined before starting: rate of valid projects, number of corrections needed and time until a person accepts a project.