Hand-Written Setup Case Study: Fifteen Errors in One Service's Agent Setup, None Flagged When It Was Written
A team that uses two or three agents writes the same setup two or three times: once as a Copilot workflow, once as a Cursor environment, once as a dev container. Each file is written at a different time, from a different example, by whoever needed it that week. Nothing checks them against each other, and most platforms say nothing when a file is wrong until an agent tries to use it.
In the Alpha demo, orders-api gets three setup files written by hand, the way they often look in real repositories, next to the spec the team agreed on. preconfig check reads them. It finds 15 errors and 2 warnings. Eight of the errors are mistakes that would show up only once an agent is at work: a misnamed job, a timeout over Copilot’s limit, a key Cursor’s schema doesn’t allow, the wrong Python, and two platforms without the databases. The other seven are files that are missing, or that don’t match what the spec produces.
The check as recorded: every finding with its file, line and code.
The Set-Up
| Hand-written setup | |
|---|---|
| The spec | orders-api’s preconfig.yaml: Python 3.12, PostgreSQL 16, Redis 7, the setup commands, pytest as the ready check |
| The Copilot workflow | 14 lines: a job named setup on ubuntu-latest, a 90-minute timeout, the database URL under the job’s env, Python 3.11, and a pip install. It runs only when started by hand |
| The Cursor environment | 4 lines: the install command under the key update, and no Dockerfile |
| The dev container | 5 lines: the Python 3.12 image and a pip install |
| The command | preconfig check |
What Happened
check read the four files and printed its findings file by file. The Copilot workflow had the most:
.github/workflows/copilot-setup-steps.yml:4:3: error C002: no job is named copilot-setup-steps (found: setup), so Copilot stops with an error instead of starting work
Rename the job "setup" to copilot-setup-steps.
.github/workflows/copilot-setup-steps.yml:6:22: error C005: timeout-minutes is 90; Copilot allows at most 59
Use 59 or less.
.github/workflows/copilot-setup-steps.yml:7:5: warning C004: Copilot ignores the job setting "env"
Set variables inside a step, or add them to the repository's copilot environment.
.github/workflows/copilot-setup-steps.yml:13: error X003: copilot-setup-steps.yml installs Python 3.11; preconfig.yaml asks for 3.12
Run preconfig build to rewrite it from preconfig.yaml.
GitHub’s documentation is clear on each of these rules: the job must be called copilot-setup-steps or Copilot won’t pick it up, only six job settings count and the rest are ignored, and the timeout can be at most 59 minutes. What a missing copilot-setup-steps job looks like in practice is an agent session that stops with “no copilot-setup-steps job found” and does no work, as one repository’s pull request shows.
check didn’t stop at the wrong job name. With one job in the file, it checked that job as the setup job it was meant to be, so every problem showed at once instead of one per round trip.
The Cursor environment used update for its install command. Cursor’s documentation says the install script “was previously called the update script”, and its schema doesn’t allow keys it doesn’t define. The dev container has an image, so it would start, but with Python only: nothing in it runs PostgreSQL or Redis, so the tests can’t pass there.
What Check Found
| File | Errors | Warnings | Findings |
|---|---|---|---|
| Copilot workflow | 6 | 2 | The job name, the timeout, the ignored env, no triggers on changes, Python 3.11, no PostgreSQL, no Redis, and a file preconfig didn’t write |
| Cursor environment | 2 | 0 | The update key, and a file preconfig didn’t write |
| Cursor Dockerfile | 1 | 0 | Missing |
| Dev container | 3 | 0 | No PostgreSQL, no Redis, and a file preconfig didn’t write |
| Dev container services | 1 | 0 | Missing |
| Setup script, cloud-init | 2 | 0 | Missing |
| All | 15 | 2 |
Each finding has a code a script can act on: the C codes are Copilot’s rules, K is Cursor’s, and the X codes compare the files with the spec. With --diff, check prints how each file differs from a fresh build.
The Numbers
| Result | |
|---|---|
| Files checked | 4: the spec and three setup files |
| Errors / warnings | 15 / 2 |
| Platforms whose file starts no database | 3 of 3 that had a file: Copilot, Cursor and the dev container |
| Platforms with no setup at all | 2: cloud-init and the setup script |
| check’s exit code | 1 |
| Time for the check, not counting program start | About 0.3 ms |
What the Alpha Revealed: The Rules Live in Prose
Most of these rules are written in documentation pages, not in schemas a tool can load. Copilot’s job name, its six settings and its 59-minute limit are sentences in GitHub’s docs; the workflow schema accepts a job with any name. Cursor’s rename from update to install is a note on a setup page. So check carries the rules itself, in the knowledge base, with the date they were last read: September 29, 2026. Keeping those sentences current is part of the project’s work for as long as it exists.
Next: More Rules, Checked on Every Pull Request
In the Beta, check runs as a GitHub Action on every pull request that touches the setup, and comments with what it found. Its rules grow with each platform the Beta runs, and each rule is tried on the platform itself as well as read from its documentation. The roadmap has the plan.
Try It Yourself
Open the live demo and press Play, or jump to step 3. Then, in the second panel under the replay, pick “Copilot: a job named setup” and check it, fix the job name, and check again. “Cursor: a trailing comma” shows a rule of Cursor’s that surprises people: comments are allowed in its file, trailing commas are not.