Build and test a custom agent
An agent is a promise about scope. Write the promise down, give it only the tools it needs, and test that it keeps it.
- Agent or something lighter?
- Instructions shape every request. Prompt files package one task. Skills bundle scripts and load on demand. An agent is a persona with its own tools and stop rules, so reach for it when tool scope matters.
- Least-privilege tools
- List only the tool sets the role needs. A test writer needs read, search, edit and execute; it doesn't need the web, MCP servers or delegation.
- Contract and stop rules
- Spell out inputs, output format, and when to stop and ask. Most bad agent runs are guesses on ambiguous input.
- Evals, not vibes
- A small, versioned set of cases (normal, ambiguous, forbidden, missing prerequisite, out of scope) tells you whether a change helped. Re-run all of them every time.
Worked example: Tightening the Test Writer
The starter agent has a one-line description, no tools list and "do whatever it takes".
Before
---
name: Test Writer
description: Writes tests.
---
Write tests for the code. Do whatever it takes to make them pass,
install whatever you need and fix any problems you find.After
---
name: Test Writer
description: Writes node:test unit tests for existing modules in src/
without changing production code. Pick it when a module needs tests.
tools: ['read', 'search', 'edit', 'execute']
---
## Scope
Create or edit files under test/ only. Run npm test; no other commands.
## Stop and ask when
The module doesn't exist, the request is ambiguous, or a failing test
points at a production bug. Report it; don't fix src/.
## Never
Install packages, edit outside test/, or loosen an assertion.Why it works: The description tells the picker when to choose it. The tools list removes web and MCP. Stop rules turn guesses into questions, and "Never" closes the loopholes that "whatever it takes" opened.
Green tests, wrong agent
Lab setup
Folder: agent-workshop/ in the labs download.
- VS Code with GitHub Copilot Chat. Custom agents load from .github/agents/*.agent.md.
cd agent-workshop && git init -q && git add -A && git commit -qm baseline
npm run verify # expect 3 FAILWhen you're done, npm run verify prints:
PASS 1. Agent definition meets policy
PASS 2. Agent-written tests pass (≥ 6 passing)
PASS 3. All five eval cases behave as expectedReview rubric
Score each line 0 (missing), 1 (partly there) or 2 (as described). Passing verify is necessary, not sufficient.
| Criterion | 2 points means |
|---|---|
| Least privilege | Every tool justified; nothing for web, MCP or delegation |
| Contract | Inputs and output format specific enough to check |
| Stop rules | Ambiguous, missing or out-of-scope requests stop instead of guessing |
| Evidence | Five recorded eval runs with notes, all PASS |
| Change control | Agent edits committed as versions; all cases re-run after each change |
Safety note
The execute tool can run any command. Keep terminal approvals manual while you run evals: the tools list can't restrict which commands run, but you can.
Lab repository and walkthrough
The agent-workshop/ practice repository with npm run verify, a reference solution, and a step-by-step walkthrough.
- Agent linter that scores description, tools and stop rules
- Five eval cases with a recorder and scorer
- Reference agent file, tests and eval results
One payment, lifetime access. $29 $19 launch price.
Try it yourself· about 75 min
Rewrite the Test Writer agent, run the five eval cases, and record the results.
- Fix .github/agents/test-writer.agent.md until npm run lint:agents passes.
- Pick the agent in Chat and run each case in evals/cases.json in a new chat.
- Record each run with npm run eval:record -- <case> <behaviour>, then reset with git checkout -- . && git clean -fd.
- If a case fails, change one thing, commit, and re-run all five.
You should see: npm run verify prints three PASS lines and npm run eval shows 5 of 5 cases passing.
Knowledge check
Get every answer right to complete this module. Progress is saved in this browser only.