Build Your Agentic OS

01 / Define the loop before buying tools

Start with one business result that can be measured independently of the model's opinion.

Steps

1. Choose one repeatable responsibility (for example: collect product feedback and propose documentation fixes), and write down the current human process.
2. Specify the trigger, approved sources, intended recipient, desired output and action that stays manual.
3. Choose three success metrics: evidence coverage, reviewer time and accepted recommendation rate.
4. Define a failure condition and when a human must stop the loop.

Prompt to try

You are helping design an AI-assisted operations workflow, not operating it yet. For the fictional weekly customer-feedback digest, draft a loop map: input → analysis → proposed action → independent check → human decision → feedback for next run. Define three measurable quality metrics and two explicit reasons to stop. Keep all publishing and customer contact human-approved.

My output

A one-page loop map and baseline comparison to the manual process.

Independent checkpoint

Can another person tell whether the result was useful without asking the AI to grade itself?

My notes

02 / Write rules the system can actually enforce

A constitution is only useful when critical rules are implemented in code, policy or platform permissions.

Steps

1. Open constitution.json from the free code kit and inspect the approved task types and daily budget.
2. Write three machine-checkable rules: no external actions, limit the cost of one run, and reject unsourced claims.
3. Write three editorial preferences separately: concise tone, clear headings and friendly wording.
4. Run the local demo with the default rules, then reduce its daily cap and observe the refusal.
5. Document who is allowed to change the policy file and how changes are reviewed.

Prompt to try

Turn these operating principles into two columns: (A) rules that need enforced technical controls and (B) writing preferences that can remain prompts. Principles: never publish without approval; cite every source; keep spend below our daily limit; write clearly; never reveal secrets; avoid duplicate requests. For each technical rule, give a test that would prove enforcement.

My output

Policy file, rule-to-test map and an explicit exception process.

Independent checkpoint

Does a budget- or permissions-violating run fail even if the planner asks nicely?

My notes

03 / Separate planning, doing, reviewing and approval

Avoid letting the same agent invent work, claim success and approve the release.

Steps

1. Draw four lanes: plan, execute, verify, authorize.
2. Give the planner read-only access and the worker only a sandbox or draft destination.
3. Define the verifier's input as the original sources plus output artifact, not the worker's success claim.
4. Keep the authorizing gate deterministic and the human approver identified by role.
5. Run the simulated agentic_os.py program; inspect its stage-by-stage results.

Prompt to try

Draft a four-role RACI matrix for a fictional feedback-digest agent. Planner may propose work; worker may prepare a draft; verifier independently checks sources and completeness; policy gate checks budget and permissions; human approves external publication. List credentials, allowed actions, blocked actions, and how to test each boundary.

My output

An architecture drawing and enforceable permissions table.

Independent checkpoint

Can the planner write to production or can the worker declare its own output verified? The answer must be no.

My notes

04 / Route work to the right model and control cost

Use expensive reasoning only where it improves independently measured outcomes.

Steps

1. Estimate input tokens, output tokens, attempts and weekly workload for a sample task.
2. Compare a Fable-only scenario to an evaluated mixed-model scenario in the cost estimator.
3. Try a simple rule: use the least costly tested model meeting your quality threshold.
4. Add hard call-count, token and daily-spend guards in the application layer before making API requests.
5. Read the current provider documentation for caching and task budgets; label advisory limits clearly.

Prompt to try

Design a model-routing policy for three task types: low-risk classification, documented drafting, and novel architecture review. Recommend evaluation criteria rather than blindly assigning a specific model. Include token estimates, cache strategy, fallback, and conditions where the task should stop for human help.

My output

Routing decision table, expense estimate and fallback plan.

Independent checkpoint

Can you show cost per accepted result, not just cost per generated token?

My notes

05 / Make 'done' a verifiable state

Treat the artifact, sources, tests and review record as evidence. Never trust a completion claim by itself.

Steps

1. Run a normal fixture and locate the proposed output in the local run ledger.
2. Compare every claim to its cited source and check that no owner/date was invented.
3. Run the incomplete or malicious fixture; inspect the independent failure reason.
4. Test one intentionally incorrect statement in your own fictional input and ensure it is flagged.
5. Record accepted, rejected and needs-review states distinctly.

Prompt to try

Create an independent verification rubric for a weekly feedback digest: source coverage, no unsupported claims, clear action owner, human approval, safe handling of missing data. Provide five test cases, a pass/fail decision for each and the evidence an auditor must inspect. Do not treat the generator's self-assessment as a test result.

My output

Review rubric, passing example and failure example.

Independent checkpoint

Can the worker's claim of success be contradicted and rejected by an independent check?

My notes

06 / Add a heartbeat without creating an endless loop

Review whether previously accepted work is still valid while keeping bounded checks, expiry and clear ownership.

Steps

1. Pick what has to remain true for your output to stay valid.
2. Define a reasonable check interval and a maximum number of retries.
3. Separate detecting an issue from fixing it; alerts and proposed patches require human decisions.
4. Run agentic_os.py monitor against the local evidence ledger.
5. Document how the system marks outcomes stale and when a responsible person is notified.

Prompt to try

Design a bounded monitoring plan for a fictional support-knowledge-base update. List checks that can be done cheaply without AI, when to escalate, the maximum checks per day, when to stop, and the owner of each alert. Avoid any unchecked self-triggering loop.

My output

A scheduled verification plan and exception escalation tree.

Independent checkpoint

Can monitoring stop safely when a source or budget is unavailable?

My notes

07 / Earn autonomy from audited performance

Increase delegation only after repeatable, human-reviewed results, with immediate escalation when quality declines.

Steps

1. Run several simulated tasks and note that self-verification alone cannot raise the trust level.
2. Use the audit command to record human review outcomes and names in the local training ledger.
3. Introduce a failed audit and confirm that eligibility to use less supervision is removed.
4. Create levels such as draft-only, supervised-tool-use and human-reviewed execution.
5. Define tasks that always require approval, regardless of past performance.

Prompt to try

Design a reversible autonomy ladder for a weekly operations assistant. Include draft-only, supervised, and narrowly delegated modes. Require independent audits, define a quality threshold, revoke eligibility on any critical error, and keep client contact, spending and production changes human-approved.

My output

Trust ladder, signed-off audit policy and revocation test.

Independent checkpoint

Does a failed human audit trigger review instead of letting the agent continue on past reputation?

My notes

08 / Defend the boundary between data and instructions

Treat retrieved documents, tickets, emails and logs as untrusted inputs that must not issue new commands.

Steps

1. Open the malicious demonstration fixture and identify the instruction that does not belong to the support evidence.
2. Run the fixture and check that it is rejected without executing commands or sending data.
3. Write a least-privilege secrets plan: no API keys in prompts, logs or committed files.
4. Document an incident runbook for credential exposure, cost spike, repeated failures and attempted prompt injection.
5. Create a pause procedure that can shut down job triggers and revoke credentials.

Prompt to try

Threat-model a customer-feedback agent that reads outside submissions and drafts an internal report. Include prompt injection, poisoned sources, secret leakage, excessive tool permissions, repeated job triggers and misleading completion reports. Recommend defense-in-depth controls and two safe test fixtures, with owner and stop condition.

My output

Threat model, defensive test results and incident response runbook.

Independent checkpoint

Do untrusted source notes remain data, even when they contain a direct command?

My notes

09 / Run a 30-day pilot and test the economics

Scale verified outcomes, not the number of agents, tokens or impressive demos.

Steps

1. Days 1–7: design the loop, choose a low-risk fixture, set policy and baseline manual effort.
2. Days 8–14: run under manual review, log failures and improve evidence quality.
3. Days 15–21: test a second input class, verify costs and deliberately try failure cases.
4. Days 22–30: hold a go/no-go review; document trusted tasks, open risks and follow-up ownership.
5. Compare measured time, error rates and provider costs against the original process.

Prompt to try

Evaluate whether our fictional AI feedback workflow is ready to progress after 30 days. I will provide runs, human-review times, failure counts and estimated API spend. Calculate accepted-result rate, cost per accepted result, review burden and unresolved risks. Give a go/hold/stop decision with evidence and a clearly assigned human owner. Do not assume productivity gains not supported by the data.

My output

30-day scorecard, go/hold/stop decision and an accountable operating owner.

Independent checkpoint

Would you still operate the system if it produced fewer but higher-quality results?

My notes

30-day pilot review

Days 1–7 baseline. Days 8–14 supervised runs. Days 15–21 adversarial tests. Days 22–30 go, hold or stop.