Development

For work on Kaimahi; contribution/PR expectations are in CONTRIBUTING.md. For use, start with getting started and Orka.

New contributors should read the codebase overview for package boundaries and data flows, then the interface baseline before changing commands or public contracts. The repository map is the mechanical inventory of what is installed, embedded, historical or test scaffolding.

Direction and current implementation

Orka remains the first-class/default platform. Kaimahi tooling helps agents get onto it. Default kmx create <name> authors one native workspace member offline, requiring an exact model and an instructions file; Orka is the default, while --runtime kagent selects the exact v0.10.3 Go profile. Create and push share workspace discovery; exact --workspace initializes without prompting. Live create/previews/wizard and user bundle commands, readers/writers/receipts are removed, not converted. Old receipts are unreadable; no files/resources are deleted. Quickstart/console compose native authoring, full eval, exact publication and reviewed Lift. Quickstart verifies placement before a separate fresh answer Task; console Ready means placement only. Kagent uses exact v0.10.3 Go/linux-amd64 packaged eval/evaluated push/create-only Lift; no installer, discovery, standalone chat/run or suite status/retirement. KMX installs Orka's chart in harness-v2 controller mode, not the upstream harness ABI. Runtimes and harnesses own execution, enforcement and model access. Orka already supplies OTLP telemetry using GenAI conventions; extend that upstream capability instead of adding a parallel KMX exporter.

Plane CLI administration, model overlays, credential exchange and workload migration are removed, together with the nested source module, image-build helper, embedded plane/observability assets and plane-only scripts/fixtures. Historical source, including checked-in SQL migrations, remains in Git history, not this checkout. No databases, credentials, caches or deployed resources are deleted. Cloud ownership/teardown records and their tests remain. The old gateway, approvals/grants, workflow runner and connector fixtures remain retired.

Packaged native suites have runtime-specific private App preparation seams. Orka observes an explicit target, Ready Provider/Secret binding and pinned runtime, then validates one exact unmarked render from workspace.Load; placement can create or update only same-owner Provider/Agent resources. Kagent snapshots author-owned KagentBindings, verifies the pinned runtime and creates new ModelConfig/Agent resources only; occupied unrecorded or changed targets refuse, never update or adopt. Secret reads project metadata and key presence, never values. Root workspace evaluation consumes the matching seam. Verification binds current ownership, UID, generation, spec and readiness to live snapshots. A late failure can leave resources for a later explicit invocation. All six native annotation keys reserve objects against legacy adoption, reuse, copying, bundle authorization and retirement, even with malformed values. These fences retain legacy invocation/adoption/copy refusals for suite-owned objects. Fake readiness is not model execution or cluster qualification.

Private full-passing receipts project Task/answer and ordered case/assertion identities only, never prompts, answers, operands, reasons or diagnostics. Their constructor binds observed raw-case/input frames and successful placement to a verified package; execution brackets observations with native verification. The private export preflight retains one private anchored directory outside authored workspace/cache trees and publishes complete bytes with an exclusive hard link, not history or replacement. The private receipt-file push consumer parses before cache writes, compares exact S, and validates packaged subject/render/inventory coherence before provenance or transfer. Source fingerprint is authored spelling, not a durable publication lock. These unsigned observations prove neither authenticity nor cross-object atomicity. Root eval prepares workspace/inventory/export offline before configuration, then consents once to native pair placement and all ordered one-shot Tasks. Only a full pass exports privately. push --evaluation FILE uses the strict file consumer; ordinary push remains unevaluated; bundle evaluation is removed. Hermetic tests cover these paths. The existing Orka runtime shard now invokes the bounded suite qualifier after its raw upstream Tool/Task and model-answer proof. It uses the fixed CPU AIKit workload, existing model configuration and installer Secret reference in the same owned cluster. One full-workspace evaluation is followed by independent live readback, exact-S local-layout transfer and ownership/unsupported-evaluator refusal audits. Private command captures are never artifacts; only passing receipt metadata and a scalar qualification report are uploaded. A successful run qualifies the fixed two-case native workspace scenario; hermetic tests alone do not establish runtime execution. Root digest Lift separately consumes the private passing receipt for deploy-only packaged Orka/kagent placement. The fixed Orka qualifier proves exact registry/local-layout reuse with source withheld and bounded registry promotion into an absent pair on a second owned cluster, followed by a fresh destination Task. It does not qualify every creation, promotion or partial-outcome path. The kagent qualifier proves source-free Lift into a fresh namespace and occupied repeat refusal without additional native execution. Standalone packaged evaluation is unavailable.

Repository layout

Consult the repository map for tracked membership and caller classification.

Path	Responsibility
cmd/kmx/, internal/kmx/, embed.go	CLI command tree, Orka and version-bounded kagent lifecycle adapter, portable authoring, shared plumbing and embedded manifests
pkg/kmx/	experimental public setup-only environment plus suite/deployment contracts, not wired into the installed CLI
k8s/ollama.yaml, k8s/orka-k8s-tool.yaml	native local model and read-only Kubernetes Tool
scripts/, Makefile	checks, packaging and repository helpers
docs/package.json, docs/package-lock.json, docs/.vitepress/	private documentation toolchain, navigation/theme and generated-site tests; published from main by the Pages workflow
.github/workflows/	verification jobs and docs-only routing

The root CLI is the only Go module. Plain make builds bin/kmx only. The supported kagent surface is exact-v0.10.3 Go/linux-amd64 packaged workspace evaluation/evaluated push/create-only digest Lift through private App seams, not a user-bundle conversion layer. Public alpha ports remain unwired.

For a narrative architecture view, including workspace preparation, native evaluation, OCI publication and user-bundle retirement, use the codebase overview. For the current visible and hidden command tree plus pkg/kmx method sets, use the interface baseline.

Build and verify

Run from the repository root. AgentSuite is a public package in the same Go module, so these commands include it without a workspace or a dependency pin. CI's Go hygiene checks remain unguarded, including on documentation-only changes; only the live and clone-free operation steps use documentation-only guards. Formatting checks must test gofmt output because gofmt -l alone exits successfully even when it lists files.

Tests that read or write KMX state must set KMX_HOME to t.TempDir(). Do not use XDG_CONFIG_HOME as the test boundary: Go uses it for the native Linux config location, while macOS uses ~/Library/Application Support. KMX_HOME is the repository's explicit cross-platform isolation contract.

formatted=$(gofmt -l agentsuite cmd internal pkg/kmx embed.go embed_test.go) && test -z "$formatted" && go vet ./... && go build ./... && go test ./...
npm --prefix docs ci --ignore-scripts && npm --prefix docs run docs:check
python3 scripts/check-doc-links.py --selftest
python3 scripts/check-doc-links.py
python3 scripts/check-secret-shapes.py --selftest
python3 scripts/check-secret-shapes.py
bash scripts/check-no-azure-ids-test.sh
bash scripts/check-no-azure-ids.sh
bash scripts/kube-guard-test.sh
python3 scripts/test_model_fixtures.py -v

Documentation preview

Node.js 22.12 or newer is required only for the browser documentation tooling. make docs-preview installs the exact lockfile and serves on 127.0.0.1; make docs-build performs the same build and generated-site assertions as CI. The preview renders the root README and docs/*.md directly, plus agentsuite/spec.md while it remains local. Do not copy guide content into VitePress configuration or commit docs/.vitepress/dist, caches, or node_modules.

The current framework pin remains explicitly reviewable. Organization GitHub Pages publishes the same build from main; pull requests only build and test it. No repository publication secret is configured. See documentation site.

App test timing

internal/kmx/app fixtures inject invocation-local polling intervals and phase deadlines. Keep setup budgets intact and shorten only the wait under test; assert that the intended readiness, admission, or result boundary was reached. An unset timing hook must retain the production intervals and deadlines.

The fake kubectl executable lives in internal/kmx/app/testdata/kubectl. App fixtures default to compiling this lightweight executable once per package run and running it with the fixture's environment and call log. A go test -race App run also compiles the helper with -race; its build directory is removed when the package run finishes. Keep helper imports independent of App and UI.

The shared stateful protocol lives in internal/kmx/app/kubectlfake for explicitly selected test captures and executable-boundary fixtures. It has no App/UI imports and is not reachable from installed commands. Normal argv/environment/stdin, guard/deadline/error boundaries remain observable. Retired bundle tests are not proof of retained bundle commands/readers; surviving helper inventory follows the Go retirement pass.

The workflow's hygiene job is authoritative for the full checker/self-test/ mutation set; do not replace it with this focused list. The retained native Orka Tool fixture tests exercise actual HTTP requests and health/refusal responses without a cluster. No plane-module or plane Postgres service job remains. CI configuration also checks its own guards and aggregator membership. The retired-command scanner rejects concrete removed commands/assets, not kagent vocabulary or version claims. It uses bounded literal command tokenization and exact inert exceptions; production command tests own runtime selection and supported-operation validation.

Local loop

make
export KIND_CLUSTER=dev-local
export KUBE_CTX=kind-dev-local
bin/kmx quickstart
bin/kmx status
# Quickstart verifies placement before its separate fresh answer Task.
bin/kmx local down

A bare bin/kmx local up installs the runtime without an Agent: kind, Ollama, the model and pinned Orka with its keyless local Provider. Author source with bin/kmx create NAME --workspace DIR --model MODEL --instructions FILE, then full eval, evaluated push and digest Lift with private evidence; or use guided bin/kmx quickstart --interactive. Root/Orka status is table-only runtime state, not a suite observer. Suite status/retirement remain unavailable.

KMX never installs/upgrades kagent. Its exact-v0.10.3 Go/linux-amd64 packaged profile uses author-owned --bindings for root eval/Lift, without runtime/model overrides. Profiles never fall back between installed runtimes. Live create and all bundle lifecycle/readers/receipts are removed.

Pick a distinct cluster name and explicit context. Keep CONTAINER_ENGINE=podman consistent if selected; Docker and Podman inventories are separate. kmx local down destroys all data in the named local cluster, including any historical database still present; export needed data first. Cloud cleanup has different ownership rules.

What CI proves

The workflow emits hygiene and e2e-hello-world merge gates. The latter fail-closed aggregates four retained shards: e2e-kagent-lift, e2e-orka-runtime, e2e-orka-helm and e2e-quickstart, plus successful classification. Its existing #315 skip/failure policy remains: only intentional skips are accepted, never failed/cancelled shards. Required-check rules are managed separately on GitHub; deleting a workflow job does not update a ruleset.

The v0.10.2 live-create shard e2e-kagent-create and clusterless bundle recorded Sessions/judge/reference-verify shard e2e-eval-loop are removed, along with the reusable eval action and its runner/tests. AIKit model YAML/warmup and native instruction/two-case fixtures remain native-qualifier assets, not an old bundle runner. state-paths-macos retains its independent state routing checks.

Orka runtime and stock Helm scenarios use raw upstream fixtures and independent run --agent Task/model-answer proofs. The Orka suite qualifier separately covers actual native create, full two-case eval, evaluated push and source-withheld exact-digest Lift with exact observed reuse, private captures and metadata-only artifacts. Registry/local exact-reuse and bounded absent-pair registry promotion include fresh answer Tasks with UID/count attribution. Promotion retains source-only evidence, GET-only plans, reviewed Apply/Verify, full placement readback and a no-effect model-mismatch refusal. Its offline reference uses the shared native scaffold, not removed live-create flags. Evaluation is not retried; broader promotion, absent creation and partial paths remain unqualified. Quickstart independently verifies placement before its separate fresh answer Task.

The checkout-only kagent qualification shell retains exact v0.10.3 bootstrap, raw upstream Tool smoke and a direct single A2A MCP proof, separately from tool-free packaged-suite two-case evaluation, evaluated push and source-withheld digest Lift into a fresh namespace, with occupied-repeat refusal. No legacy bundle receipts/status/judge proof is claimed. A2A proof requires one Tool call, two model calls and one terminal answer without resend. Suite Lift sends no native execution calls. Python/RemoteMCP, managed clusters and hosted providers remain unqualified.

Build the final Go state with go build -o bin/kmx ./cmd/kmx, then run bash scripts/ci/kagent-lift.sh on Linux amd64 with Docker. Explicit bootstrap stops after readiness/discovery, not execution qualification. The shell refuses an existing cluster and retains private kubeconfig/cache/Helm/registry state. Inherited Helm auth/connection overrides are cleared. Only owned temporary work and a successfully created owned cluster are cleaned; ambiguous kind creation is left for inspection. Raw payloads/logs/manifests/credentials are not artifacts. Hygiene tests these boundaries without a cluster. Final retirement Go build/live runs are required before claiming runtime qualification; docs/hermetic tests alone are not execution evidence.

Clone-free main/manual CI authors a workspace, performs full eval, evaluated layout publication and digest Lift, then reads the existing evaluation Task result without resend. It claims no separate answer Task or raw suite direct run. No Azure credentials belong in fork-exposed CI. A docs-only skip is not a live rerun.

CI registry mirrors

CI's Docker-using jobs configure the host daemon with scripts/ci/registry-mirrors.py and report pull endpoint evidence. Setup merges registry-mirrors=["https://mirror.gcr.io"] and debug=true into existing daemon configuration, preserving unrelated settings, engine version and running containers. It reloads Docker with SIGHUP rather than restarting or replacing it. Only Docker Hub routing changes: references, digest pins, manifests and source stay unchanged; GHCR and ECR are unaffected.

Component/direct-kind jobs configure nodes after cluster creation and before workloads. The pinned kind node uses containerd 2.3.4's default /etc/containerd/certs.d path: docker.io/hosts.toml keeps server = "https://registry-1.docker.io" as the origin fallback and adds https://mirror.gcr.io with pull and resolve capabilities. Setup enables containerd debug logging for the endpoint report.

Compound quickstart and clone-free journeys use a CI-only Docker-named symlink wrapper. It forwards Docker calls unchanged, then hooks a successful kind-node creation identified by reserved kind cluster and node-role labels, configuring hosts and debug logging before kind returns. This keeps full kmx local up and fetched-kind proof intact without changing KMX or kind. Clone-free CI fetches only this helper at the workflow SHA into RUNNER_TEMP; it does not check out the repository.

Reports print only successful HTTP 200 endpoint responses: endpoint, image and manifest/blob category, without headers, query strings or payloads. A cache hit is not mirror-routing evidence; successful Docker Hub fallback is identified separately. Hygiene runs scripts/test_registry_mirrors.py to test configuration merging, wrapper/node selection and evidence filtering.

Check coverage against existing image references and digests, including kindest/node and Ollama. A cache miss or mirror outage can fall back to Docker Hub; reports do not call that a mirror success. Existing Postgres ECR pins stay unchanged. Model-weight downloads from registry.ollama.ai are separate from OCI pulls and do not use these mirrors.

Invariants to preserve

1. Use platform capabilities rather than rebuilding them. Inspect the relevant upstream version first; prefer configuration, integration or an upstream contribution. Legacy-shaped helpers do not authorize expanding Kaimahi into another agent runtime.
2. Fail closed on missing proof: HTML with 200 is not a valid endpoint answer, unreadable or ambiguous state is not absent, and scanner failure is not a clean scan. Admission and dry-run acceptance are not readiness or execution evidence.
3. Keep artifacts reviewable and user-owned. Generated resource YAML remains readable, diffable and usable with the underlying tools; KMX must not hide resource ownership inside an entry point.
4. Generated files contain Secret references, never values. Keys never enter argv, logs, manifests or ConfigMaps; do not add token flags or advise users to commit credentials. Native Provider Secrets are provisioned separately; host Copilot credentials remain managed by its installed CLI. Preserve each native input contract.
5. Billed work must be recorded even when the surrounding operation fails. Historical cloud ownership records remain necessary for conservative teardown.
6. Mutations name their exact target and retain context/cloud guards; name and loopback server are independent evidence. Confirmation is scoped consent, not a general bypass, and a friendly context name does not prove disposability.
7. Repository cleanup does not authorize deleting databases, caches, credentials or deployed resources. Historical source remains available in Git history.
8. Keep one implementation of an operation. Scripts and Make targets may delegate to the native command but must not maintain a competing path or preserve retired connectors as checkout-only alternatives.
9. Metrics labels must not leak tokens or channel/user/request/delivery IDs.
10. State evidence accurately: distinguish unit and fake-service tests, continuously exercised cluster paths, one-off cloud measurements, schema validation and proposals. Label evidence as continuously tested, demonstrated, schema-valid, proposed or unbuilt. A one-off measurement is not durable live coverage; successful admission is not downstream completion, readiness is not an answer, and an old run is not a current availability claim.
11. Optimize for fewer prerequisites, a useful first result, clear recovery and less code of our own, not command count or adoption of this repository.

Troubleshooting traps

• A Provider can exist without resolving a model. Inspect Provider and Agent readiness, not only controller pod health; then prove a real Task answer.
• Setup does not adopt or redirect existing applications. Owners must deliberately remove or replace obsolete gateway references and external subscriptions.
• A Secret created after an optional mount may need a rollout restart.
• A Service port-forward selects one pod, not every replica. Use distinct fixed ports across concurrent checks; native chat allocates a free loopback port.
• Ambiguous-disconnect retries can repeat effects; read the retry limits before resending work.
• A bare shell wait waits on long-running forwards too; collect worker PIDs.
• Podman machines need checkout mounts for image builds; restarted machines may leave kind nodes stopped. The cluster step recovers named nodes and checks API/DNS.
• Python scripts supporting macOS Python 3.9 need postponed annotations before PEP 604 annotations. Ignore rules such as bin/ match at every depth; verify tracked membership and beware newly unignored files entering stash -u.

Use kmx orka status, Agent/Provider/Task conditions and context-pinned controller logs. Never paste live infrastructure IDs into evidence: scan shapes and manually redact names too.