merchant-stack-setup skill is the bring-up from the
evaluation stack and the production architecture
pages, written as a checklist for a coding agent. Give it to the agent that stands up the
stack for you. Every step is Run → Check → Expect. A table at the end maps each
known failure to its check and fix.
It covers:
- Evaluation track, Docker Compose on one machine: the organization in the console, the join
token,
.env,docker compose up, confirming the engine’s keys, the signer pairing, the policy signed with key B, the first wallet, the first signed payout, a restart check and the teardown. - Production track, the Helm chart or the VM installer: the same steps on separate engine and signer hosts, plus the signer bundle, custodian keys and backups before any funding.
Connect it to an agent
The skill is one file,SKILL.md: the block below. Give the agent that file unchanged, front
matter included: the name and description fields tell the agent when to load the skill.
- Claude Code: save it as
.claude/skills/merchant-stack-setup/SKILL.mdin the project, or as~/.claude/skills/merchant-stack-setup/SKILL.mdfor every project. - Other agents: import the file as a skill, or put its text in the agent’s instructions.
kubectl and helm (or SSH to the VMs). It does not need the
console password or the join token in its instructions: the token stays in a 0600 file
on that machine. Fill the placeholders at the
top of the file (<console-url>, <api-url>, <registry>, <version>,
<chart-repo-url>, <download-url>) with your values, or give them to the agent.
The skill
Use the copy button on the block to take the whole file.SKILL.md
---
name: merchant-stack-setup
description: Step-by-step bring-up of a Citadel merchant stack on the merchant's own infrastructure (the seed's OpenBao, the engine's OpenBao, policy-engine = key B, signer), from a new organization in the Citadel console to the first derived wallet and the first signed payout. Two tracks — evaluation (Docker Compose on one machine) and production (the citadel-merchant Helm chart, or the citadel-vm installer on Linux VMs). Use when asked to stand up, verify, re-pair, restart or tear down a merchant stack.
---
# Merchant stack setup
Every step is: **Run** → **Check** → **Expect**. Do them in order. Do not skip a Check.
If a Check does not give the Expect, go to [If a step failed](#if-a-step-failed).
Placeholders:
| Placeholder | Meaning |
|---|---|
| `<console-url>` | the Citadel console |
| `<api-url>`, `<nats-url>` | the platform's API and NATS addresses, given with the organization |
| `<registry>`, `<version>` | where you pull Citadel's images from, and the release; images are `<registry>/citadel-openbao:<version>`, `<registry>/citadel-policy-engine:<version>`, `<registry>/citadel-signer:<version>` |
| `<chart-repo-url>` | the Helm repository of the `citadel-merchant` chart |
| `<download-url>` | where Citadel's public binaries are published |
| `<slug>` | a short name for this stack, e.g. `fabrikam` |
| `<org>` | the org token the engine logs, e.g. `org-60a2d4ae380f` |
| `<hex>` | a 64-character hex public key |
Never paste a join token, an invite link, a signer bundle, a recovery key, a seed share or an
OpenBao token into a chat, ticket or doc: write `ojt_…`, `inv_…`. Stop and hand over to a
human where the steps say so: a fingerprint mismatch, funding a wallet, the custodians.
## Evaluation track (Docker Compose, one machine)
Demo unseal mode only: anyone who reads the volumes owns the seed. Never fund wallets of this
stack with real money; use a test network.
### 0. Preflight
1. **Run** `docker compose version`
**Expect** Compose v2 (`Docker Compose version v2.…`).
2. **Run** `docker pull <registry>/citadel-openbao:<version>`, the same for
`citadel-policy-engine` and `citadel-signer`.
**Expect** all three pulled, one `<version>`.
3. **Run** `docker compose ls`
**Expect** the projects already on the machine. Pick a `<slug>` whose `merchant-<slug>` is
not in the list. Never run a compose command against another project.
4. **Run** create a directory `merchant-<slug>` and put two files in it: `compose.yml` and
`engine-policy.hcl` from the Citadel docs page *Evaluation stack (Docker Compose)*,
section *Start the stack*. In `compose.yml` set the images to
`<registry>/citadel-openbao:<version>`, `<registry>/citadel-policy-engine:<version>` and
`<registry>/citadel-signer:<version>`. Change nothing else. From that directory:
```bash
dc() { docker compose -p merchant-<slug> --env-file .env -f compose.yml "$@"; }
```
Every compose command below is `dc …`, run from that directory. Redefine `dc` in every new
shell.
5. **Run** `command -v xxd openssl || command -v python3`
**Expect** a path: step 5 hashes keys with `xxd` and `openssl`, or with `python3`.
### 1. Sign in to the organization (console, owner)
1. Citadel creates the organization and gives its owner an **owner invite link** (valid
72 h) and a **join token** (`ojt_…`, shown once). Keep the token in a file with mode
`0600` outside any repository; never commit it.
2. **Run** open the invite link, set full name and a password (≥ 12 characters),
**Join <org name>**, then sign in at `<console-url>`.
**Expect** the sidebar shows the organization and `Owner`.
### 2. Provision the tenant and check the join token (console, owner)
1. **Run** **Settings → Signer onboarding** → **Connect the org** → **Provision tenant**.
**Check** the panel. **Expect** *Tenant account* `org-…`, not `not provisioned`.
2. Join token: the one from step 1, or **Mint join token** here (shown once).
**Check** the token row. **Expect** `token live`. It expires 24 h after minting and is
redeemable once by the engine and once by the signer.
### 3. Fill `.env`
**Run** write `merchant-<slug>/.env`, mode `0600`:
```ini
CITADEL_API=<api-url>
CITADEL_JOIN_TOKEN=ojt_…
# Only when the NATS address in the join response is not reachable from your containers:
NATS_URL=<nats-url>
```
### 4. Start the stack
1. **Run** `dc up -d`, wait ~30 s, then `dc ps --format '{{.Service}}\t{{.Status}}'`
**Expect** `openbao`, `openbao-engine`, `bao-init`, `bao-init-engine` healthy;
`policy-engine`, `signer` `Up`, not `Restarting`.
2. **Run** `dc logs policy-engine | grep -E '"joined"|verification key|key A|key B|policy document'`
**Expect**, in this order:
```text
{"msg":"joined","org":"org-60a2d4ae380f","org_id":"60a2d4ae-…","nats_url":"nats://…",…}
{"msg":"engine verification key (signer pins this)","pub":"7613a44a…","fingerprint":"b955 b4b1 … 9493"}
{"msg":"request key A (…)","pub":"48bada9a…"}
{"msg":"policy key B (…)","pub":"6ff210d1…"}
{"level":"WARN","msg":"no policy document yet — every payout will be refused until the console publishes one"}
```
The WARN is expected until step 7. Write down the three `pub` values and the engine
`fingerprint`.
### 5. Confirm the engine's keys (Pending → Confirmed)
1. **Run** the fingerprints of keys A and B (the engine logs them as hex only). The
fingerprint is the SHA-256 of the 32 raw key bytes, in groups of four hex characters:
```bash
fp() { printf '%s' "$1" | xxd -r -p | openssl dgst -sha256 -r | cut -c1-64 | sed 's/..../& /g; s/ $//'; }
# without xxd/openssl:
# fp() { python3 -c 'import sys,hashlib;h=hashlib.sha256(bytes.fromhex(sys.argv[1])).hexdigest();print(" ".join(h[i:i+4] for i in range(0,64,4)))' "$1"; }
fp <key A pub>; fp <key B pub>
```
2. **Run** console **Settings → Policy engine** → **Trust anchors**.
**Check** three rows `Pending`: *policy engine*, *request key A*, *policy key B*.
**Expect** each fingerprint equals the engine log (`fingerprint` field) or your `fp`
output, character for character. Any mismatch: stop, do not confirm.
3. **Run** **Confirm** on each row, then **Confirm** in the dialog.
**Expect** `3 of 3 confirmed`.
### 6. Check the signer pairing
1. **Run** `dc logs signer | grep -E 'connected|pinned|bao key|serving'`
**Expect**:
```text
msg=connected org=org-60a2d4ae380f …
msg="engine key pinned" fingerprint="b955 b4b1 … 9493" engine_url=http://policy-engine:9800
msg="bao key present" mount=multichainsign key=org
msg=serving org=org-60a2d4ae380f sign=citadel.<org>.signer.sign derive=citadel.<org>.signer.derive engine_key="b955 b4b1 … 9493" …
```
The pinned fingerprint must equal the engine key's fingerprint in step 4 and in the
console. On later boots the line is `engine key already pinned`.
2. **Check** console **Settings → Policy engine → Signer**.
**Expect** *Engine key* `pinned on your side`, *Wallets derived* `0`. Onboarding reads
`2 of 5` here; that is correct until the first wallet.
### 7. Sign and publish the policy with key B
1. **Run** console **Settings → Policy document** → **Network rules** → add rules for the
network you will test on. Example, TRON: **Add TRON rules** → **Recognised assets →
Asset**: `native` / `TRX/6` → **Transaction ceilings → Ceiling**: `TRX/6`, `100` →
**Save draft v1**.
2. **Run** **Download** → `policy-v1.yaml`, save it in `merchant-<slug>/`.
**Expect** (TRON example):
```yaml
---
ceiling:
TRX/6: '100000000'
chains:
tron:
assets:
native: TRX/6
chain_module_version: 1
wallets: {}
whitelist: []
references: {}
version: 1
```
3. **Run** (flags **before** the file name):
```bash
dc cp policy-v1.yaml policy-engine:/state/policy-v1.yaml
dc exec policy-engine /citadel policy-engine sign-policy \
--bao-addr http://openbao-engine:8200 --bao-token-file /engine/bao.token /state/policy-v1.yaml
```
**Expect** `signed /state/policy-v1.yaml with policy key B 6ff210d1…`, a 128-hex
signature, `bootstrap wrapper written: /state/policy-v1.signed.json`. The key B prefix
must match key B's `pub`.
4. **Run** paste the signature into the console field *Signature, hex or base64* →
**Publish v1**.
**Expect** page header `v1 in force`.
5. **Run** `dc logs --since 2m policy-engine | grep 'policy document accepted'` and
`dc exec policy-engine /citadel policy-engine check-policy --engine http://policy-engine:9800`
**Expect** `policy document accepted`, and `POLICY DOCUMENT version 1`,
`chains ruled tron`, `CEILING … TRX/6 100000000`.
### 8. Derive the first wallet
1. **Run** console header network switch (`All networks`) → the test network your policy
covers. Never Mainnet with this stack.
2. **Run** **Vaults and wallets** → **Issue wallet** → tab **New vault** → name `Payouts`,
tier **Hot** → **Create**. The dialog returns to **Issue wallet**: click the vault pill
**Payouts Hot** (**Issue** stays disabled until a vault is picked) → network **TRON** →
*Identifier* `payouts-tron`, *End user* `treasury` → **Issue**.
3. **Run** `dc logs --since 1m policy-engine signer | grep -E 'forwarded|derived'`
**Expect**:
```text
policy-engine-1 | {"msg":"authorized and forwarded","org":"org-…","kind":"keygen","subject":"citadel.org-….signer.derive"}
signer-1 | msg="signer.derive derived" correlation_id=derive:<wallet_id> wallet_id=<wallet_id> …
policy-engine-1 | {"msg":"derived and registered","org":"org-…","wallet_id":"…","family":"tron","address":"T…"}
```
4. **Check** **Settings → Signer onboarding** and **Settings → Policy engine**.
**Expect** Onboarding `5 of 5`; *Engine key* `pinned`; *Wallets derived* `1`;
*Policy document* `In force v1`. The org badge no longer reads `onboarding`: the first
derived wallet makes the organization active.
### 9. First signed payout (needs funds)
1. The agent does not fund. Ask the human operator to send a small amount (≤ the ceiling,
e.g. `10` TRX) of the test network's coin to the address from step 8, and wait for
their answer.
2. **Run** after funding: create a payout in the console from `Payouts` to an owned
address (an external address needs a request signed with key A, or a whitelist entry
in the policy).
3. **Run** `dc logs --since 5m policy-engine signer | grep -E 'kind":"sign|signer.sign'`
**Expect** `"authorized and forwarded" … "kind":"sign"` and
`msg="signer.sign signed" correlation_id=… tx_id=… chain=tron`.
`refused` with a reason → the policy refused it: read the reason, fix the policy (new
version, step 7), retry.
### 10. Restart check (idempotency)
**Run** `dc restart policy-engine signer`, then `dc logs --since 15s policy-engine signer`
**Expect** no `joined`, no `engine key pinned`:
```text
signer-1 | msg="already connected" org=org-…
signer-1 | msg="engine key already pinned" fingerprint="…"
policy-engine-1 | {"msg":"tenant from join state","org":"org-…",…}
policy-engine-1 | {"msg":"owned wallets loaded","count":1}
policy-engine-1 | {"msg":"policy document loaded","version":1,"source":"journal"}
```
### 11. Tear down (only this project)
```bash
docker volume ls --filter label=com.docker.compose.project=merchant-<slug> --format '{{.Name}}'
docker volume inspect merchant-<slug>_openbao-data # check before you drop it
dc down -v
```
**Expect** four volumes listed (`_engine-state`, `_merchant-state`, `_openbao-data`,
`_openbao-engine-data`), all removed. `down -v` destroys the seed and the engine keys for
good. `down` without `-v` keeps them; the next `up` rejoins nothing.
## Production track (Helm; VM variant inline)
Separate host roles, a three-node OpenBao cluster per role, the `awskms` seal, a signer that
accepts no inbound connections. Reference: the Citadel docs page *Production architecture*.
### P0. Prerequisites
- **Kubernetes**: two clusters, one per role (or at least separate node pools with taints
and no shared nodes), Kubernetes ≥ 1.27, a CNI that enforces NetworkPolicy, Helm 3. The
engine role needs a `ReadWriteMany` storage class with working POSIX locks (EFS, Filestore,
Azure Files NFS).
- **VMs instead**: three Linux hosts per role with bash, systemd, docker, nft and openssl; an
NFSv4 share mounted on the three engine hosts.
- **AWS KMS**: one symmetric key per role, multi-region, deletion denied in the key policy.
On Kubernetes an IAM role per role for IRSA, on VMs the instance role; each with only
`kms:Encrypt`, `kms:Decrypt`, `kms:DescribeKey` on its key. VPC endpoints for KMS and STS
in the signer's segment.
- **TLS**: one CA per cluster, a certificate per OpenBao node (or cert-manager on
Kubernetes).
- **Network**: the matrix of the *Production architecture* page. The signer segment accepts
only SSH from your corporate VPN.
- **Artifacts**:
- Helm: `helm repo add citadel <chart-repo-url> && helm repo update`, then
`helm show values citadel/citadel-merchant --version <version>`.
- Images: mirror `<registry>/citadel-openbao:<version>`, `citadel-policy-engine` and
`citadel-signer` into the registry your clusters pull from.
- VMs: download `citadel-vm` and `SHA256SUMS` from `<download-url>/<version>/`, check
`grep citadel-vm SHA256SUMS | sha256sum -c -`. On the bastion, `docker pull` the role's
two images and `docker save -o citadel-images.tar <images…>`; copy the tar to each host
over SSH (the signer hosts have no route to a registry).
- `citadel-recovery` for every custodian, from the same `<download-url>/<version>/`,
checked the same way.
### P1. Organization, tenant, token
Steps 1–2 of the evaluation track.
### P2. Signer bundle
Owner: **Settings → Policy engine → Signer bundle → Issue bundle**, then **Download** once
(`signer-bundle-<org>.json`, valid 24 h). Deliver it to the signer side over a corporate
channel; mode `0600`.
**Expect** the bundle row `downloaded`. *This environment issues no NATS users for the
signer*: stop and report to Citadel.
### P3. Custodians
Five GPG public keys, base64: `custodian-1.b64` … `custodian-5.b64`
(`gpg --export <id> | base64 > custodian-<n>.b64`).
### P4. Engine role
- Helm: in namespace `citadel-engine` create the TLS Secret (`citadel-engine-openbao-tls`:
`tls.crt`, `tls.key`, `ca.crt`) and the join token Secret. The token never goes on a
command line: `install -m 600 /dev/null join.env`, put `CITADEL_JOIN_TOKEN=ojt_…` in it with
an editor, then
```bash
kubectl -n citadel-engine create secret generic citadel-engine-join --from-env-file=join.env
shred -u join.env
helm install citadel-engine citadel/citadel-merchant --version <version> -n citadel-engine \
-f engine-host.yaml \
--set-file custodians.custodian-1=custodian-1.b64 --set-file custodians.custodian-2=custodian-2.b64 \
--set-file custodians.custodian-3=custodian-3.b64 --set-file custodians.custodian-4=custodian-4.b64 \
--set-file custodians.custodian-5=custodian-5.b64 --wait
```
Start `engine-host.yaml` from the example on the *Production architecture* page, with
your registry, KMS key, IAM role, NATS and API addresses and CIDRs.
- VM: `sudo ./citadel-vm install engine engine-host.conf --images citadel-images.tar` on the
initializer (`INITIALIZER=true`), then on the other two. Start `engine-host.conf` from the
example on the same page.
**Check** Helm: `kubectl -n citadel-engine logs -l app.kubernetes.io/instance=citadel-engine,app.kubernetes.io/component=engine | grep -E '"joined"|verification key|key A|key B'`;
VM: `journalctl -u citadel-engine | grep -E '"joined"|verification key|key A|key B'`.
**Expect** the `joined` line and the three keys, as in evaluation step 4.
Then copy `recovery-keys.json` off node 0
(`kubectl -n citadel-engine exec citadel-engine-openbao-0 -c bao-init -- cat /bao/state/recovery-keys.json > recovery-keys.json`;
VM: `sudo cat /var/lib/citadel/bao-init/recovery-keys.json`), and delete the join token
Secret (VM: the engine deletes its copy once joined; delete the file `JOIN_TOKEN_FILE` names).
### P5. Confirm the keys
Evaluation step 5. **Expect** `3 of 3 confirmed`.
### P6. Signer role
The engine key `pub` (64 lowercase hex) goes in `signer.engineKey` (VM: `ENGINE_KEY`).
- Helm: in `citadel-signer` create the TLS Secret and
`kubectl -n citadel-signer create secret generic citadel-signer-bundle --from-file=signer-bundle.json=signer-bundle-<org>.json`,
then `helm install citadel-signer citadel/citadel-merchant --version <version> -n citadel-signer -f signer-host.yaml`
with the same five `--set-file custodians.…` and `--wait`.
- VM: `sudo ./citadel-vm install signer signer-host.conf --images citadel-images.tar`
with `ENGINE_KEY` and `SIGNER_BUNDLE`, the initializer first.
**Check** `kubectl -n citadel-signer logs citadel-signer-openbao-0 -c bao-init -f | grep -m1 'ready; watching the seal'`
(VM: `journalctl -u citadel-bao-init` on the initializer), then the signer logs.
**Expect** `bao-init: root token revoked`, `bao-init: ready; watching the seal`; every
signer replica logs `connected from signer bundle` and `serving` with the engine key's
fingerprint. Then copy `recovery-keys.json` and `seed-shares.json` off the initializer
(`/bao/state/` in `citadel-signer-openbao-0`; VM: `/var/lib/citadel/bao-init/`), hand the
shares out with `citadel-recovery split-shares seed-shares.json`, and delete the bundle
Secret.
### P7. Back up and verify before any funding
All required: a raft snapshot of both clusters restored on a separate node (VM:
`sudo citadel-vm snapshot`); the KMS key multi-region with deletion denied; each custodian
ran `citadel-recovery check-share`; three custodians ran
`citadel-recovery verify --fingerprint <hex>` offline. These are human steps: the agent
lists what is done and what is missing, and stops until a human confirms all four.
### P8. Policy
Evaluation step 7, signed inside the engine's container, which already holds the OpenBao
login (`BAO_ADDR` and `BAO_K8S_ROLE`, or cert auth, in its environment). The engine image has
no shell, so the policy goes in on stdin:
- Helm:
```bash
kubectl -n citadel-engine exec -i deploy/citadel-engine-engine -- \
/citadel policy-engine sign-policy --out /dev/null /dev/stdin < policy-v1.yaml
```
- VM, on an engine host:
```bash
docker exec -i citadel-engine /citadel policy-engine sign-policy --out /dev/null /dev/stdin < policy-v1.yaml
```
**Expect** `signed /dev/stdin with policy key B <prefix>…` and the signature; publish it as
in evaluation step 7.4, then `check-policy` in the same container
(`… -- /citadel policy-engine check-policy`).
### P9. First wallet, first payout
Evaluation steps 8–9 on the network your policy covers. **Expect** `derived and
registered`, Onboarding `5 of 5`, then `signer.sign signed`.
## If a step failed
| Symptom | Check | Fix |
|---|---|---|
| Engine `join … answered 422: {"error":"not_provisioned"}`, signer `discovery: http 422 (token spent or expired?)`, both `Restarting` | Onboarding → Connect the org: *Tenant account* `not provisioned` | **Provision tenant**. The token is not spent by this refusal (row stays `token live`); both containers join on their next restart, no `.env` change. |
| Engine `join token unknown, expired, or already used by this engine` | token row `expired`/`redeemed`; age > 24 h | **Mint join token**, put it in `.env`, `dc up -d`. A component that already has state never reads the token again. |
| Engine restarts with `ensure key engine: bao 403 permission denied` | `dc logs bao-init-engine`; is `engine-policy.hcl` mounted over `/openbao/signer-policy.hcl` in `bao-init-engine`? | Use `compose.yml` and `engine-policy.hcl` from the docs unmodified; `dc up -d bao-init-engine`, then `dc restart policy-engine`. |
| Signer stuck on `waiting for pairing code` | `dc logs policy-engine | grep 'pairing code'` — code is valid 15 min from engine boot | `dc restart policy-engine`, the signer pairs on the fresh code. |
| Signer `nats connect failed` / no `nats connected` | `.env` `NATS_URL`; can the containers reach it? | Set `NATS_URL` to the NATS address Citadel gave you, `dc up -d`. |
| Console fingerprint ≠ engine log | step 5 `fp`, Trust anchors row | **Stop.** Confirm nothing, change nothing. Report both fingerprints (console row and engine log) to a human and wait. Re-pairing does not fix this. |
| Signer pin ≠ engine log | step 6 `engine key pinned` / `already pinned` vs step 4 `fingerprint` | **Stop and report** both fingerprints. Re-pair only when both hold: (1) the engine key was replaced on purpose — you recreated `merchant-<slug>_engine-state` / `merchant-<slug>_openbao-engine-data` while `merchant-<slug>_merchant-state` stayed; (2) a human said yes. Re-pair: `docker run --rm --user root --entrypoint rm -v merchant-<slug>_merchant-state:/state <registry>/citadel-openbao:<version> /state/signer/engine-key.json`, then `dc restart policy-engine signer`. Then step 5 again: new keys `Pending` → compare → **Confirm**; step 6 **Expect** `engine key pinned` with the engine's new fingerprint. |
| `sign-policy` prints only its usage line | file name placed before the flags | Put every flag before the file name. |
| Publish refused | signature from another key | The `signed … with policy key B <prefix>` must match key B in **Trust anchors**, and key B must be `Confirmed`. Re-sign the exact downloaded bytes. |
| Every payout `refused`: no policy document | `check-policy` | Steps 7.3–7.5. |
| **Issue** stays disabled in Issue wallet | is the vault pill pressed? | Click the vault pill; fill *Identifier* and *End user*. |
| Onboarding stuck at `2 of 5` after keys and pairing | *Wallets derived* `0` | Expected. Derive the first wallet (step 8). |
| Production signer exits `production mode: --pe-public-key is required` | `signer.engineKey` / `ENGINE_KEY` | Set the engine key `pub` from P4 and install again. |
| Production signer `signer bundle expired at …` | bundle age > 24 h before first start | Issue and download a new bundle (P2), replace the Secret / `SIGNER_BUNDLE`, restart. |