To try the bring-up order on one machine first, use the evaluation stack. This page is the production setup.
A coding agent can follow this setup with the merchant stack setup skill, production track.

​
Requirements

​
Two host roles

Production has two roles. Each role is its own set of machines in its own network segment. The two roles never share a machine, a cluster, a volume or a network segment. Nothing on the engine side can reach the signer side, and the reverse. They meet only on the platform’s NATS: the engine publishes signed verdicts, and the signer answers with signatures.

​
Why the engine is active/standby

The engine’s journal counts every authorization and decides limits from those counts, in one transaction. Two engines writing two journals would each see half the spend, and a limit would let through twice its amount. Only one engine instance serves at a time. The standby takes over when the active one stops holding its lease. Both read keys from the same OpenBao cluster, so a failover does not change any public key.

​
The lease

The lease is one KV v2 key in the engine’s own OpenBao cluster, not a Kubernetes Lease: the engine already reaches that cluster, so nothing else is needed for it, and the same mechanism works on virtual machines. Writes are compare-and-set on the key’s version (options.cas), so two engines can never both win.
The lease goes through the same OpenBao client as the keys, so it works with either way the engine logs in: a token (BAO_TOKEN, BAO_TOKEN_FILE) or kubernetes auth (BAO_K8S_ROLE), including the one retry with a fresh login on a 403. The Helm chart turns it on. Its bao-init runs with CITADEL_ENGINE_LEASE=citadel-engine/lease: it mounts citadel-engine/ (KV v2) and writes the citadel-engine policy with the path above added. Every engine runs with ENGINE_LEASE=citadel-engine/lease. Without that variable bao-init writes the engine’s policy without the path: the evaluation stack runs one engine without a lease. The record is {holder, epoch, journal_epoch, renewed_at}. How it moves:
  • Standby. Polls the record every TTL/5. It takes over when the record’s version has not changed for TTL by its own clock, or at once when the holder is empty (released). No clock is compared between machines. GET /policy answers {"status":"standby"}.
  • Active. Renews every TTL/5. It counts its deadline from the moment it sent its last renewal that landed, and exits at 2/3·TTL without one. That is before any standby can start. A renewal that finds the record changed by someone else also makes it exit. Before every authorization the journal asks whether the lease is still held.
  • Clean stop. On SIGTERM the active finishes in-flight messages (up to 10 s), closes the journal and releases the lease. The standby takes over without waiting the TTL.
  • Takeover. The new holder joins the platform if the state dir has no join state yet, opens the journal and only then subscribes to NATS. Joining happens under the lease because a join token redeems once. PAIRING_CODE_FILE is written only by the holder, right after it takes the lease: the file sits in the shared state dir, and a standby writing there would overwrite the active’s code with its own.

​
The journal moves with the lease

The standby does not have its own journal. Both engines mount one state dir (join state, NATS credentials, journal.db), and only the lease holder opens it:
  • The holder opens the journal with locking_mode=EXCLUSIVE. SQLite holds the file lock until the process exits, so an old active that is still alive can neither write nor read. In this mode SQLite also keeps the WAL index in process memory, not in a -shm file. That makes a journal on a network volume usable when one machine at a time opens it.
  • Every holder gets the next epoch and stamps it into the journal (table lease_fence). An authorization checks the stamp inside its own transaction. Once a newer engine has stamped the file, an older one cannot record there.
  • Before stamping, the new holder checks the journal continues the previous one: its stamp must be at least the record’s journal_epoch (the last confirmed stamp) and at most the previous holder’s epoch. A journal that missed a predecessor’s writes would serve with every limit refilled. The engine logs missed authorizations, releases the lease and exits, so an engine with the right journal can take over.
What this asks of the deployment: Two machines each with its own disk is not a standby: the second engine takes the lease and refuses to serve from its own journal. On NFS, failover takes max(TTL, NFS lease), not just the TTL. When the old host vanishes together with its NFS client, the server keeps that host’s POSIX lock on journal.db until its NFSv4 lease expires (about 90 s by default). Until then the new holder fails on OpenExclusive after the 5 s busy timeout, releases the lease and exits; it, or the other engine, tries again after a restart. The engine fails closed throughout: nobody serves until the lock is free. The signer keeps little state: the pinned engine key (engine-key.json), and what the bundle gave it (state.json, nats.creds, nats-ca.pem). No signing key is among them.

​
Signer replicas

More than one signer can serve the same org. Every replica subscribes to citadel.<org>.signer.sign and citadel.<org>.signer.derive in the NATS queue group citadel-signer, so NATS hands each request to one replica and it is signed once. There is no leader and no lease: a replica that dies stops getting requests, and the others carry on. In production every replica starts from the same bundle and the same --pe-public-key, and reaches the same signer OpenBao cluster. A request can still arrive twice: the platform’s stream redelivers an operation nobody acknowledged, and the engine answers it with a fresh verdict for the same digest. Each replica remembers what it answered, by correlation_id, for 10 minutes and up to 10 000 answers: The verdict bytes are not part of the comparison: the engine answers a replayed operation with a fresh verdict for the same digest, and that is still the same request. Replicas do not share this memory. A repeat that lands on another replica is signed again, over the same digest the verdict bound. Both signatures are for one transaction, and a chain does not accept one transaction twice, so the second signature moves no money. The evaluation stack runs one signer next to its own OpenBao. The Helm chart runs the signers (signer.replicas, default 2) as their own pods against the OpenBao cluster of OpenBao clusters. Each replica keeps its own state volume and applies the bundle on its first start.

​
Network matrix

Every connection is opened by the side on the left. The only connection ever opened toward the signer-host segment is SSH from the corporate VPN. What is not in the table is the point:
  • No path between engine-host and signer-host, in either direction.
  • No HTTP from the signer to the platform. The signer reads its NATS bundle from a file, see below.
  • No inbound port on the signer host other than SSH. SSH accepts connections only from the corporate VPN, through a bastion; every other source is dropped at the segment’s firewall. If the signer role runs on Kubernetes, its control plane sits inside the segment and operators reach it the same way, through the bastion. This part is yours: the firewall of the segment closes SSH to everything but the VPN, and the kubelet and any NodePort to everything outside the segment. The chart’s NetworkPolicy covers only the pods’ traffic, see The network. On VMs the installer’s own firewall enforces it on each host (The firewall).
  • The signer’s OpenBao listens on 8200 and 8201 for the signer and for its own peers, inside the segment only. Health checks run locally.
  • No scrape of the signer or the engine. Neither serves /metrics; both push, see Telemetry.
  • The policy engine’s HTTP port 9800 (/healthz, /policy, pairing) binds to 127.0.0.1:9800 unless ENGINE_HTTP_ADDR says otherwise. The pairing endpoint is still served there, but only a process on the engine’s own host or pod can reach it, and a production signer never calls it: it pins from --pe-public-key. The evaluation stack sets ENGINE_HTTP_ADDR=:9800 so the signer next door can pair; there the endpoint is open to the stack’s network.

​
NATS over TLS

Both clients dial NATS with their creds and, over TLS, with up to three PEM files: the CA the server’s certificate chains to, and a client certificate with its key for a server that verifies clients.
  • The platform sends its CA. When the platform’s bus uses a private CA, the engine gets it with its creds at join and the signer with its bundle (or at discovery); each keeps it as nats-ca.pem in its state directory and uses it when NATS_CA / --nats-ca is unset.
  • Without a CA the server’s certificate is checked against the system roots.
  • --nats-ca wins over the bundle’s CA. The bundle’s CA stays in the state directory and is used again when the flag is dropped.
  • Any of the three makes TLS required, whatever the URL’s scheme: a nats:// URL to a server without TLS fails instead of falling back to plaintext.
  • Checked at start. A certificate without its key, an unreadable file, a CA file without a PEM certificate or a key that does not match its certificate stops the process before it touches OpenBao or the join token. The engine exits; the signer exits with the reason. A handshake the server refuses (wrong CA, no client certificate) is a connect error: the engine exits, the signer logs nats connect failed, retrying.
  • TLS 1.2 at least. The server name checked is the host of the NATS URL.
  • The client certificate is not in the bundle. The bundle authenticates the signer to NATS with its creds. A client certificate is for a platform NATS that also requires mutual TLS; the owner issues it and puts it on the host beside the bundle.

​
Telemetry

The signer and the engine push their metrics and logs to your OpenTelemetry collector. Nothing listens for telemetry and nothing is scraped. The push is OTLP over HTTP with JSON bodies, to the collector’s OTLP/HTTP receiver (port 4318). OTLP over gRPC (4317) is not supported: both components speak only this one protocol, without the OpenTelemetry SDK, so the signer keeps its short dependency list. Turn it on with one variable, on either component, or with --otlp-endpoint on citadel policy-engine up and citadel signer up:
The component then POSTs metrics to <endpoint>/v1/metrics and logs to <endpoint>/v1/logs. Without the variable it pushes nothing, and its logs go to its own output only, as before. With it, every log line still goes to the local output too. How the push behaves:
  • Logs are batched and sent within a second, or as soon as 512 are waiting. At most 4096 wait; past that, new lines are dropped from the push, never from the local output. A batch the collector refuses or cannot take is dropped, not retried, and counted.
  • Metrics are cumulative, so a lost push loses no count: the next one carries the totals.
  • A dead collector never stops a component. The first failed push logs telemetry push failing locally, once per signal, and recovery logs telemetry push recovered. A redirect from the collector is not followed, so the headers never reach another host; it counts as a failed push.
  • On exit the component pushes what is queued, for at most 3 seconds. The one exception is an engine that lost its lease: it exits at once, because the next holder is waiting for the journal.
Metrics: Worth an alert: citadel.signer.requests{result="failed"} above zero, any publish.errors, nats.connected at 0, policy.version at 0, and no engine with lease.active at 1. A rise in result="refused" on the signer means verdicts that do not check out: a wrong pinned key, or someone other than your engine asking. Collecting all of it, the metrics derived from the logs, a dashboard and example alerts: Monitoring. Logs carry the same fields as the local output, as attributes. One line is never pushed: the engine’s pairing code goes to its stdout only, so reading the collector does not let anyone spend the code before the signer does.

​
OpenBao clusters

Each role has its own cluster: three OpenBao nodes with integrated storage (raft), on three machines of that role. The signer’s cluster holds only the seed. The engine’s cluster holds only the engine’s three keys. The two clusters are never shared. A shared cluster would put the seed and key B behind one KMS key, one set of recovery keys and one root, so one compromise would hold both halves of the money path. It would also need traffic between the two segments, which requirement 7 forbids. Each node runs the openbao image with CITADEL_BAO_CONFIG=cluster. The image’s entrypoint renders the node’s config into CITADEL_BAO_RUN_DIR (default /tmp/citadel-openbao), puts the KMS plugin declarations beside it, and starts bao server on that directory. Nothing node-specific is baked into the image; it all comes from the environment: The certificate names the node’s API host and 127.0.0.1, which the node’s own bao-init uses. One CA per cluster signs every node. Rendered for signer-bao-1, with the other two nodes as its peers:
config.hcl (signer-bao-1)
  • The seal takes nothing from the file. The awskms plugin reads AWS_REGION, VAULT_AWSKMS_SEAL_KEY_ID and AWS_KMS_ENDPOINT from the server’s environment, which it inherits. Credentials come from the node’s role: on Kubernetes, IRSA (AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, injected through the service account’s eks.amazonaws.com/role-arn annotation). No access key and no secret is in a file or in the image. A server whose KMS key cannot be described does not start.
  • disable_mlock = true. Raft maps its database into memory, and mlock would pin all of it. OpenBao’s advice for integrated storage is to leave mlock off and keep swap off on the host.
  • performance_multiplier = 1. A lost leader is replaced within seconds. With the default (5) a full restart waited about 20 seconds for an election.
  • Port 8210 is loopback only. It exists for generate-root after the root token is revoked (Unseal and recovery). On 8200 that call stays refused: anyone on the network could cancel a root generation in flight. On 8210 anyone inside the pod could, so a node’s pod holds only openbao and bao-init, and the custodians check every attempt (After the root token is gone).
  • The audit device is in the config. OpenBao takes audit devices from the config file only. A device that cannot write blocks requests, so /openbao/audit goes on a volume, and its log is shipped off the node.
Use static retry_join addresses. Cloud auto-join (auto_join = "provider=aws …") would need the cloud’s instance API, which is one more egress from the signer segment. A joined node starts as a non-voter. Autopilot promotes it once it has been stable for a few seconds. Until all three are voters, losing the first node loses quorum. Raft is also the only storage of the image. OpenBao 2.7 removed the file backend, so the evaluation stack runs raft with one voter. A volume written by the 2.6 image is moved to raft when the server starts, by the image’s entrypoint: bao operator migrate copies the encrypted entries as they are, so the unseal keys, the root token and the seed stay the same. The old file tree is kept in /openbao/data/file-backend.migrated. OpenBao 2.8 drops file from bao operator migrate as well, so such a volume has to start once on a 2.7 image first.

​
What survives what

Back up each cluster with bao operator raft snapshot save: once right after init and the seed’s birth, as a required step of the bring-up, and then on a schedule (on VMs, the snapshot timer). A snapshot is encrypted by the vault. Restoring it needs the same KMS key, so a snapshot and its KMS key are useless apart. The mnemonic does not need the snapshot. Any three SLIP-39 shares rebuild the seed’s mnemonic on an offline machine, with citadel-recovery restore or any other tool that implements SLIP-0039. Neither OpenBao’s data nor the KMS nor the platform is involved. A rebuilt seed is burnt, though: there is no import, so it only moves the funds to a new seed. The snapshot and its KMS key are what bring the service back, after lost nodes or a move, without evacuating the funds.

​
Unseal and recovery

With a KMS seal, nobody types unseal shares. The vault asks the KMS to decrypt its root key on every start. That is how requirement 4 holds: a restart needs nobody. “3 of 5” means the seed. At setup, five custodians each hand over a GPG public key. The seed’s mnemonic is split into SLIP-39 shares, one sealed to each key. Any three custodians can rebuild the mnemonic. That is the scheme requirement 5 asks for. OpenBao still makes a Shamir split at init time. With a KMS seal it is a set of recovery keys, not unseal keys. They stay as a technical mechanism, sealed to the same five GPG keys.

​
The init step

The init step is the image’s bao-init with CITADEL_UNSEAL_MODE=kms, a sidecar of every node: On the initializer, on first start:
  1. The custodians’ keys are tried first. A failed bao operator init is not undone. OpenBao bootstraps raft and the barrier before it seals the recovery keys, so a key it cannot encrypt to would leave a vault that is initialized and that nobody holds a key or a token for. The sidecar therefore runs the same init with -pgp-keys on a throwaway in-memory server on 127.0.0.1:8299. If a key is unusable, it stops with FATAL: the custodians' keys cannot be used for the recovery keys — nothing was initialized and the reason.
  2. Init.
    The recovery keys, each sealed to one custodian, go to $STATE_DIR/recovery-keys.json (mode 0600). The root token is not in it. It stays in the sidecar’s memory.
  3. Setup with that root token: the multichainsign plugin and mount, the client’s policy, and the client’s login (kubernetes or a token).
  4. The seed is born on the signer’s vault, with its SLIP-39 shares sealed to the same keys:
    The response goes to $STATE_DIR/seed-shares.json. It is the only copy of the shares. Take it off the volume, hand it out with citadel-recovery split-shares seed-shares.json, and delete it from the volume once every custodian has confirmed their share (What the custodians get).
  5. The root token is revoked (bao token revoke -self). The sidecar checks that the token no longer works, and logs root token revoked.
The other nodes do not initialize. They join raft, through retry_join in their config and through bao operator raft join to CITADEL_RAFT_PEERS from the sidecar, then unseal through the same KMS key. Setup lives in raft, so they share it. From then on every sidecar only watches: it logs once if its node is sealed (the KMS does not answer) and renews the client’s token in token mode. There is no unseal watchdog. If the data volume is empty while recovery-keys.json is still on the state volume, the sidecar stops with FATAL: OpenBao is NOT initialized but … exists instead of initializing a new vault. The opposite case has its own stop. The sidecar writes $STATE_DIR/init-pending just before bao operator init, and removes it once recovery-keys.json is on disk. If it dies in between, the vault is initialized and nobody holds its recovery keys or its root token. No seed is in it yet, so no funds are at risk. On the next start the sidecar finds init-pending without recovery-keys.json and stops with FATAL: OpenBao is initialized, but its init did not finish. Wipe the data volumes of every node, remove init-pending, and start again. The sidecar reaches its node over TLS with the CA from BAO_CACERT, or by default $CITADEL_BAO_TLS_DIR/ca.crt (/openbao/tls/ca.crt) when that file is mounted into it. With no CA it skips bao operator raft join and says so; the node then joins through retry_join in its own config.

​
After the root token is gone

A later setup needs an admin token: for example, after an image upgrade the plugin’s new sha256 must be registered. Without one the sidecar skips setup and says so. A new root token comes from three recovery-key holders. OpenBao 2.7’s CLI speaks only the authenticated generate-root-token endpoints, so the unauthenticated ones are called directly, on the node’s loopback listener (a shell in the node’s pod; on a VM, docker exec citadel-openbao). Port 8210 needs no token. Any process in the pod’s network namespace can start an attempt there, with its own pgp_key. If the custodians then send their keys to that attempt, the new root token comes out sealed to someone else’s key. Two things close this:
  • The pod holds two containers: openbao and bao-init. Nothing else runs in a node’s pod. The signer and the engine run in pods of their own. The chart keeps it this way. 8210 is safe only while this holds.
  • Every holder checks the attempt before sending a key. The operator announces, before the session and over a separate channel (in person, a call), the fingerprint of their GPG key and the nonce the attempt returned. Each holder reads the attempt and sends nothing unless both match.
Stop and cancel the attempt (curl … -X DELETE https://127.0.0.1:8210/v1/sys/generate-root/attempt) if any of these is true:
  • pgp_fingerprint is not the announced one;
  • nonce is not the announced one;
  • progress is higher than the number of keys the session has sent. For example, it is above 0 before the first holder sends. Someone else then started or fed the attempt.
  • the operator’s start fails with “root generation already in progress”. An attempt the operator did not start is already running.
After that, find out what else runs in the pod, and only then start over. A cancelled attempt throws away the keys sent to it. No holder sends a key to an attempt they have not checked. Pass the token as BAO_TOKEN or BAO_TOKEN_FILE to the initializer’s sidecar for one start, then revoke it.

​
Recovery keys and seed shares

Both use the same five custodians and the same GPG keys, but they guard different things: The engine’s cluster gets recovery keys in the same way. Its keys have no SLIP-39 shares. If the engine’s vault is lost, you create a new engine with new keys, confirm them in the console, and pin the new verification key on the signer. No funds are at risk, because the engine’s keys do not control any wallet. The KMS key becomes the vault’s single point of loss. If it is lost, nothing can decrypt the vault’s data. What is left is rebuilding the seed from 3 of 5 SLIP-39 shares and moving the funds. Protect the key, whichever option below you choose:
  • One KMS key per cluster. Never reuse the signer’s key for the engine.
  • Deny kms:ScheduleKeyDeletion in the key policy, and set the longest deletion window the provider allows, so a deletion can still be cancelled. Alert on any ScheduleKeyDeletion or DisableKey.
  • Make the key survive a region: a multi-region KMS key.
  • Alert on Decrypt errors from the OpenBao nodes. A node that cannot decrypt stays sealed after its next restart.
  • Grant only kms:Encrypt, kms:Decrypt and kms:DescribeKey, and only to the role the OpenBao nodes run as. Use workload identity, not static access keys.
  • Keep -seal-wrap on the multichainsign mount. With a KMS seal the seed is then encrypted twice: by the barrier and by the KMS.

​
Rebuilding the seed

When the vault is gone for good, three custodians rebuild the seed’s mnemonic on an offline machine with citadel-recovery. It is the custodians’ tool on its own: split-shares, check-share, verify and restore, and nothing else.

​
Get the binary

Get it when you bring the stack up, not on the day you need it, and keep a verified copy with each custodian’s share. Each release has a SHA256SUMS file beside the binaries. Download your platform’s file and SHA256SUMS from <download-url>/<version>/, then check the hash:
It prints citadel-recovery_<version>_linux_amd64: OK. Compare SHA256SUMS itself with the copy Citadel sends you through a second channel. Then name the file citadel-recovery, make it executable and check its version:
The binaries are static (no cgo) and contain only the recovery code: no network code and no keys. They are built reproducibly: every build of one version, on any build machine, gives the same bytes and the same SHA256SUMS.

​
What you need

  • An offline machine: no network, nobody watching the screen.
  • The seed’s fingerprint, 16 hex characters. split-shares printed it at the seed’s birth (seed fingerprint: …); keep it with the shares. While the vault still answers, bao read multichainsign/keys/org reports it too.
  • Three of the five custodians, each with their share decrypted from share-<n>-<gpg fingerprint>.asc with their GPG key.
  • A wallet to import the mnemonic into, and a new seed to move the funds to.

​
Rebuild it

Rehearse first: verify takes the same shares and prints only whether they rebuild the seed with that fingerprint.
Each custodian types their share in turn; the input is hidden. restore stops at the set’s threshold, checks the fingerprint and only then prints the mnemonic:
  • The mnemonic is shown only if the shares rebuild the seed with that fingerprint. Otherwise the command stops with MISMATCH: the shares rebuild seed …, the vault holds … and shows nothing.
  • The mnemonic goes to the screen only. A redirect of the output to a file is refused.
  • There is no SLIP-39 passphrase and no BIP-39 passphrase: the vault never sets one, and the command does not ask for one. Shares made elsewhere with a passphrase, or split into several groups, are refused.
  • Import the mnemonic into a wallet only to move the funds to a new seed.
A rebuilt seed is burnt. There is no import: it never goes back into a vault. Move every address’s funds to a new seed, retire the old one, and never fund it again.
It matches the reference implementation. SLIP-39 is SatoshiLabs’ standard (SLIP-0039). Its reference implementation is Trezor’s python-shamir-mnemonic, and its shamir recover rebuilds the same shares without Citadel. The recovery code is held to all 45 official SLIP-39 vectors (vectors.json at tag v0.3.0): the valid sets rebuild the exact master secret and BIP-32 xprv, the invalid ones are refused. A cross-check has the reference split random secrets for Citadel to rebuild, and rebuild the shares of real seed births itself; Citadel and the reference agree byte for byte.

​
How the signer segment reaches the seal

Each node calls Decrypt once when it starts, and only then unseals. Without a network path to the KMS, a restarted node stays sealed. Nodes that are already running keep signing while the KMS is unreachable. The production seal is awskms through a private endpoint. It is the one the image’s cluster config and init step build:
  • Two AWS VPC endpoints (PrivateLink) inside the signer segment’s VPC: KMS, and STS (sts.<region>.amazonaws.com). With IRSA the node first exchanges its web-identity token for credentials through STS (AssumeRoleWithWebIdentity), and only then calls KMS. A node that cannot reach STS gets no credentials and stays sealed after a restart. The segment needs 443 to the two endpoints’ IPs and nothing else: no internet egress.
  • The key policy admits calls only through that endpoint (aws:SourceVpce) and only from the OpenBao nodes’ role.
  • The key is held by AWS, in the region you choose.

​
KMS providers

The openbao image is built on openbao/openbao-ubi:2.7.1. OpenBao 2.7 no longer builds the cloud and HSM seals into the binary: they are KMS plugins. transit, kmip and static stay built in. The image carries four of the plugins in /openbao/plugins: /openbao/config/kms-plugins.hcl declares them with plugin "kms" "<seal name>". A KMS plugin can only be declared in the config file, not through the API, because it must run while the vault is sealed. The names are the 2.6 seal names. A declared plugin runs only when a seal names it; without a seal stanza the vault keeps the Shamir seal. bao status reports Seal Type awskms and Recovery Seal Type shamir. awskms is the only seal the cluster config renders and the chart offers (CITADEL_BAO_SEAL=awskms, OpenBao clusters). The other plugins are in the image, but Citadel does not document or test a seal built on them. The base is UBI and not Alpine because the pkcs11 plugin links against glibc and does not start on musl. For awskms, the node’s environment gives AWS_REGION, VAULT_AWSKMS_SEAL_KEY_ID and AWS_KMS_ENDPOINT (the VPC endpoint), and its role the credentials: IRSA on Kubernetes (OpenBao clusters), the instance role on VMs (On virtual machines). The role needs kms:Encrypt, kms:Decrypt and kms:DescribeKey on the key, and nothing else. Two built-in seals do not fit the signer: transit needs another OpenBao that the signer segment must reach and that then holds its unseal power, and static keeps the unseal key on the same machine, which is the evaluation stack’s weakness again.

​
Pairing without a connection

On the evaluation stack the signer fetches the engine’s key over HTTP with a pairing code. In production the signer never connects to the engine, and the engine never connects to the signer. The engine’s verification key travels through a person, the same way a fingerprint does:
  1. The engine prints its key The engine logs its verification key on every start, in the JSON field pub of the line engine verification key (signer pins this) (with its fingerprint next to it).
  2. An owner confirms it in the console Settings → Policy engine shows the same key as pending. The owner compares the fingerprint and confirms it.
  3. The signer is installed with the key The hex key goes into the signer role’s values, signer.engineKey. The chart starts the signer with --pe-public-key <hex>. The signer pins it on first boot and refuses any other key afterwards.
In production --pe-public-key is required, and the pairing-code path is off. A signer is in production once it is started with --bundle, and it stays there: state.json records "production": true, and no flag turns it back. From then on:
  • citadel-signer up without --pe-public-key exits with production mode: --pe-public-key is required. The key is given on every start, and it must match the pinned one; a different key exits with an engine key is already pinned. Re-pinning means deleting engine-key.json in the state directory by hand.
  • citadel-signer up exits if any of CITADEL_API, CITADEL_JOIN_TOKEN, ENGINE_URL, PAIRING_CODE or PAIRING_CODE_FILE is set, before any connection is made.
  • citadel-signer pair and citadel-signer connect --api … --token … exit with this signer is in production mode.
The evaluation stack is unchanged: without --bundle the signer still connects through discovery and pins from --pe-public-key or a pairing code. Why not deliver the key over NATS? Citadel runs the NATS server. A key that arrives on the bus is a key the platform could have replaced. The contract says the platform never stands between your engine and your signer. A key copied by your operator and confirmed in the console keeps that true. The key is public, so copying it exposes nothing.

​
Signer bundle

In production the signer does not call the platform’s HTTP API. An owner downloads a signer bundle from the console: org, NATS URL, the CA certificate and NATS credentials of a NATS user of the signer’s own, scoped to its four signer.* subjects. The owner puts the bundle on the signer host. The signer then makes no HTTP request to the platform and opens exactly one kind of connection to it: NATS. Its only other outbound traffic is the telemetry push to your collector, when you turn it on. Nobody connects to the signer. The bundle is one JSON file, version 1:
signer-bundle.json
Production start:
--bundle reads SIGNER_BUNDLE and --pe-public-key reads PE_PUBLIC_KEY when the flag is not given. citadel signer up passes both through. citadel-signer connect --bundle <file> applies a bundle without serving. What the signer does with the file:
  • Copies it into its state directory. state.json (org, NATS URL, production, the sha256 of the bundle), nats.creds and nats-ca.pem, all mode 0600. The CA is passed to the NATS client as its root, so a tls:// URL is verified against it; --nats-ca overrides it (NATS over TLS). A bundle without nats_ca removes an older nats-ca.pem.
  • Deletes the file. If it cannot, for example a read-only secret mount, it logs could not remove the signer bundle — delete it by hand and carries on. The same bytes read again are recognised by their sha256 and ignored, so a mounted secret does not stop a restart after the bundle has expired.
  • Refuses an expired bundle that it has not applied before: signer bundle expired at … — download a new one from the console.
  • Accepts a re-issued bundle for the same org. The new creds and CA replace the old ones, for example after the owner revoked the old creds. A bundle for another org is refused.
  • Starts after the file is gone. The state carries the org, the creds and the production mode.
The bundle carries credentials, so it is handled like one:
  • One download, short life. The console lets the bundle be downloaded once, by the owner who issued it, within 24 hours of issuing. The creds leave the console in that download and are not kept. The signer refuses a first use after expires_at.
  • Scoped and revocable. The NATS credentials inside belong to a user of the signer’s own, allowed its four subjects only. An owner revokes them from the console, and the signer is cut off at once. See Issuing and revoking.
  • Delivered over a corporate channel. Never by email or chat. On the host the file is owned by the signer’s user with mode 0600.
  • Deleted after connect. As above. On Kubernetes, delete the secret after the first start; the signer cannot.
A bad CA or revoked creds do not stop the signer: it keeps retrying and logs nats connect failed, retrying with the reason. Why not a bootstrap over NATS? The signer would present the join token on a bootstrap subject. That needs no file, but NATS would have to accept anonymous connections, which adds attack surface on the platform’s side.

​
Issuing and revoking in the console

Settings → Policy engine → Signer bundle. Every action needs security.manage; issuing and revoking also pass the step-up check of security.manage.
  • Issue bundle mints a NATS user with the role signer in the org’s NATS account and keeps its creds, encrypted, until the download. The list shows the user’s public key and the state: pending, downloaded, expired (never downloaded) or revoked.
  • A bundle nobody downloads in time is cleared. A minute-by-minute job drops its creds from the console and revokes its NATS user. If NATS does not take the revocation, the creds are gone anyway, so nobody holds that user’s seed, and the next run tries the revocation again. The list keeps showing it as expired.
  • Download is offered to the owner who issued the bundle, while it is pending. It is a POST, because it spends the bundle: the answer is signer-bundle-<org>.json, and the creds are removed from the console in the same transaction. A second download answers 410 Gone.
  • Revoke adds the user to the account’s revocation list and pushes the account to the NATS server. The server drops the signer’s connection at once (authentication revoked) and refuses the creds from then on; the signer keeps retrying and logs nats connect failed, retrying. The console marks the bundle revoked only after the server took the revocation. If the push fails, nothing changes and the console says NATS did not take the change. Revocations of one org take turns, so two at once never overwrite each other.
  • Issuing a new bundle does not revoke the old one. The old creds keep working until the owner revokes them, so a signer moves to new creds without a gap: issue, apply the new bundle on the signer host, then revoke the old one.
What the two merchant users may do on the bus: The signer’s user cannot publish a request to itself, hear policy.* or touch another org. The engine’s merchant user is unchanged: the evaluation stack’s signer still connects through discovery and shares it. Neither user may subscribe to _INBOX.>: the engine and the signer make no requests on the bus. Both users carry limits: at most 512 KiB per message and 64 subscriptions per connection.

​
Where the signer’s OpenBao sits

Inside the signer segment, on the signer-host machines. The signer reaches it on 8200, and its nodes reach each other on 8201. Its only paths out are to the seal (the KMS private endpoint) and, on Kubernetes, to its own cluster’s Kubernetes API, which sits inside the segment, to check logins. On VMs it checks a client certificate itself and needs no such path. It has no route to the platform, to the engine segment or to the internet.

​
How the signer and the engine log in to OpenBao

No OpenBao token is written to a file. On VMs the clients log in with a TLS client certificate instead (How the clients log in). On Kubernetes the signer and the engine log in with a service-account token minted for audience openbao, through OpenBao’s auth method kubernetes:
  • The client keeps the token in memory. It logs in on start, again when two thirds of the lease have passed, and once more if OpenBao answers 403 to a token it still thought valid. It reads the service-account token from disk on every login, so the kubelet’s rotation needs nothing. A restart simply logs in again.
  • OpenBao checks each login with the Kubernetes API (TokenReview), using its own service account. That account needs the ClusterRole system:auth-delegator. OpenBao reads its own token and CA from the pod, so the auth config holds only the API address.
  • The client never holds an API token. It gets a projected token with audience openbao at /var/run/secrets/openbao/token (BAO_K8S_JWT_FILE). The Kubernetes API refuses that token (401), so the TokenReview right stays with OpenBao. OpenBao refuses the API token in turn: the role accepts only audience openbao. A token the pod uses for something else is no use against OpenBao.
  • Nothing else changes. The same least-privilege policies as before; only the way the token arrives is different.
The setup, done once per cluster by the init step with an admin token:
The OpenBao image’s bao-init does this with CITADEL_BAO_AUTH=kubernetes and CITADEL_K8S_SERVICE_ACCOUNT=<client's service account>. The namespace is the pod’s own (CITADEL_K8S_NAMESPACE, which the chart fills from the downward API), the audience CITADEL_K8S_AUDIENCE (default openbao). The role is named after the policy, so the same script sets up citadel-engine on the engine’s vault (CITADEL_BAO_ROLE=engine). In token mode (the default, used by the evaluation stack) it still writes bao.token. When it switches a volume to kubernetes mode, it revokes the old token and deletes the file. The client side: BAO_K8S_MOUNT (default kubernetes) and BAO_K8S_JWT_FILE (default /var/run/secrets/kubernetes.io/serviceaccount/token) override the mount and the token file. The chart sets BAO_K8S_JWT_FILE=/var/run/secrets/openbao/token. The default path holds the API token, which the role refuses for its audience. A role together with --bao-token or --bao-token-file is refused. The engine waits up to 5 minutes for its first login, and the signer retries while OpenBao is sealed, so both can start before the init step has finished. sign-policy run inside the engine’s pod (Bring-up order) logs in as citadel-engine, signs with key B and keeps no token. In the Helm chart OpenBao and its client do not share a pod. Each has its own service account: <release>-openbao carries the API token, the system:auth-delegator binding and the KMS role; <release>-engine or <release>-signer gets only the projected token for audience openbao, and bao-init binds the role to that account. Neither account mounts a token automatically. The clients reach OpenBao at https://<release>-openbao.<namespace>.svc.cluster.local:8200 and trust its CA through SSL_CERT_DIR=/etc/citadel/openbao-ca:/etc/ssl/certs: Go adds every certificate in those directories to the system roots, so the collector and NATS are still checked against the public ones. Why kubernetes and not static tokens. A token on a volume is a credential anyone who reads the volume can reuse, for as long as it lives (768 hours, renewed). A service-account login is bound to one account in one namespace, the token it yields lives an hour and only in the client’s memory, and revoking the role cuts the client off at its next login.

​
One command per role

Each role is installed with the Helm chart citadel-merchant, once per role, into a namespace of its own. The chart needs Kubernetes 1.27 or later and Helm 3. Add its repository once, on the machine you run Helm from:
What has to exist in the namespace before, because it is a credential and does not belong in the values (Helm keeps values in its release history): The certificate must carry *.<release>-openbao-internal.<namespace>.svc.cluster.local, <release>-openbao.<namespace>.svc.cluster.local, <release>-openbao.<namespace>.svc and 127.0.0.1 (bao-init reaches its node there; 8210, the loopback listener, uses the same certificate). Each role takes two of the three images, at one <version>: the engine role <registry>/citadel-openbao and <registry>/citadel-policy-engine, the signer role <registry>/citadel-openbao and <registry>/citadel-signer. Set repository to <registry>/…, or to the mirror your clusters pull from: the chart’s defaults name no registry. The upstream openbao image does not work here: it has neither the KMS plugins nor bao-init.
engine-host.yaml
signer-host.yaml
The custodians’ keys go in on the command line, --set-file custodians.custodian-1=… through custodian-5. The chart refuses to render values that cannot make a working role: no role, a number of custodian keys other than the number of recovery shares (openbao.init.recoveryShares), a seal without region or key, fewer than three OpenBao nodes, an engine key that is not 64 lowercase hex characters, a role without its Secret.

​
What a release runs

Every container runs as non-root with all capabilities dropped and the RuntimeDefault seccomp profile; all but openbao, which renders its config into /tmp, also with a read-only root filesystem. The chart sets the node’s environment from the release: CITADEL_BAO_API_ADDR and CITADEL_BAO_CLUSTER_ADDR from the pod’s name under the headless Service <release>-openbao-internal, CITADEL_RAFT_PEERS with all three nodes, AWS_REGION, VAULT_AWSKMS_SEAL_KEY_ID and AWS_KMS_ENDPOINT from seal.awskms. bao-init gets CITADEL_BAO_AUTH=kubernetes bound to the client’s service account, the custodians’ keys as files from the ConfigMap <release>-openbao-custodians (custodians.* in name order), and its state on a volume of its own (/bao/state). bao-init runs as the image’s openbao user. On the engine role it also mounts the lease’s KV v2 and writes its path into the citadel-engine policy (CITADEL_ENGINE_LEASE, The lease).

​
What the custodians get

Node 0’s bao-init initializes the cluster on the first install and leaves the custodians’ files on its state volume, PGP-encrypted to them (the init step). They are the only copies. The install notes print the commands:
Delete them from the volume once every custodian has confirmed. A later setup (an image with a new plugin binary) needs a root token from generate-root: put it in a Secret named by openbao.init.adminTokenSecret (key token), restart node 0, then revoke the token and delete the Secret. helm install --wait returns once every pod is ready, and an OpenBao pod is ready as soon as its node is initialized and unsealed. That is before the genesis ends: node 0’s bao-init then sets the node up, births the seed, writes seed-shares.json and revokes the root token, and has no readiness probe of its own. The genesis is over when node 0’s bao-init logs bao-init: root token revoked followed by bao-init: ready; watching the seal; on the signer role /bao/state/seed-shares.json exists by then. Wait for that line before you copy the files:

​
The network

networkPolicy.enabled (default on) puts a default-deny policy on the whole namespace, ingress and egress, and opens only these paths: Each egress.* is a list of CIDRs and ports. An empty list closes that path, and the install notes warn about NATS, the KMS and the Kubernetes API. A NetworkPolicy matches IP blocks only, so the values take the CIDRs your names resolve to: the subnets of both VPC endpoints, KMS and STS (there is no separate rule for STS: put its endpoint’s subnets in egress.kms too), the NATS address, the API server’s endpoint IPs. For the Kubernetes API that is the API server’s own addresses, not the kubernetes Service IP: most CNIs match egress after the Service address is translated.

​
Why a chart, and not a Compose profile

  • Raft needs three machines and an ordered start. A StatefulSet with stable names, retry_join and a PodDisruptionBudget gives that in one command. Compose runs on one host and cannot spread a cluster.
  • Requirement 7 splits in two. The pods’ traffic is the chart’s: it ships a default-deny NetworkPolicy, the signer gets no ingress, and egress only to its OpenBao, NATS, the collector and DNS; its OpenBao reaches only its peers, the KMS and STS endpoints and the Kubernetes API. That holds only on a CNI that enforces NetworkPolicy. The hosts are the operator’s: SSH only from the VPN, no kubelet and no NodePort reachable from outside the segment, on the segment’s firewall. The chart does not do that part.
  • KMS credentials without static keys. Workload identity (IRSA, GKE Workload Identity, Azure managed identity) gives each OpenBao pod its KMS grant through a service-account annotation.
Physically separate machines means separate Kubernetes clusters for the two roles, or at the least separate node pools with taints and no shared nodes. Separate clusters are the default. “One command” covers the machines, not the people. The genesis ceremony stays a human step on purpose: five custodians hand over GPG public keys, and the init step uses them for the recovery keys and the seed’s SLIP-39 shares. Until that step has run with the custodians’ keys, the signer waits and signs nothing.

​
On virtual machines

For a merchant without Kubernetes, the installer citadel-vm puts a role on Linux machines: one command per host, three hosts per role. It builds the same thing as the chart: three OpenBao nodes on raft per role with TLS, the awskms seal and the audit device (OpenBao clusters); the image’s bao-init with the custodians’ keys, the seed’s birth and the root token revoked (the init step); the role’s client; a default-deny firewall. Each host is a Linux machine with bash, systemd, docker, nft and openssl. The engine hosts also mount one NFSv4 share for the engines’ state dir. For an evaluation the installer also puts a whole role on one host, without a KMS: the three OpenBao nodes as three containers on that host, unsealed by the custodians (One host per role, without a KMS). That layout is not for production.

​
Get the installer and the images

citadel-vm is one file, published with each release beside a SHA256SUMS file. Download both from <download-url>/<version>/ and check the hash before you copy the installer to a host:
The hosts never pull images. On a machine that can reach your registry (the bastion), pull the role’s two images and save them into one tar:
Copy citadel-vm, the tar, the host file and its certificates to each host over SSH. Then:
install copies the script to /usr/local/sbin/citadel-vm, for the timer and for the operator; later commands run as sudo citadel-vm ….

​
The host file

One file per host, KEY=value lines. A key the script does not know is refused, so a typo cannot fall back to a default. The other two hosts of a role get the same file with their own NODE_ADDR, NODE_ID, PEERS, TLS_* and INITIALIZER=false.
signer-host.conf
engine-host.conf
The script refuses, all at once, a host file that cannot make a working host: no address, fewer than three nodes or an even number, NODES other than 1 or 3, PEERS with NODES=3, an engine key that is not 64 hex characters, a node certificate without the IP SAN of NODE_ADDR, 127.0.0.1 or (with NODES=3) a node’s address, fewer custodian keys than shares, an awskms seal without region or key, a KMS key or KMS CIDRs with SEAL=shamir, a path the role needs with no CIDRs in the firewall (NATS, the KMS, SSH, the API and NFS for an engine, the collector when OTLP_ENDPOINT is set), a first signer start without the bundle, an image name that is not one word. install also refuses an image that is not on the host, and a host installed before with the other role, another NODES or another SEAL: each of those is a new vault, so uninstall --purge comes first.
install refuses an SSH session that its own firewall would cut: one from outside SSH_CIDRS, or one over IPv6 (loopback stays open). sudo drops SSH_CLIENT, so the installer reads the address from its own environment first, then from the processes above it (sudo, the login shell) up to sshd; a session under sshd whose address is found nowhere is refused too, not let through. Under tmux or screen the walk stops at the multiplexer’s server: it carries the address of the SSH session that started it, or none if it was started on the console, and then the installer takes the session for a console one. CITADEL_VM_LOCKOUT_OK=1 overrides that one check.

​
What a host runs

Every process is a container under a systemd unit, docker run in the foreground with Restart=always:
  • The node keeps a network namespace of its own. Its loopback listener 8210 (generate-root without a token) is reachable from the node and from bao-init, and from nothing else on the host: not from the signer, not from the engine. That is the same rule as the chart’s two-container pod (After the root token is gone). The operator reaches 8210 with docker exec citadel-openbao curl --cacert /openbao/tls/ca.crt https://127.0.0.1:8210/….
  • The client reaches its own node, at https://NODE_ADDR:8200. A standby node forwards to the leader. Each signer host is one replica of Signer replicas, with its own state in /var/lib/citadel/signer. Each engine host is one engine of the lease; all of them mount the same ENGINE_STATE_DIR.
  • Read-only containers. Every container drops all capabilities and runs with no-new-privileges; all but the node, which renders its config into /tmp (a tmpfs), also with a read-only root filesystem.
  • Files. The host file as installed, the environment files (0600) and the certificates are in /etc/citadel; keys are 0400 and owned by the user that reads them. Raft data, the audit log, bao-init’s state and the snapshots are in /var/lib/citadel, 0700, owned by 100.
  • The bundle is applied once. It is copied to /etc/citadel/bundle only while the signer has no state, and the signer deletes it once applied. A later install does not copy it again.
  • The join token is used once. install copies it to /etc/citadel/join (0400, owned by the engine), and the engine deletes it there once it has joined: no second install. An engine that has state is not given it again. On the other two hosts of the role the copy goes when their engine first takes the lease; the token is spent by then. Delete the file JOIN_TOKEN_FILE names yourself.
install returns once the host’s node is initialized and unsealed (CITADEL_VM_WAIT, 600 seconds); with SEAL=shamir, once the vault is initialized, and it stays sealed. A node that is not the initializer joins once the initializer’s node is up, so the initializer goes first.

​
How the clients log in

There is no Kubernetes to vouch for a client, and a token in a file is what kubernetes auth exists to avoid. On a VM the client logs in with a TLS client certificate, through OpenBao’s auth method cert:
bao-init does this with CITADEL_BAO_AUTH=cert: The client side: --bao-cert-role on citadel-signer up, citadel-signer run, citadel signer up, citadel policy-engine up and citadel policy-engine sign-policy, or BAO_CERT_ROLE (also the engine’s). The certificate is BAO_CLIENT_CERT, its key BAO_CLIENT_KEY, the mount BAO_CERT_MOUNT (default cert). As with kubernetes auth the token lives in memory only, the client logs in again at two thirds of its lease and once more on a 403, and reads the certificate on every login, so a renewed pair needs no restart. A role together with a token is refused, as are two roles. The engine waits up to 5 minutes for its first login. The client trusts the node’s CA through SSL_CERT_DIR=/etc/citadel/client/ca:/etc/ssl/certs. The key of the client certificate is the credential now, so it stays on the host, 0400, owned by 65532. Each role has its own: the signer’s certificate is refused by citadel-snapshot, and the snapshot’s by citadel-signer. A snapshot is the whole vault, and on the same VM the instance role can decrypt it: the signer must not be able to take one. The roles tell clients apart by common name only, and trust the cluster’s CA. So the two roles take two CAs: with one, the engine host’s citadel-snapshot certificate would log in to the signer cluster too. sign-policy on an engine host runs in the engine’s container, which already holds the login: docker exec citadel-engine /citadel policy-engine sign-policy /state/policy.yaml.

​
The firewall

citadel-firewall.service loads one nftables table, inet citadel, before the network comes up. Each of its three chains drops by default: So the signer host keeps requirement 7: the only inbound connection is SSH from the VPN, the signer talks only to NATS, its node and the collector, and only the OpenBao node reaches the KMS. Docker’s own rules still apply after these, and both have to accept. Drops are logged with the prefixes citadel-in-drop, citadel-fwd-drop and citadel-out-drop, at most ten a minute. On AWS, SEAL_CREDENTIALS=instance reaches the metadata service from the node’s container, one hop further than the host: set the instance’s HttpPutResponseHopLimit to 2 for IMDSv2. No registry is in the list: the signer host has no route to one. Images arrive as a tar over SSH from the bastion (Get the installer and the images), and install never pulls.

​
What the custodians get on a VM

The initializer’s bao-init leaves the custodians’ files in /var/lib/citadel/bao-init, PGP-encrypted to them, and install names them when it returns:
Take them off the host over SSH and delete them there once every custodian has confirmed.

​
Backups

citadel-snapshot.timer runs citadel-vm snapshot on every host (SNAPSHOT_SCHEDULE, daily by default, with up to ten minutes of random delay, and Persistent=true, so a missed run happens at the next boot). It logs in as citadel-snapshot from the host to its node and writes /var/lib/citadel/snapshots/<NODE_ID>-<UTC time>.snap (through a .tmp that a failed save removes); the newest SNAPSHOT_KEEP stay. A standby forwards the request, so any host can take it. The signer host sends nothing out, so the snapshots leave it the way the operator comes in: fetched over SSH from the bastion. The first one is part of the bring-up: sudo citadel-vm snapshot right after the genesis.

​
Upgrades

The same command, with the new image names in the host file and the new tar: citadel-vm install <role> <host.conf> --images <tar>. It writes the files again, reloads the firewall, restarts the units and waits for the node. A restart is a node restart, so upgrade one host at a time and wait for citadel-vm status to show Sealed: false before the next. An image with a new plugin binary needs a later setup: put a root token from generate-root in ADMIN_TOKEN_FILE on the initializer, install it, then remove the key and install again; the token file is deleted.

​
One host per role, without a KMS

For an evaluation, such as a demo or an integration test, a role can run on one Linux host without a KMS. NODES=3 puts all three OpenBao nodes of the role on that host. With SEAL=shamir, the custodians unseal the nodes instead of a KMS. The command stays the same:
This is not a production layout. One host is one failure: when the host is down, the role is down. Every start of a node also waits for the custodians, and a reboot at night stops signing until three of them unseal. A vault started this way is not moved into the production layout later. Production starts with its own genesis ceremony and its own seed.
signer-single-host.conf
engine-single-host.conf
What differs from three hosts:

​
Who unseals, and when

The vault is sealed after the install and after every start of a node: a restarted unit, a reboot of the host, an upgrade. Nothing on the host unseals it, because the unseal keys exist only sealed to the custodians. Until three custodians unseal it, the role is down: the signer does not sign and the engine does not answer.
1

Install

install returns once node 0 is initialized and reports it sealed. Node 0’s bao-init has written /var/lib/citadel/bao-init-0/unseal-keys.json, five unseal keys, each sealed to one custodian, in the order of CUSTODIAN_KEYS. Copy it off the host over SSH and give it to the custodians:
The file holds no root token. The root token from the init stays in the memory of node 0’s bao-init until the setup below has run, and is then revoked.
2

Each custodian decrypts their own key

On the custodian’s own machine, with their private key, where <n> is their position in CUSTODIAN_KEYS, counting from 0:
3

Three custodians unseal

Each over SSH from SSH_CIDRS:
It asks for the key without echoing it and gives it to every sealed node on the host, through the node’s own API on its loopback. The key never goes on a command line or to disk. The command prints each node’s progress: sealed, 2 of 3 keys, then unsealed.A custodian can also send the key from their own machine without typing it: the command reads it from stdin when stdin is not a terminal, so the decrypted key goes straight into the SSH session.
4

The first install: the three at the same time

On the first install only node 0 is in the vault. Nodes 1 and 2 join it once it is unsealed: its bao-init runs the setup (the plugin, the policy, cert auth, and on the signer role the seed with its shares, seed-shares.json beside unseal-keys.json, handed out as in What the custodians get on a VM), revokes the root token, and the other two nodes join within seconds.So on the first install the three custodians start citadel-vm unseal at the same time, each in their own SSH session, and leave it running. Each run gives its key to node 0, prints not joined yet … Waiting to give them your key too, and keeps the key in its own memory, never on disk. When the third key unseals node 0 and nodes 1 and 2 join, every run gives them its key and returns. The runs report the progress they saw: the first ones end with sealed, 1 of 3 keys or 2 of 3 for nodes 1 and 2, which is expected; the last one reports them unsealed. One round unseals all three nodes.A run waits up to 15 minutes (CITADEL_VM_UNSEAL_WAIT, in seconds). If it stops first (it timed out, or a custodian pressed Ctrl-C), the key is gone with it: that custodian runs sudo citadel-vm unseal once more after node 0 is unsealed. sudo citadel-vm status shows each node’s Sealed and HA Mode.
After that, every restart needs one round: three custodians, one citadel-vm unseal each, unseal all three nodes. Delete unseal-keys.json and seed-shares.json from the host once every custodian has confirmed their key. Things to know:
  • The key passes through the host. With a KMS no person ever holds an unseal key. Here each custodian types their key on the host, so whoever is root on it while the key is typed can read it. Three custodians on a compromised host hand over the vault.
  • Node 0 is the clients’ node. When node 0 alone restarts, nodes 1 and 2 keep the cluster, but the signer, the engine and the snapshot wait until node 0 is unsealed again.
  • An interrupted first unseal. If node 0’s bao-init restarts after the init but before the custodians unseal node 0, the root token from the init is gone and the setup does not run. Three unseal-key holders then generate a new one (After the root token is gone: with this seal the holders send unseal keys, not recovery keys), put it in ADMIN_TOKEN_FILE and install again.
  • Upgrades restart the three nodes together, so the custodians unseal after every upgrade, and after every later install.
  • Backups. The snapshots are on the same host as the vault. Fetch them off it (Backups): with the host’s disk lost, a snapshot and three unseal keys are the only way back.
  • Memory. Three OpenBao nodes, their bao-init sidecars and the client share the host. On an idle engine role the containers took about 460 MiB together: 50 to 70 MiB per node, 45 to 120 MiB per bao-init, and about 30 MiB for the engine. Docker and the OS come on top of that. Give the host 2 GB. With 1 GB it starts, but has no headroom.

​
Why containers, cert auth and CIDRs

  • Containers under systemd, not packages. The openbao image is the only build of OpenBao with the KMS plugins and the multichainsign plugin, and its entrypoint renders the cluster config; the VM runs the same image as the chart. A container also gives the node the network namespace that keeps 8210 away from the client on the same host.
  • Cert auth, not tokens. Kubernetes auth needs a Kubernetes API, and the token mode of the evaluation stack writes a 768-hour token to a volume that only the initializer’s host would have. A client certificate per role keeps the rules of kubernetes auth: one role, one-hour tokens in memory, nothing reusable on disk but a key file that only that role’s user reads.
  • CIDRs, not host names, in the firewall, as in the chart’s NetworkPolicy: nftables matches addresses. The KMS endpoint’s subnets, the NATS address, the API’s.

​
Bring-up order

1

Get the join token and the signer bundle

An owner mints a join token in the console, and downloads the signer bundle once. The bundle reaches the signer host over a corporate channel and is stored there with mode 0600. It expires in 24 hours if unused.
2

Collect the custodians' public keys

Five GPG public keys, custodian-1.b64 … custodian-5.b64, one per custodian.
3

Install the engine role

Add the chart’s repository (helm repo add citadel <chart-repo-url>) and mirror the images. Create the namespace with its TLS Secret and the join token Secret, then helm install citadel-engine citadel/citadel-merchant --version <version> …. The engine’s cluster inits with recovery keys, the engine creates its three keys, takes the lease, joins the platform and logs its public keys. Take recovery-keys.json off node 0’s state volume, then delete the join token Secret.On VMs: get citadel-vm and the image tar (Get the installer and the images), then citadel-vm install engine engine-host.conf on the initializer, then on the other two. The keys are in journalctl -u citadel-engine, recovery-keys.json in /var/lib/citadel/bao-init. The engine deletes its copy of the join token once joined; delete the file JOIN_TOKEN_FILE names.
4

Confirm the keys in the console

Compare the fingerprints and confirm the engine key, key A and key B.
5

Install the signer role with the engine key

Create the namespace with its TLS Secret and the bundle Secret, then helm install citadel-signer citadel/citadel-merchant --version <version> … with signer.engineKey. The signer’s cluster inits with recovery keys and the seed is born with its SLIP-39 shares (the init step); the root token is revoked. Take recovery-keys.json and seed-shares.json off the initializer’s state volume. Delete the bundle Secret once every replica logged connected from signer bundle.On VMs: citadel-vm install signer signer-host.conf with ENGINE_KEY, the initializer first. Each host’s signer applies the bundle and deletes its copy.
6

Back up and verify, before any funding

Required. No wallet is funded until every item holds:
  • First snapshot. bao operator raft snapshot save on the signer’s cluster, and on the engine’s (on VMs, sudo citadel-vm snapshot). Restore it on a separate node with the same seal, and check the restored vault unseals and lists the multichainsign mount.
  • KMS key. Its backup is in place: a multi-region key. Its protection is in place: deletion denied in the key policy, the longest deletion window, alerts on ScheduleKeyDeletion, DisableKey and Decrypt errors.
  • Every share confirmed. Each of the five custodians decrypts their share with their GPG key, writes it down offline and runs citadel-recovery check-share. Shares are handed out once, at the seed’s birth, and stored nowhere else. A share that is not confirmed now cannot be recovered later.
  • Offline drill. Three custodians run citadel-recovery verify --fingerprint <hex> on an offline machine. It rebuilds the seed in memory and prints only whether it matches the vault’s fingerprint.
7

Sign and publish the policy

Download the policy from Settings → Policy document and sign it with key B inside the engine’s container, which already holds the OpenBao login. The engine image has no shell, so the document goes in on stdin:
Paste the signature into the console, Publish, then read back what is in force with /citadel policy-engine check-policy in the same container.
8

Derive the first wallet

Issue a wallet in the console. The engine logs derived and registered.