Monitoring

The policy engine and the signer push their metrics and logs over OTLP; nothing on them is scraped (Telemetry lists the variables and every metric they emit). OpenBao answers a health endpoint and, once enabled, serves its own Prometheus metrics. This page puts the three together with OpenTelemetry Collectors and any backend — the examples use Prometheus for metrics, Loki for logs and Grafana on top — and adds what the components do not count themselves: refusals by reason, NATS and lease events, all derived from log records in the collector. Two kinds of collector:

The main collector, one, wherever your collectors run. It receives every push, derives the counters below from the log records and hands everything to your backends.
A vault collector beside each OpenBao, on the same host or in the same namespace. Your vaults take no connections from outside their side (the signer’s side accepts none at all), so this one reads its vault locally and pushes what it reads to the main collector. It accepts nothing itself.

Everything travels outbound to the main collector, over the same path the engine and the signer already push on.

1. Point the components at the main collector

On the engine and on the signer:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.merchant.internal:4318
# only if your CA is not in the system roots: the PEM of the CA that signed the collector's certificate
OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/citadel/otlp/ca.crt
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production

The Helm chart sets them from otlp.endpoint and otlp.caSecret, the VM installer from OTLP_ENDPOINT and OTLP_CA. Start the main collector before them. Logs are pushed once and never retried: whatever the engine logs while no collector answers — its keys, its policy version, policy engine ready — stays in its own output only (and otlp.logs.dropped{reason="export"} counts it).

Do not ship the engine’s container output to a log store. The engine prints its one-time pairing code to its own stdout at every start — on purpose never to the collector, so that whoever reads your logs cannot spend it. A node-level log agent that tails every container (the usual Kubernetes setup) would carry it into your log store anyway. Exclude the engine container from that agent, or drop the line that starts with pairing code:. Everything the engine and the signer log is already in the OTLP push.

2. OpenBao

Health — works on every vault, no token. GET /v1/sys/health?standbyok=true answers 200 when the node is initialized and unsealed (active, or a standby of a three-node cluster), 503 sealed, 501 not initialized; a node that is down does not answer. Metrics. OpenBao serves /v1/sys/metrics?format=prometheus only when its server configuration has a telemetry stanza, and only to a token allowed to read sys/metrics:

telemetry {
  prometheus_retention_time = "1h"
  disable_hostname          = true
}

# policy citadel-metrics
path "sys/metrics" {
  capabilities = ["read"]
}

A sealed vault serves no metrics — it cannot check the token. Take the seal state from the health endpoint, never from vault_core_unsealed. The Helm chart and the VM installer do not set the telemetry stanza yet; until they do, the health endpoint is what OpenBao gives you.

Worth watching from its telemetry: vault_route_update_multichainsign_sum / _count (time per sign, derive and verdict signature inside the vault — the closest thing to signing latency the stack has), vault_raft_peers, vault_autopilot_healthy, vault_autopilot_failure_tolerance, vault_core_active.

3. The main collector

It needs the contrib distribution of the collector (the transform and deltatocumulative processors, the count connector). Replace the hosts, the certificate and the log backend with yours.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        # the engine and the signer push https:// — the receiver must speak TLS
        tls:
          cert_file: /etc/otelcol/tls/tls.crt
          key_file: /etc/otelcol/tls/tls.key

processors:
  batch: {}
  # Counted events reach Prometheus as running totals.
  deltatocumulative: {}
  # the 0/1 gauges carry unit "1", which Prometheus would turn into a _ratio suffix
  transform/units:
    error_mode: ignore
    metric_statements:
      - context: metric
        statements:
          - 'set(unit, "") where unit == "1"'
  # A refusal's text names wallets, addresses and amounts — fine in a log line, unusable as
  # a metric label. Fold it into a fixed class; the full text stays in the log record.
  transform/classify:
    error_mode: ignore
    log_statements:
      - context: log
        conditions:
          - 'attributes["reason"] != nil'
        statements:
          - 'set(attributes["reason_class"], "request_signature") where IsMatch(attributes["reason"], "request signature does not verify|carries no merchant signature|no merchant request key")'
          - 'set(attributes["reason_class"], "request_mismatch") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "^policy: (action \\d+:|request has|request wallet|merchant request)")'
          - 'set(attributes["reason_class"], "ceiling") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "exceeds .*ceiling|no ceiling configured")'
          - 'set(attributes["reason_class"], "limit") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "would exceed its|authorizations per")'
          - 'set(attributes["reason_class"], "reference") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "^reference .* is not registered")'
          - 'set(attributes["reason_class"], "destination") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "whitelist|rules no addresses|not an owned wallet|may only lend|not an asset this merchant recognises|has not named")'
          - 'set(attributes["reason_class"], "expiry") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "intent expired|not yet valid|no not_after")'
          - 'set(attributes["reason_class"], "wallet") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "source wallet|does not match wallet|wallet_ref")'
          - 'set(attributes["reason_class"], "digest") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "sighash mismatch|re-derive|digest")'
          - 'set(attributes["reason_class"], "replay") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "already authorized as another|replacement of operation")'
          - 'set(attributes["reason_class"], "signer_unreachable") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "could not reach the signer")'
          - 'set(attributes["reason_class"], "openbao") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "sign verdict|bao")'
          - 'set(attributes["reason_class"], "document") where attributes["reason_class"] == nil and IsMatch(attributes["reason"], "document|policy-admin")'
          - 'set(attributes["reason_class"], "other") where attributes["reason_class"] == nil'
      - context: log
        conditions:
          - 'attributes["err"] != nil and IsMatch(body, "^signer\\.(sign|derive) (refused|failed)")'
        statements:
          - 'set(attributes["reason_class"], "verdict_signature") where IsMatch(attributes["err"], "verdict signature does not verify")'
          - 'set(attributes["reason_class"], "verdict_denied") where attributes["reason_class"] == nil and IsMatch(attributes["err"], "verdict does not allow")'
          - 'set(attributes["reason_class"], "digest") where attributes["reason_class"] == nil and IsMatch(attributes["err"], "different digest|already signed a different")'
          - 'set(attributes["reason_class"], "openbao") where attributes["reason_class"] == nil and IsMatch(attributes["err"], "^bao:")'
          - 'set(attributes["reason_class"], "other") where attributes["reason_class"] == nil'
      # Name the kind and the event for the counters below (the components log them as the
      # message only).
      - context: log
        statements:
          - 'set(attributes["kind"], "document") where body == "policy document refused"'
          - 'set(attributes["kind"], "register_interface") where body == "interface registration refused"'
          - 'set(attributes["event"], "connected") where IsMatch(body, "^nats connected")'
          - 'set(attributes["event"], "reconnected") where IsMatch(body, "^nats reconnected")'
          - 'set(attributes["event"], "disconnected") where IsMatch(body, "^nats disconnected")'
          - 'set(attributes["event"], "connect_failed") where IsMatch(body, "^nats connect failed")'
          - 'set(attributes["event"], "acquired") where IsMatch(body, "^active: journal fenced")'
          - 'set(attributes["event"], "lost") where IsMatch(body, "^lost the lease")'
          - 'set(attributes["event"], "released") where IsMatch(body, "^(released the lease|could not release the lease)")'
          - 'set(attributes["event"], "standby") where IsMatch(body, "^standby: waiting for the lease")'

connectors:
  count:
    logs:
      citadel.engine.refusals:
        description: engine refusals by kind and reason class
        conditions:
          - 'resource.attributes["service.name"] == "citadel-policy-engine" and (body == "refused" or body == "policy document refused" or body == "interface registration refused")'
        attributes:
          - key: kind
          - key: reason_class
            default_value: other
      citadel.signer.refusals:
        description: signer refusals and failures by reason class
        conditions:
          - 'resource.attributes["service.name"] == "citadel-signer" and IsMatch(body, "^signer\\.(sign|derive) (refused|failed)")'
        attributes:
          - key: reason_class
            default_value: other
      citadel.engine.lease.events:
        description: lease transitions of an engine
        conditions:
          - 'resource.attributes["service.name"] == "citadel-policy-engine" and attributes["event"] != nil'
        attributes:
          - key: event
      citadel.nats.events:
        description: NATS connection events logged by the signer
        conditions:
          - 'IsMatch(body, "^nats (connected|reconnected|disconnected|connect failed)")'
        attributes:
          - key: event
      citadel.log.warnings:
        conditions:
          - 'severity_number >= SEVERITY_NUMBER_WARN and severity_number < SEVERITY_NUMBER_ERROR'
      citadel.log.errors:
        conditions:
          - 'severity_number >= SEVERITY_NUMBER_ERROR'

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    # each point keeps its push time: time() - timestamp(m) is the age of the last push
    send_timestamps: true
    resource_to_telemetry_conversion:
      enabled: true
  otlphttp/logs:
    endpoint: https://loki.merchant.internal/otlp

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [transform/classify, batch]
      exporters: [otlphttp/logs, count]
    metrics:
      receivers: [otlp, count]
      processors: [deltatocumulative, transform/units, batch]
      exporters: [prometheus]

In Prometheus the dots become underscores and counters get _total: citadel.engine.decisions is citadel_engine_decisions_total, citadel.engine.lease.active is citadel_engine_lease_active. service.name and service.instance.id become job (with service.namespace in front, if you set one) and instance.

4. A vault collector beside each OpenBao

The same configuration twice, once beside each vault, with two variables:

Beside	`VAULT`	`OPENBAO_ADDR`
the engine’s OpenBao	`engine`	its API, `host:port`
the signer’s OpenBao (the seed)	`seed`	its API, `host:port`

Use exactly engine and seed: the dashboard and the alerts below tell the two vaults apart by the label vault. Put the metrics token (policy citadel-metrics) of that vault in /etc/otelcol/openbao-metrics.token.

# A vault collector: polls one OpenBao, pushes to the main collector, accepts nothing.
receivers:
  httpcheck/openbao:
    collection_interval: 15s
    targets:
      - endpoint: https://${env:OPENBAO_ADDR}/v1/sys/health?standbyok=true
        method: GET
        tls:
          ca_file: /etc/otelcol/openbao-ca.crt
  prometheus/openbao:
    config:
      scrape_configs:
        - job_name: openbao-${env:VAULT}
          scrape_interval: 15s
          scheme: https
          metrics_path: /v1/sys/metrics
          params:
            format: [prometheus]
          authorization:
            credentials_file: /etc/otelcol/openbao-metrics.token
          tls_config:
            ca_file: /etc/otelcol/openbao-ca.crt
          static_configs:
            - targets: ["${env:OPENBAO_ADDR}"]
              labels:
                vault: ${env:VAULT}

processors:
  batch: {}
  transform/vault:
    error_mode: ignore
    metric_statements:
      # one series per vault and status class (a status-code label would leave the last
      # "200" series standing for minutes after the vault stopped answering)
      - context: datapoint
        statements:
          - 'delete_key(attributes, "http.status_code") where metric.name == "httpcheck.status"'
          - 'set(attributes["vault"], "${env:VAULT}") where metric.name == "httpcheck.status"'

exporters:
  otlphttp:
    endpoint: https://otel.merchant.internal:4318
    tls:
      ca_file: /etc/otelcol/otel-ca.crt

service:
  pipelines:
    metrics:
      receivers: [httpcheck/openbao, prometheus/openbao]
      processors: [transform/vault, batch]
      exporters: [otlphttp]

A three-node vault: list every node — one httpcheck target and one scrape target per node, the per-pod addresses on Kubernetes (<release>-openbao-0.<release>-openbao-internal.<namespace>.svc.cluster.local:8200, …). Each node serves its own metrics and its own health; the dashboard and the alerts count a vault as up while any of its nodes is unsealed. Where it runs:

Kubernetes. A small Deployment in the role’s namespace. With the chart’s network policy on (the default) that namespace denies all traffic but the chart’s own paths. Add a NetworkPolicy of yours — policies add up, the chart’s stays as it is — that lets the collector pod reach the OpenBao pods on 8200, DNS and the main collector, and lets the OpenBao pods accept it on 8200.
A VM. On the host itself (the collector’s own package, a systemd service). The installer’s firewall lets this host’s processes reach their node on NODE_ADDR:8200 and your collector on OTLP_CIDRS; a collector in a container on a docker bridge network is dropped, run it with --network host. The node’s certificate carries NODE_ADDR and 127.0.0.1.

5. OpenBao’s log (optional)

The dashboard’s OpenBao log panel shows OpenBao’s server log once something sends it to the main collector with service.name openbao and citadel.vault set to the vault. OpenBao writes it to its own output, so where it is depends on where the vault runs. Read only the OpenBao container — never every container (see the warning above). Kubernetes — add a receiver to the node agent that already reads /var/log/pods (a DaemonSet), not to the vault collector, which runs once:

receivers:
  filelog/openbao:
    include: ["/var/log/pods/<namespace>_<release>-openbao-*_*/openbao/*.log"]
    include_file_path: true      # the container parser reads the pod's name from the path
    start_at: end
    resource: { service.name: openbao, citadel.vault: seed }   # engine in the engine's namespace
    operators:
      - type: container          # containerd / CRI-O / docker formats; adds the pod's name
      # 2026-10-19T12:00:00.000Z [INFO]  core: vault is sealed
      - type: regex_parser
        regex: '^(?P<ts>\S+) \[(?P<level>[A-Z]+)\]\s+(?P<msg>[^\n]*)\n?$'
        on_error: send_quiet
        severity:
          parse_from: attributes.level
      - type: move
        if: attributes.msg != nil
        from: attributes.msg
        to: body
      - type: remove
        if: attributes.ts != nil
        field: attributes.ts

A VM — the vault collector on the host reads the node’s unit from the journal (journalctl must be on the host; an evaluation host with three nodes runs citadel-openbao-0.service, -1 and -2 instead):

receivers:
  journald/openbao:
    units: [citadel-openbao.service]
    start_at: end
    resource: { service.name: openbao, citadel.vault: "${env:VAULT}" }
    operators:
      - type: move
        from: body.MESSAGE
        to: body
      - type: regex_parser
        regex: '^(?P<ts>\S+) \[(?P<level>[A-Z]+)\]\s+(?P<msg>[^\n]*)\n?$'
        on_error: send_quiet
        severity:
          parse_from: attributes.level
      - type: move
        if: attributes.msg != nil
        from: attributes.msg
        to: body
      - type: remove
        if: attributes.ts != nil
        field: attributes.ts

Either way the receiver joins a logs pipeline that exports to the main collector:

service:
  pipelines:
    logs:
      receivers: [journald/openbao]      # or filelog/openbao
      processors: [batch]
      exporters: [otlphttp]

6. What to watch

Policy engine

Metric	Source	What it tells you	Alert when
`citadel_engine_decisions_total{kind, result}`	pushed	every verdict: payouts (`sign`), new wallets (`keygen`), policy documents (`document`), contract registrations (`register_interface`)	refused share of `sign` above 50% with 5 or more in 10 min
`citadel_engine_refusals_total{kind, reason_class}`	derived from logs	why payouts are refused: `ceiling`, `limit` (window or velocity), `reference` (end user not registered), `request_signature` (not signed by your key A), `request_mismatch`, `destination`, `expiry`, `wallet`, `digest`, `replay`, `openbao`, `signer_unreachable`, `document`, `other`	`request_signature` or `digest` above 0 — someone asks without your key, or the platform’s transaction differs from your request
`citadel_engine_policy_version`	pushed	the signed policy in force; steps up on every publish	0 for 5 min — every payout is refused
`citadel_engine_lease_active`	pushed	1 on the engine that serves, 0 on a standby	sum over engines `< 1` for 2 min; two engines at 1 for 5 min; more than 4 changes in 30 min
`citadel_engine_lease_events_total{event}`	derived	`acquired`, `lost`, `released`, `standby`	`lost` above 0
`citadel_engine_nats_connected`	pushed	the engine is on the platform’s bus	0 for 2 min, per instance
`citadel_engine_derived_total{result}`	pushed	wallets registered after the signer derived them	`failed` above 0
`citadel_engine_owned_wallets`	pushed	addresses the engine treats as yours — the only valid payout sources	an unexpected drop
`citadel_engine_publish_errors_total{subject}`	pushed	a verdict or a result that never reached NATS	above 0

Signer

Metric	Source	What it tells you	Alert when
`citadel_signer_requests_total{op, result}`	pushed	`signed`, `derived`, `replayed`, `refused` (no valid verdict for that digest), `failed` (its OpenBao did not answer), `malformed` (not a request at all)	`refused` above 0 (critical: a verdict not signed by the pinned engine key, a denial, or another digest); `failed` above 0; `malformed` above 0
`citadel_signer_refusals_total{reason_class}`	derived	why: `openbao`, `verdict_signature`, `verdict_denied`, `digest`	—
`citadel_signer_nats_connected`	pushed	the signer is on the bus	0 for 2 min, per instance
`citadel_signer_nats_reconnects_total`, `citadel_nats_events_total{event}`	pushed, derived	reconnects; `connect_failed` every 2 s while it is cut off	—
`citadel_signer_publish_errors_total{op}`	pushed	a signature that never reached NATS	above 0

Approved by the engine versus answered by the signer: every sign approval should be followed by a signer answer (signed, failed or refused). An approval with no answer was lost between them — typically a signer that was off the bus when the verdict was published. Compare citadel_engine_decisions_total{kind="sign",result="approved"} with citadel_signer_requests_total{op="sign"} over 15 minutes.

OpenBao, both vaults

Metric	Source	What it tells you	Alert when
`httpcheck_status{vault, http_status_class="2xx"}`	health endpoint	1 = initialized and unsealed	0 on every node for 1 min
`vault_route_update_multichainsign_sum / _count`	telemetry	ms per key operation	above 500 ms for 5 min
`vault_raft_peers`, `vault_autopilot_healthy`, `vault_autopilot_failure_tolerance`	telemetry	cluster size and health	not healthy for 2 min; tolerance 0 on a three-node cluster

Liveness and the telemetry itself

There is nothing to scrape on the engine or the signer, so no up either: a component is alive while its pushes keep arriving. With the exporter sending push times, time() - timestamp(citadel_engine_policy_version) is the age of the engine’s last push (and citadel_signer_nats_connected for the signer). Alert when it passes three push intervals — 180 s at the default 60 s. otlp_logs_dropped_total above 0 means log lines reached only the component’s own output.

7. Counting right

Two habits of Prometheus meet two habits of these components:

A counter series is born at its first event. The components create a series the first time it counts, so Prometheus never sees it at 0 and increase() drops that first event — the first signer refusal ever would never fire an alert. Count events in a window W as
clamp_min(M - M offset W, 0) or (M unless M offset W) or increase(M[W])
(a series that did not exist W ago counts in full; one that is gone by now — a replaced pod — falls back to increase()).
A replaced pod’s last value lingers for minutes, beside the new pod’s. Take state gauges only from fresh pushes, M unless (time() - timestamp(M) > 180), and alert on connectivity per instance rather than on max() over instances, which would let the old pod’s last 1 hide the new pod’s 0.

8. Alerts

Prometheus rules

# Example alerts for a merchant stack. The engine and the signer push every
# OTEL_METRIC_EXPORT_INTERVAL (60 s by default), so "down" is measured as the age of the last
# push, never as a failed scrape: there is nothing to scrape. The 180 s below are three
# default intervals; scale them if you push less often.
#
# Counting events: the components create a counter series at its FIRST event, so Prometheus
# never sees it at 0 and increase() drops that first event — the first signer refusal ever
# would not fire. Every "how many in the last W" below is therefore
#   clamp_min(M - M offset W, 0) or (M unless M offset W) or increase(M[W])
# (a series that did not exist W ago counts in full; one that is gone by now — a replaced
# pod — falls back to increase()).
#
# State gauges (connected, lease, policy version, health) go through
#   M unless (time() - timestamp(M) > 180)
# so a replaced pod's last value, which Prometheus would keep answering for its 5-minute
# lookback, does not stand beside the new pod's (the exporter sends the push time).
groups:
  - name: citadel-merchant-stack
    rules:
      # ---- liveness ------------------------------------------------------------------------
      - alert: CitadelEngineDown
        expr: |
          (time() - max(timestamp(citadel_engine_policy_version)) > 180)
          or absent(citadel_engine_policy_version)
        labels: { severity: critical }
        annotations:
          summary: No push from the policy engine for 3 minutes — every payout waits.
      - alert: CitadelSignerDown
        expr: |
          (time() - max(timestamp(citadel_signer_nats_connected)) > 180)
          or absent(citadel_signer_nats_connected)
        labels: { severity: critical }
        annotations:
          summary: No push from the signer for 3 minutes — approved payouts are not signed.
      - alert: CitadelNoActiveEngine
        expr: sum((citadel_engine_lease_active unless (time() - timestamp(citadel_engine_lease_active) > 180))) < 1
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: No engine holds the lease — payouts wait until one does.
      - alert: CitadelTwoActiveEngines
        # 90 s: a replaced pod's last "1" must age out before it counts as a second holder
        expr: count((citadel_engine_lease_active == 1 unless (time() - timestamp(citadel_engine_lease_active == 1) > 90))) > 1
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: Two engines report holding the lease at once.
      - alert: CitadelLeaseFlapping
        expr: sum(changes(citadel_engine_lease_active[30m])) > 4
        labels: { severity: warning }
        annotations:
          summary: The engine lease changed hands more than twice in 30 minutes.

      # ---- connectivity --------------------------------------------------------------------
      - alert: CitadelEngineOffBus
        # per instance: aggregating would let a replaced pod's last "1" hide the new one's 0
        expr: (citadel_engine_nats_connected unless (time() - timestamp(citadel_engine_nats_connected) > 180)) == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Policy engine {{ $labels.service_instance_id }} is not connected to the platform's NATS."
      - alert: CitadelSignerOffBus
        # per instance: aggregating would let a replaced pod's last "1" hide the new one's 0
        expr: (citadel_signer_nats_connected unless (time() - timestamp(citadel_signer_nats_connected) > 180)) == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Signer {{ $labels.service_instance_id }} is not connected to the platform's NATS."

      # ---- policy and signing --------------------------------------------------------------
      - alert: CitadelNoPolicy
        expr: max((citadel_engine_policy_version unless (time() - timestamp(citadel_engine_policy_version) > 180))) == 0
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: No signed policy document in force — the engine refuses every payout.
      - alert: CitadelDenySpike
        expr: |
          sum((clamp_min(citadel_engine_decisions_total{kind="sign",result="refused"} - citadel_engine_decisions_total{kind="sign",result="refused"} offset 10m, 0) or (citadel_engine_decisions_total{kind="sign",result="refused"} unless citadel_engine_decisions_total{kind="sign",result="refused"} offset 10m) or increase(citadel_engine_decisions_total{kind="sign",result="refused"}[10m]))) >= 5
          and
          sum((clamp_min(citadel_engine_decisions_total{kind="sign",result="refused"} - citadel_engine_decisions_total{kind="sign",result="refused"} offset 10m, 0) or (citadel_engine_decisions_total{kind="sign",result="refused"} unless citadel_engine_decisions_total{kind="sign",result="refused"} offset 10m) or increase(citadel_engine_decisions_total{kind="sign",result="refused"}[10m])))
            / sum((clamp_min(citadel_engine_decisions_total{kind="sign"} - citadel_engine_decisions_total{kind="sign"} offset 10m, 0) or (citadel_engine_decisions_total{kind="sign"} unless citadel_engine_decisions_total{kind="sign"} offset 10m) or increase(citadel_engine_decisions_total{kind="sign"}[10m]))) > 0.5
        labels: { severity: warning }
        annotations:
          summary: More than half of the payouts in 10 minutes were refused by the policy.
      - alert: CitadelSignerRefusing
        expr: sum((clamp_min(citadel_signer_requests_total{result="refused"} - citadel_signer_requests_total{result="refused"} offset 10m, 0) or (citadel_signer_requests_total{result="refused"} unless citadel_signer_requests_total{result="refused"} offset 10m) or increase(citadel_signer_requests_total{result="refused"}[10m]))) > 0
        labels: { severity: critical }
        annotations:
          summary: The signer refused to sign — no valid verdict for that digest (not signed by the pinned engine key, a denial, or another digest).
      - alert: CitadelSignerMalformed
        expr: sum((clamp_min(citadel_signer_requests_total{result="malformed"} - citadel_signer_requests_total{result="malformed"} offset 10m, 0) or (citadel_signer_requests_total{result="malformed"} unless citadel_signer_requests_total{result="malformed"} offset 10m) or increase(citadel_signer_requests_total{result="malformed"}[10m]))) > 0
        labels: { severity: warning }
        annotations:
          summary: The signer received a message on its subject that is not a request at all.
      - alert: CitadelSignerFailing
        expr: sum((clamp_min(citadel_signer_requests_total{result="failed"} - citadel_signer_requests_total{result="failed"} offset 10m, 0) or (citadel_signer_requests_total{result="failed"} unless citadel_signer_requests_total{result="failed"} offset 10m) or increase(citadel_signer_requests_total{result="failed"}[10m]))) > 0
        labels: { severity: warning }
        annotations:
          summary: The signer could not get a signature from its OpenBao.
      - alert: CitadelApprovedNotSigned
        expr: |
          sum((clamp_min(citadel_engine_decisions_total{kind="sign",result="approved"} - citadel_engine_decisions_total{kind="sign",result="approved"} offset 15m, 0) or (citadel_engine_decisions_total{kind="sign",result="approved"} unless citadel_engine_decisions_total{kind="sign",result="approved"} offset 15m) or increase(citadel_engine_decisions_total{kind="sign",result="approved"}[15m])))
            - (sum((clamp_min(citadel_signer_requests_total{op="sign"} - citadel_signer_requests_total{op="sign"} offset 15m, 0) or (citadel_signer_requests_total{op="sign"} unless citadel_signer_requests_total{op="sign"} offset 15m) or increase(citadel_signer_requests_total{op="sign"}[15m]))) or vector(0)) > 0
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: The engine approved payouts the signer never answered — lost between them.
      - alert: CitadelPublishErrors
        expr: |
          (sum((clamp_min(citadel_engine_publish_errors_total - citadel_engine_publish_errors_total offset 10m, 0) or (citadel_engine_publish_errors_total unless citadel_engine_publish_errors_total offset 10m) or increase(citadel_engine_publish_errors_total[10m]))) or vector(0))
            + (sum((clamp_min(citadel_signer_publish_errors_total - citadel_signer_publish_errors_total offset 10m, 0) or (citadel_signer_publish_errors_total unless citadel_signer_publish_errors_total offset 10m) or increase(citadel_signer_publish_errors_total[10m]))) or vector(0)) > 0
        labels: { severity: warning }
        annotations:
          summary: A verdict or a signature could not be published to NATS.

      # ---- OpenBao -------------------------------------------------------------------------
      - alert: OpenBaoSealedOrDown
        expr: max by (vault) ((httpcheck_status{http_status_class="2xx"} unless (time() - timestamp(httpcheck_status{http_status_class="2xx"}) > 180))) == 0
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "OpenBao ({{ $labels.vault }}) is sealed or not answering."
      - alert: OpenBaoRaftUnhealthy
        expr: min by (vault) ((vault_autopilot_healthy unless (time() - timestamp(vault_autopilot_healthy) > 180))) == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "OpenBao ({{ $labels.vault }}) raft cluster is not healthy."
      - alert: OpenBaoSlowSigning
        expr: |
          sum by (vault) (rate(vault_route_update_multichainsign_sum[5m]))
            / sum by (vault) (rate(vault_route_update_multichainsign_count[5m])) > 500
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "OpenBao ({{ $labels.vault }}) takes over 500 ms per key operation."

      # ---- the pipeline itself -------------------------------------------------------------
      - alert: CitadelTelemetryDropping
        expr: sum((clamp_min(otlp_logs_dropped_total - otlp_logs_dropped_total offset 15m, 0) or (otlp_logs_dropped_total unless otlp_logs_dropped_total offset 15m) or increase(otlp_logs_dropped_total[15m]))) > 0
        labels: { severity: warning }
        annotations:
          summary: Log records did not reach the collector (they are still in the component's own output).

9. The dashboard

The “Citadel merchant stack” Grafana dashboard is one JSON file, citadel-merchant-stack.dashboard.json, published with each release beside the other artifacts and their SHA256SUMS. Download both from <download-url>/<version>/, check the hash and import the file in Grafana (Dashboards → New → Import):

grep ' citadel-merchant-stack.dashboard.json$' SHA256SUMS | sha256sum -c -

It reads exactly what the configurations above produce, and expects:

a Prometheus data source with uid prometheus and a Loki one with uid loki (give yours these uids, or replace them in the file before the import);
the label vault set to engine and seed by the two vault collectors;
for its OpenBao log panel, OpenBao’s log (section 5); without it that panel stays empty and the rest is unaffected.

Row	Panels	Read it as
Overview	engine and signer last push, active engines, policy version, both NATS links, both vaults, firing alerts	all green and pushes a few seconds old = the stack can sign
Policy engine	decisions, refusals by reason, refused share, approved ↑ / answered ↓, policy version, derivations, owned wallets, lease and publish errors	a red bar on decisions has its reason next to it; an approval with nothing under it on approved ↑ / answered ↓ was lost
Signer	requests by result, failures by reason, NATS connection, NATS events	`failed` with reason `openbao` = the seed vault; `refused` = a verdict that does not check out
OpenBao	seal state, time per key operation, operations per minute, raft	red seal state = signing is down until it is unsealed
Logs	engine refusals with their full reason, signer warnings, OpenBao, warnings and errors per service	the line behind every red bar

Host and container resources (CPU, memory, disk) are not on it: take them from what your platform already collects for its nodes and pods.

What is not measured

Latency of a decision or a signature — the components push counters and gauges only. The time per key operation inside OpenBao is the closest measure.
Traces — none are emitted.
The engine’s NATS events — it logs no disconnects or reconnects; you get its nats_connected gauge, sampled at every push.
Holds for approval — approval rules hold a payout on the platform, before your engine sees it. Follow them with the operation.awaiting_approval webhook.
Why the signer refused, as a metric attribute — only in its log line (hence citadel_signer_refusals_total, derived).

Get started

Merchant stack

Conventions

Notifications

Policy

1. Point the components at the main collector

2. OpenBao

3. The main collector

4. A vault collector beside each OpenBao

5. OpenBao’s log (optional)

6. What to watch

Policy engine

Signer

OpenBao, both vaults

Liveness and the telemetry itself

7. Counting right

8. Alerts

9. The dashboard

What is not measured

​1. Point the components at the main collector

​2. OpenBao

​3. The main collector

​4. A vault collector beside each OpenBao

​5. OpenBao’s log (optional)

​6. What to watch

​Policy engine

​Signer

​OpenBao, both vaults

​Liveness and the telemetry itself

​7. Counting right

​8. Alerts

​9. The dashboard

​What is not measured

1. Point the components at the main collector

2. OpenBao

3. The main collector

4. A vault collector beside each OpenBao

5. OpenBao’s log (optional)

6. What to watch

Policy engine

Signer

OpenBao, both vaults

Liveness and the telemetry itself

7. Counting right

8. Alerts

9. The dashboard

What is not measured