Activity Export to Monitoring / SIEM

Grizzly can stream its activity/audit log to your systems two ways:

Webhooks — signed JSON batches POSTed to your endpoint (Standard Webhooks format)
OTLP — OpenTelemetry log records sent to any OTLP/HTTP collector or vendor endpoint

Both are driven by a durable cursor over the activity store, with these delivery semantics:

At-least-once, ordered per integration, dedupe by event id. A batch that fails is retried in full; consume idempotently by id.

The event envelope (v1)

{
  "schemaVersion": "1",
  "type": "activity",                 // or "heartbeat" — see below
  "id": "activity-6f1c…",             // dedupe on this
  "timestamp": "2026-07-06T18:22:31.114Z",
  "action": "encrypt",
  "outcome": "success",               // or "denied"
  "reasons": ["The API Key does not have the entitlements"],  // array, only when denied
  "reasonCodes": ["missing_entitlements"],                    // same length and order as reasons
  "actor": { "apiKeyId": "…", "accountId": "…", "accountUid": "…" },
  "keyring": { "name": "…", "id": "…" },
  "service": { "name": "dashboard-api" },
  "asset": { },
  "metadata": { }
}

v1 only ever gains optional fields. Unknown keys appear under metadata — ignore what you don't recognize.

Why a denial has both `reasons` and `reasonCodes`

Key your rules on reasonCodes, not on reasons. reasons is human-readable English intended for a person reading an alert; its exact wording may be improved over time. reasonCodes is a stable identifier per cause, and the two arrays are the same length and the same order, so reasonCodes[i] is the code for reasons[i].

code	what it means
`invalid_key`	the API key presented does not exist
`key_inactive`	the key exists but is deactivated
`key_not_yet_active`	the key's activation window has not opened
`key_expired`	the key's validity period has passed
`usage_limit_exceeded`	the key's usage limit for this action was reached
`missing_entitlements`	the key does not hold an entitlement the action required
`missing_permissions`	the request did not satisfy the permissions the route required
`unresolved_reference`	the request named a key or KeyRing that could not be resolved, so entitlements could not be evaluated at all
`caller_supplied`	the text in `reasons` was written by the internal service that reported the denial, not chosen from the list above

reasonCodes may be absent on a denial that has reasons, and that is not an error. It means the event was recorded before codes existed. It is deliberately not back-filled by matching the English, because that is the coupling codes exist to remove. Treat an absent reasonCodes as "unknown cause", not as "no cause" — reasons still carries the text.

Codes are added the same way any other field is: additively. Treat an unrecognized code as an unclassified denial rather than discarding the event, the same as any unknown key under metadata.

If you also read the X-Grizzly-Auth-Failure header, one code differs

The API returns a short failure token on a rejected request — in the reason field of a 401 body and in the X-Grizzly-Auth-Failure response header. That token and reasonCodes agree for every condition except one:

condition	`reasonCodes` (this feed)	`reason` / `X-Grizzly-Auth-Failure`
activation window has not opened	`key_not_yet_active`	`key_not_yet_valid`
validity period has passed	`key_expired`	`key_expired`
usage limit reached	`usage_limit_exceeded`	`usage_limit_exceeded`

Both spellings are stable and neither is going to change without notice. If you correlate request-time rejections against this export feed, match that pair explicitly rather than joining on the token.

Webhooks

Verifying deliveries

Deliveries follow the Standard Webhooks spec, so use the reference library — do not hand-roll verification:

npm install standardwebhooks

import { Webhook } from "standardwebhooks"

const wh = new Webhook(process.env.GRIZZLY_WEBHOOK_SECRET)   // the whsec_… value from creation

// In your handler — rawBody must be the EXACT raw request body string:
const payload = wh.verify(rawBody, {
  "webhook-id": req.headers["webhook-id"],
  "webhook-timestamp": req.headers["webhook-timestamp"],
  "webhook-signature": req.headers["webhook-signature"]
})

Reference libraries exist for Python, Go, Java, Ruby, PHP, Rust, C# and more — same three headers everywhere.

The request body is { "schemaVersion": "1", "deliveryId": "…", "events": [ … ] }; deliveryId equals the webhook-id header. Respond with any 2xx to acknowledge. Anything else (or a timeout) means the whole batch is retried later.

Timestamps and retries — read this before setting a tolerance

We recommend rejecting deliveries whose webhook-timestamp is outside a 5-minute window (the ecosystem default; the reference libraries enforce it). Grizzly retries failed batches with backoff of up to 15 minutes, and every retry is re-signed with a fresh webhook-timestamp precisely so those retries still pass your tolerance window. webhook-id stays constant across retries of the same batch — dedupe attempts on it, and dedupe events on event.id.

Rotating your signing secret (zero downtime)

Deliveries are signed with every active secret — the webhook-signature header carries a space-delimited list and your library tries each. Rotation is therefore:

Add a new secret (dashboard → integration → Secrets → Add). It is shown exactly once.
Deliveries now carry two signatures. Migrate your receiver at your own pace — the old secret keeps verifying throughout.
Retire the old secret. Grizzly refuses to remove the last one, so deliveries are never unsigned.

Heartbeats — alarm on silence

With no activity to export, Grizzly sends a heartbeat event ("type": "heartbeat") at a fixed cadence (default: 15 minutes). Alert on absence, not just on errors — a paused or broken export otherwise looks identical to a quiet system:

SIEM rule: alert when no Grizzly event (any type) has arrived for 1 hour.

OTLP

Point the integration at any OTLP/HTTP endpoint. If your URL has no path, /v1/logs is appended automatically; a URL with an explicit path is used verbatim. Configured auth headers are stored encrypted and attached to every request.

Each activity becomes one log record: body is the full envelope as a JSON string, severityNumber 9/INFO (13/WARN for denied), and grizzly.* attributes (grizzly.type, grizzly.action, grizzly.outcome, grizzly.activity_id, grizzly.account_id, grizzly.keyring) for indexing without parsing the body.

Partial success: if your collector accepts a batch but rejects some records (partialSuccess), Grizzly does not retry them — the same records would reject again forever. The rejection count appears on the integration's status in the dashboard; fix the schema disagreement there.

Rotating auth headers

Auth-header values are write-only — they are stored encrypted and never returned by a read, so the dashboard cannot show you the current value. To change one, send the full header set you want to PATCH /integrations/:id:

PATCH /integrations/{id}
Content-Type: application/json

{ "headers": { "x-honeycomb-team": "<new-key>" } }

Three things to know:

The set is replaced wholesale, not merged. Send every header you want to keep. Omitting one removes it — which is how a header gets deleted, since there is no separate delete.
The change applies to the next delivery. There is no dual-credential window like webhook signing secrets have, so rotate in the order your vendor supports: if the old key is revoked before the PATCH lands, the deliveries in between fail and retry, and no events are lost.
Either transport. Auth headers work on webhook integrations as well as OTLP ones. A webhook integration carries both credentials, and they do different jobs: the signature proves to your receiver that a delivery came from us, and the auth header authenticates us to a destination that demands one (Splunk HEC, Cribl, Panther). Configured header names are lower-cased, and names reserved by the webhook protocol itself are dropped rather than sent, so a configured header can never shadow webhook-signature.

Header names must be valid HTTP tokens and values must not contain newlines; both are rejected at the write rather than at delivery time.

Editing one header at a time

PATCH /integrations/:id with a headers object replaces the whole set — anything you omit is removed. That is the right shape when you know every header, and the wrong one when you want to rotate a single token, so there are per-header routes as well:


`GET /integrations/:id/headers`	the header names you have configured
`PUT /integrations/:id/headers/:name`	set one, leaving the rest alone — body `{"value": "…"}`
`DELETE /integrations/:id/headers/:name`	remove one

GET returns names and never values. A header value is a credential, so it is write-only in the same way a signing secret is: stored encrypted, never returned by any read, and redacted out of error messages — as sent, and as a destination echoes it back JSON-escaped or percent-encoded. An echo in any other encoding — for example HTML entities, base64, \x or %u escapes, or two encodings combined — is not recognised, and neither is an echo of only part of the value. Nothing — not the dashboard, not this API, not support — can read it back. If you lose it, set a new one.

The dashboard uses exactly these routes. Editing an integration lists the header names you have set, with an Add / Replace control that writes one at a time and a per-header remove. A "Replace ALL headers on save" switch is there for the case where you do mean the wholesale PATCH, and it warns you which headers it is about to remove.

The name goes in the URL and the value goes in the body, deliberately: URLs get logged, cached and put in bug reports, and a credential should not be in one.

OAuth2 destinations

Some destinations will not take a static header — they want a bearer token that you mint and refresh. Pick the Generic endpoint (OAuth2 client credentials) destination and supply the client instead of a header:

Field
`tokenUrl`	where the client-credentials grant is exchanged for an access token
`clientId`	your client identifier
`clientSecret`	write-only, exactly like a header value
`scope`	optional, sent as the grant's `scope` parameter

Grizzly mints a token, caches it, and re-mints it shortly before it expires. A 401 from your destination earns one forced re-mint before it is treated as a real credential failure — a token that merely aged out between being issued and being read should not pause your export, and a revoked one should.


`GET /integrations/:id/oauth`	the client without its secret; `secretSet` says whether one is stored
`PUT /integrations/:id/oauth`	replace the client — body `{"oauth": {…}}`

Unlike headers there is no per-field merge and no delete. An OAuth client is a single credential: a clientId from one tenant next to a clientSecret from another is broken rather than partially updated, so it is replaced whole. And an OAuth integration with no client cannot deliver at all, so there is nothing to remove — rotate by PUTting a new one.

Replacing the client discards the cached access token. This is deliberate. The cached token was minted by the client you just replaced; keeping it would mean your next deliveries carry a token issued to the old registration, which works until you tear that registration down — at which point the failure arrives with nothing to connect it to the change that caused it.

In the dashboard the OAuth fields appear only for a destination that uses them, and in edit mode Replace client applies on its own button, separately from Save. Renaming an integration must not rotate its credential.

tokenUrl is checked at configuration time the same way the destination URL is, because it is a request we will make on your behalf.

Naming your service (`resourceAttributes`)

Every OTLP export carries the resource attribute service.name: grizzly. Destinations use it to group your data — New Relic, for example, creates a service entity literally named grizzly in your account from it.

Set config.resourceAttributes on the integration to change that, or to add any other resource attribute your platform groups by:

{
  "config": {
    "url": "https://otlp.example.com",
    "resourceAttributes": {
      "service.name": "acme-audit",
      "deployment.environment": "production"
    }
  }
}

Anything you set here overrides our default for the same key, so service.name becomes exactly what you supply. Useful when you run more than one Grizzly environment into one destination and need them told apart.

Tuning batch size

Grizzly sends up to a fixed number of events per request. To send smaller batches — because your collector caps request size, or you want lower per-request latency — set batchSize in the integration's config:

PATCH /integrations/{id}
Content-Type: application/json

{ "config": { "url": "https://collector.example.com", "batchSize": 50 } }

It is a ceiling you lower, not raise: values above the server-wide limit are clamped to it, and a missing, zero, negative or unparseable value falls back to the server-wide default. Send the whole config object — like headers, it is replaced rather than merged.

Smaller batches mean more requests for the same volume; the export stays ordered and at-least-once either way.

Splunk HEC /raw: batch size is a correctness setting, not a tuning one

Everywhere else on this page, batchSize trades requests against latency and nothing is lost at the default. Splunk's HEC raw endpoint is the exception. It stores the whole POST body as one Splunk event and truncates that event at TRUNCATE — 10,000 bytes by default in props.conf — while still answering 200 {"text":"Success","code":0}. Acceptance is not retention, and nothing in the response distinguishes the two.

Measured against splunk/splunk:9.4.0 with 1,000 activities: at the default batch of 100, 259 of 1,000 records survived; at batchSize: 20, all 1,000 did (largest stored event 8,032 bytes). Set batchSize to 20 on a Splunk HEC raw integration, or raise TRUNCATE for your sourcetype if you would rather change Splunk than the batch.

This does not apply to the HEC /event endpoint reached through a collector: the collector's splunk_hec exporter posts one event per record, measured at 265–444 bytes each, nowhere near the limit.

Grafana Cloud: a single oversized activity, and batchSize cannot help

The advice above is to lower batchSize. Here that does nothing, and the reason is worth stating plainly: Grafana Cloud's ceiling is per record, not per request. Smaller batches still contain the same record.

Measured 2026-09-16 against otlp-gateway-prod-us-west-0.grafana.net: a log entry over 262,144 bytes is refused with 400 max entry size '262144' bytes exceeded. Grizzly puts the whole activity as JSON into the log line, so one activity carrying a very large metadata or asset payload can cross it while every other record in the batch is fine.

A 400 is permanent — the delivery is not retried indefinitely; the integration pauses after three attempts with the gateway's message on lastError, which names the limit. The fix is to reduce what goes into that activity, or to send it somewhere without the limit; there is no batch setting that works around it.

A separate 65,536-byte ceiling applies to Grafana's structured metadata. Grizzly's own six attributes are far too small to meet it, but resourceAttributes is yours to set, and Grafana refuses every delivery that carries an oversized block. Grizzly therefore refuses an OTLP integration whose resourceAttributes total more than 32,768 bytes of UTF-8 keys and values, with a 400 when you create or update it. The total includes service.name — yours if you set it, otherwise our default service.name: grizzly, which counts as 19 bytes. 32,768 is half of Grafana's 65,536, because the attributes on each record count against the same limit.

Some keys become Grafana stream labels instead, and labels have tighter limits of their own: service.name, service.namespace, service.instance.id, deployment.environment, deployment.environment.name, cloud.region, cloud.availability_zone, container.name, and k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.container.name, k8s.deployment.name, k8s.replicaset.name, k8s.statefulset.name, k8s.daemonset.name, k8s.cronjob.name and k8s.job.name. Grizzly refuses a value over 2,048 bytes for any of these, and more than 16 of them on one integration, counting service.name. Both limits were measured against Grafana Cloud on 2026-09-22 — the 2,048-byte boundary on service.name and deployment.environment, the count by adding keys in the order listed — and are assumed to hold for the other keys. A Grafana stack configured to promote other keys to labels is not covered.

Grizzly applies these resourceAttributes checks to every OTLP integration, not only Grafana Cloud ones. The per-record line limit is Grafana's own: Honeycomb accepted log lines of 64 KiB, 256 KiB, 1 MiB and 4 MiB with 200 on the same date.

OpenTelemetry Collector recipe

receivers:
  otlp:
    protocols:
      http:

exporters:
  # pick yours: datadog, splunk_hec, elasticsearch, … (contrib build; the core image carries
  # only otlp/otlphttp/kafka/prometheus/zipkin/file/debug). There is NO `loki` exporter any
  # more — for Loki, use `otlphttp` against its own OTLP endpoint, or skip the collector
  # entirely and point Grizzly there. See the Loki note below.
  debug:
    verbosity: normal

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]

Picking your vendor in the dashboard

The create form has two controls that look similar and are not.

Destination is what the integration is — a generic HTTPS receiver, or one that authenticates with OAuth2. It is stored, and it decides how we talk to your endpoint.

Vendor is optional guidance about the platform you are configuring. Choosing one sets the destination and transport for you, fills in the auth header name the vendor expects, and shows what their endpoint looks like. Everything stays editable afterwards — the guidance is what we know, not a constraint.

The vendors we have something verified to say about are below, in the same two groups the dropdown uses. Direct means the vendor takes our export as it is. Via a collector means it cannot — picking one configures the collector-shaped setup from the recipe above and explains why.

One vendor we investigated is in neither group, because it turned out not to be a destination at all: Splunk Observability Cloud — see the warning below the table. It is kept on this page rather than dropped, because "we measured it and there is nothing there" is worth more to you than silence.

Vendors we have verified

Vendor	Transport	Reach	Endpoint	Auth header	How we know
OpenTelemetry Collector	OTLP	Direct	`http://collector.internal:4318`	—	Measured — `otel/opentelemetry-collector:0.159.0`
Grafana Cloud	OTLP	Direct	`https://otlp-gateway-<zone>.grafana.net/otlp/v1/logs`	`Authorization`	Documented
Grafana Loki (OTLP)	OTLP	Direct	`https://loki.example.com/otlp/v1/logs`	—	Measured — `grafana/loki:3.1.0`
Honeycomb	OTLP	Direct	`https://api.honeycomb.io`	`x-honeycomb-team`	Documented
New Relic	OTLP	Direct	`https://otlp.nr-data.net`	`api-key`	Documented
Uptrace	OTLP	Direct	`http://<uptrace-host>:14318`	`Authorization`	Measured — `uptrace/uptrace:2.0.3`
Sumo Logic (HTTP Source)	Webhook	Direct	`https://endpoint<n>.collection.<region>.sumologic.com/receiver/v1/http/<token>`	—	Documented
Splunk HEC (raw endpoint)	Webhook	Direct	`https://<splunk-host>:8088/services/collector/raw`	`Authorization`	Measured — `splunk/splunk:9.4.0`
Elasticsearch (_bulk API)	OTLP	Via a collector	`http://collector.internal:4318`	—	Measured — `docker.elastic.co/elasticsearch/elasticsearch:8.13.4`
Splunk HEC (event endpoint)	Webhook	Via a collector	`http://collector.internal:4318`	`Authorization`	Measured — `splunk/splunk:9.4.0`
Datadog (Logs Intake)	OTLP	Via a collector	`http://collector.internal:4318`	—	Documented

Splunk Observability Cloud cannot receive the activity export

This row used to read like the others, and it was wrong. Measured on 2026-09-15 against a live us1 organisation: Splunk Observability Cloud publishes no logs ingest endpoint at all. /v1/logs, /v2/logs and /v2/logs/otlp all answer 404 with a valid token, while /v2/datapoint, /v2/event, /v2/datapoint/otlp and /v2/trace/otlp answer 401 when unauthenticated — so routing happens before authentication on that host and the 404 is an absent route, not a credential fault. /v2/datapoint/otlp additionally answers 415 to content-type: application/json, which our OTLP channel sends, so even the OTLP paths that do exist are closed to us.

Splunk Observability ingests metrics, events and traces — not logs. Logs reach it through Log Observer Connect, which is a different product surface and not an OTLP destination we can deliver to. If you need Grizzly activity in Splunk, use Splunk HEC (both rows above are measured against splunk/splunk:9.4.0).

Measured names the exact product build we ran a real export against. Documented means we read the vendor's current published documentation and have not executed against their endpoint — true of the SaaS-only platforms, which we cannot reach without an account. The dashboard shows the same label beside each vendor.

This table is the dashboard's Vendor list, not the list of destinations that work. Anything speaking OTLP/HTTP or taking a plain JSON POST works as a Generic endpoint with a URL you supply, whether or not it has a row here — a row exists only where we have something specific to tell you.

The Grafana Loki (OTLP) row is Loki's working front door, and it is the only Loki we offer. Loki 3.0 and later ingest OTLP natively, so picking it configures a Generic endpoint on the OTLP transport and you supply the full path — https://<loki-host>:3100/otlp/v1/logs. No collector is involved. Measured against grafana/loki:3.1.0 and driven end to end. Loki's other front door, the push API at /loki/api/v1/push, is not offered as a vendor and is refused at creation — see Destinations that are refused at creation below for why, because the failure is silent and worth understanding even though you can no longer configure your way into it.

The endpoint column is a shape, not a value: fill in your own zone, realm, host or token. The auth header column names the header only — you supply its value, and Grizzly never shows it back.

A vendor you do not see listed is not unsupported. It means we have nothing verified to tell you about it; choose Not listed — configure manually and use the generic destination.

Endpoint paths and throttling

/v1/logs is appended only when your URL has no path. A vendor endpoint that already carries a path (like Grafana Cloud's /otlp) is used verbatim, so append the log path yourself — which is why the Grafana Cloud row above prints a full path and Honeycomb's does not.

If an endpoint throttles you with Retry-After, Grizzly honors it, and throttling never counts toward auto-pause. That is a guarantee about Grizzly's behaviour, and it is the part worth relying on: this sentence used to assert that these endpoints do throttle that way, which nobody here had seen. We have since looked. Grafana Cloud and Honeycomb were driven to 8.1 and 9.1 MB/s sustained on 2026-09-16 without either returning a 429 or a Retry-After at all — so the header handling is real, and the claim that you will meet it was not ours to make.

Do not point an integration at Grafana Loki's push API

Loki accepts our events and throws them away, reporting success.

Loki's push API (/loki/api/v1/push) wants {"streams": [...]}. Grizzly sends its own envelope. Loki parses the JSON, finds no streams, stores nothing — and answers 204 No Content, which is a success.

Nothing downstream can tell. Every delivery is a 2xx, the export position advances, no failure is recorded, and the dashboard's Test button reports success. The only symptom is that Loki has no data, and you would find that out whenever you next went looking — which, for an audit trail, is usually during an investigation.

Measured against Grafana Loki 3.1.0. This is Loki behaving as designed; it is not a bug in either product, and it is not something we can detect at delivery time.

Grizzly now refuses a URL ending in /loki/api/v1/push when you create or update an integration, and points you at Loki's OTLP endpoint instead. That check matches on the path, so it is a safety net rather than a guarantee: if you serve Loki behind a reverse proxy at some other path, the refusal will not fire and the silent discard will.

Send to Loki's OTLP endpoint instead — no collector needed. Loki 3.0 and later ingest OTLP natively. Create a Generic endpoint integration on the OTLP transport and type the full path: https://<loki-host>:3100/otlp/v1/logs. We send that URL exactly as you type it — a path is never rewritten — and Loki stores the records, with our grizzly.* attributes arriving as labels. Measured end to end against grafana/loki:3.1.0.

On Loki 2.x, which has no OTLP endpoint, front it with an OpenTelemetry Collector — the recipe above, with an otlphttp exporter. Not a loki exporter: that component was removed from the Collector and no longer exists in either the core or the contrib distribution.

It is the push API that is not reachable — not Loki

The rows above are not a contradiction. Loki has two front doors, and they behave oppositely. /otlp/v1/logs is a genuine OTLP endpoint that understands our envelope, and it works directly. /loki/api/v1/push is a different, older protocol that takes our body, finds no streams key, and answers 204 having stored nothing. Grafana Cloud's OTLP gateway is the same first door under a hosted name, which is why it has always worked.

The one thing to get right is the URL. Point at the OTLP path and you are done; point at the push path and we refuse it.

Destinations that are refused at creation

Measured, so you can tell a configuration mistake from an unsupported destination:

Naming one of these as a destination is refused at creation — before you supply a URL, with the finding, the way out, and the exact product build we ran it against. That refusal lives in the API, so a script or an SDK caller hits it too. The dashboard no longer offers them in the Destination list at all: they appear under Vendor, where picking one configures the collector setup instead of rejecting your form. Choosing Generic endpoint and pasting a Loki push URL is still caught, by a separate URL-path check.

Each row names a specific API, not a product. Loki appears here for its push API while its OTLP endpoint is in the directly-reachable table above; Splunk HEC appears here for /event while its /raw sibling is above. Which URL you paste is what decides it.

Destination	Direct?	What happens
Grafana Loki push API	No	Accepts and silently discards — see above. Refused at creation, by name or by URL path. Loki's other front door, `/otlp/v1/logs`, works directly and needs no collector
Elasticsearch `_bulk`	No	Refused at creation when named. Otherwise refused loudly — `400 "The bulk request must be terminated by a newline [\n]"`. `_bulk` wants newline-delimited action/document pairs, which is not a shape we send. Setting `Content-Type: application/x-ndjson` does not help
Splunk HEC `/services/collector/event`	No	Refused at creation when named. Otherwise refused loudly — `400 {"text":"No data","code":5}`. It wants `{"event": ...}` wrappers. Its sibling `/services/collector/raw` accepts our body and works directly (with one caveat that costs records silently — see the batch-size warning above) — so the two paths behave oppositely and the only difference is the URL you copied. Both authenticate with `Authorization: Splunk <token>`, which a webhook integration can now carry

A loud refusal is the safe case: repeated failures pause the integration and the reason appears on it, so you find out immediately. Loki is called out separately because it is the one that fails quietly.

Operations

Self-hosted deployments: clock synchronisation is required

Skip this if Grizzly runs it for you — it is a requirement on the hosts, not on your receiver.

Activity timestamps are assigned by whichever Grizzly node served the request, and the exporter tails those timestamps in order. A node whose clock runs behind its peers writes activities that land behind the exporter's position, and those activities are never exported — no error, no retry, no gap indicator, because the exporter cannot see rows it has already read past.

Two things follow, and both are requirements rather than recommendations:

Every node writing activities must be clock-synchronised (NTP or chrony), kept within 1 second of its peers.
INTEGRATIONS_WATERMARK_MS must exceed commit latency plus the worst-case skew. The 5000 ms default budgets roughly 1 s of commit latency and 4 s of skew — four times the requirement above, deliberately.

If clock discipline cannot be guaranteed, raise the watermark to cover the real skew. The cost is delivery latency (events are held that much longer before export); the cost of getting it wrong is silent gaps in an audit trail. POST /integrations/:id/replay is the recovery if you discover skew after the fact.

Pause: repeated failures pause an integration (misconfigured endpoints pause within minutes; genuine outages get ~90 minutes of retries first). The position is preserved.
Resume: re-enable the integration in the dashboard; the backlog drains from where it stopped.
Replay: rewind an integration to any timestamp to re-deliver a window (recovery, or backfilling history into a new integration). Replay also clears a pause and resumes delivery, so it works as the recovery action on a paused integration. Replayed events have the same ids — your dedupe handles them.
Test: the dashboard's Test button sends one synthetic event ("action": "integration.test") through the real pipeline without affecting your stream.