Documentation
MidnightDocumentation
Recommended pages

OPERATE SERVICES

Diagnose from evidence, then make one reversible change.

Midnight brings deployment events, build and runtime logs, metrics, monitors, host observations, and audit history into one control plane. Use the narrowest action that restores service and preserve the operation trail.

LogsMetricsMonitorsAuditBackups
On this page

SIGNALS

Start with the layer that is actually failing.

Operation events
Authoritative progress and terminal reason codes for builds, deploys, actions, backups, and restores.
Build logs
Tenant-safe build output associated with a deployment operation.
Runtime logs
Application output from the selected project, service, environment, and deployment.
Metrics
Service and host observations for resource use and HTTP behavior when their real origins are configured.
Monitors
Configured checks, notification channels, and delivery history. A monitor is only useful after its target and channel are verified.
Audit evidence
Mutation admission and actor history. Export and verify it without including credentials or private key material.
Observe a linked servicebash
midnight logs -f --build
midnight logs -f
midnight observability metrics --project-id prj_example --service-id svc_example

midnight monitor create   --project-id prj_example   --environment-id env_example   --service-id svc_example   --name high-memory   --signal memory_percent   --threshold 85
midnight monitor incident list --project-id prj_example
midnight monitor delivery list --project-id prj_example
Export and verify audit evidencebash
MIDNIGHT_URL=https://your-midnight.example
AUDIT_PUBLIC_KEY=/path/to/midnight-audit-public.pem

curl --fail-with-body   "$MIDNIGHT_URL/api/v1alpha/audit/export"   -H "Authorization: Bearer $MIDNIGHT_ADMIN_TOKEN"   --output audit-export.jsonl

midnight audit verify   --from-jsonl audit-export.jsonl   --public-key "$AUDIT_PUBLIC_KEY"   --json

INCIDENT LOOP

Stabilize before changing configuration.

  1. 1

    Name the impact

    Record the affected hostname, service, environment, start time, and latest known-good deployment.

  2. 2

    Capture current evidence

    Save operation IDs, request IDs, reason codes, recent events, host status, and redacted logs before taking action.

  3. 3

    Choose the smallest action

    Restart for a stuck process, redeploy for drift, roll back for a bad release, and restore only for data loss or corruption.

  4. 4

    Verify at every layer

    Check operation success, runtime health, route state, and an external request. A green control-plane row is not enough.

  5. 5

    Close the loop

    Preserve the audit/event trail, explain the cause, and add a monitor or runbook improvement that would shorten the next incident.

RELEASE RECOVERY

Roll back by creating a new release.

Linked servicebash
midnight status
midnight rollback --previous
midnight logs -f --build
midnight logs -f

PERSISTENT DATA

Plan, stage, apply, verify.

A volume restore stages a replacement volume, validates the restore plan, then performs an immutable redeploy to attach the recovered data. It should be treated as a production change, not a retry button.

  1. 1

    Prove the backup

    Confirm its destination, completion state, timestamp, volume identity, and integrity evidence. Off-host storage must be configured.

  2. 2

    Create the restore plan

    Review the exact source backup, target service/environment, expected resource versions, and replacement volume.

  3. 3

    Apply once

    Use a stable idempotency key for the exact plan. Do not create competing restore operations.

  4. 4

    Verify data and service

    Check operation success, runtime health, expected application data, route reachability, and cleanup of superseded staging resources.