MidnightDocumentation

TROUBLESHOOTING

Find the first broken boundary.

Work from request identity to durable operation, worker, runtime, router, and external probe. Preserve IDs and typed reason codes; redact values rather than deleting the evidence that explains the failure.

Request IDReason codeOperation events
On this page

TRIAGE

Collect facts before retrying.

Client and control planebash
midnight version
midnight whoami
midnight status
curl --fail-with-body https://your-midnight.example/api/v1alpha/compat
401
The credential is missing, expired, revoked, or the wrong class for the route. Log in again or supply the intended project token.
403
The authenticated principal lacks the exact capability or resource grant. Changing token shape will not widen authority.
409
The requested state conflicts with current state or an existing singleton. Refresh before deciding whether to retry.
412
Your If-Match or expected resource version is stale. Read the current representation and reapply your intent.
429
The request hit an admission or upload budget. Honor Retry-After and reduce concurrent work.
5xx
Keep the request ID. Before repeating a mutation, check whether its operation exists and reuse the same idempotency key only for unchanged intent.

IDENTITY

The MFA field is conditional.

  1. 1

    Confirm the endpoint

    Use the same HTTPS installation URL in the browser and CLI. Remote plain HTTP login is rejected.

  2. 2

    Submit primary credentials

    The first login exchange uses email and password. The CLI or console displays the TOTP field only after the server returns MFA_REQUIRED.

  3. 3

    Check the clock

    If a current TOTP fails, synchronize both the device and server clock before resetting MFA.

  4. 4

    Revoke stale sessions

    Use logout or the console's session controls. Do not copy the installation recovery credential into normal developer configuration.

BUILD AND RELEASE

Separate intake, build, and runtime failures.

Upload rejected
Check the 50 MiB compressed human limit, .midnightignore, workspace retention quota, and concurrent upload backpressure.
Build failed
Read backed build events with midnight logs -f --build. Confirm the build definition and that required non-secret files were included.
Image rejected
Use a complete lower-case @sha256 digest and confirm the operator image-trust policy admits it.
Release unhealthy
Inspect runtime logs, configured port and health check, variable metadata, volume mount, and agent observation.
Operation appears stuck
Check worker service health and events before cancelling. Restarting the CLI does not advance server-side work.

INGRESS

Test from the backend outward.

Route triagebash
SERVER_IP=203.0.113.10
sudo systemctl status traefik midnight-worker midnight-agent
sudo -u midnight test -w /etc/midnight/traefik/dynamic
dig +short app.example.com
curl --resolve "app.example.com:443:$SERVER_IP" https://app.example.com/
curl --fail-with-body https://app.example.com/

If --resolve succeeds but normal curl fails, investigate public DNS. If both fail, inspect route apply state, Traefik logs, certificate state, host firewall, and backend health in that order.

SYSTEMD

Verify the single-host dependency chain.

Host evidencebash
sudo midnight-doctor --json --live
sudo systemctl --no-pager --full status   midnight-server midnight-worker midnight-agent midnight-buildkit   midnight-buildkit-worker midnight-registry docker traefik
sudo journalctl -u midnight-server -u midnight-worker -u midnight-agent   --since "20 minutes ago" --no-pager