TROUBLESHOOTING
Find the first broken boundary.
Work from request identity to durable operation, worker, runtime, router, and external probe. Preserve IDs and typed reason codes; redact values rather than deleting the evidence that explains the failure.
On this page
TRIAGE
Collect facts before retrying.
midnight version
midnight whoami
midnight status
curl --fail-with-body https://your-midnight.example/api/v1alpha/compat- 401
- The credential is missing, expired, revoked, or the wrong class for the route. Log in again or supply the intended project token.
- 403
- The authenticated principal lacks the exact capability or resource grant. Changing token shape will not widen authority.
- 409
- The requested state conflicts with current state or an existing singleton. Refresh before deciding whether to retry.
- 412
- Your If-Match or expected resource version is stale. Read the current representation and reapply your intent.
- 429
- The request hit an admission or upload budget. Honor Retry-After and reduce concurrent work.
- 5xx
- Keep the request ID. Before repeating a mutation, check whether its operation exists and reuse the same idempotency key only for unchanged intent.
IDENTITY
The MFA field is conditional.
- 1
Confirm the endpoint
Use the same HTTPS installation URL in the browser and CLI. Remote plain HTTP login is rejected.
- 2
Submit primary credentials
The first login exchange uses email and password. The CLI or console displays the TOTP field only after the server returns
MFA_REQUIRED. - 3
Check the clock
If a current TOTP fails, synchronize both the device and server clock before resetting MFA.
- 4
Revoke stale sessions
Use logout or the console's session controls. Do not copy the installation recovery credential into normal developer configuration.
BUILD AND RELEASE
Separate intake, build, and runtime failures.
- Upload rejected
- Check the 50 MiB compressed human limit, .midnightignore, workspace retention quota, and concurrent upload backpressure.
- Build failed
- Read backed build events with midnight logs -f --build. Confirm the build definition and that required non-secret files were included.
- Image rejected
- Use a complete lower-case @sha256 digest and confirm the operator image-trust policy admits it.
- Release unhealthy
- Inspect runtime logs, configured port and health check, variable metadata, volume mount, and agent observation.
- Operation appears stuck
- Check worker service health and events before cancelling. Restarting the CLI does not advance server-side work.
INGRESS
Test from the backend outward.
SERVER_IP=203.0.113.10
sudo systemctl status traefik midnight-worker midnight-agent
sudo -u midnight test -w /etc/midnight/traefik/dynamic
dig +short app.example.com
curl --resolve "app.example.com:443:$SERVER_IP" https://app.example.com/
curl --fail-with-body https://app.example.com/If --resolve succeeds but normal curl fails, investigate public DNS. If both fail, inspect route apply state, Traefik logs, certificate state, host firewall, and backend health in that order.
SYSTEMD
Verify the single-host dependency chain.
sudo midnight-doctor --json --live
sudo systemctl --no-pager --full status midnight-server midnight-worker midnight-agent midnight-buildkit midnight-buildkit-worker midnight-registry docker traefik
sudo journalctl -u midnight-server -u midnight-worker -u midnight-agent --since "20 minutes ago" --no-pager