Troubleshooting | Logister

Quick checks

Confirm the baseline before chasing optional services.

Open /up. If it fails, fix the web process, database connection, or deploy health first.
For an unclaimed instance, confirm RAILS_ENV, RAILS_MASTER_KEY, DATABASE_URL, and LOGISTER_SETUP_TOKEN. After claim, inspect effective values and environment overrides under Admin → Installation.
Confirm one web process and one Sidekiq worker process are running from the same release image or source revision.
Create a project, create an API key on its setup path, and keep the token available for one direct test request.
Look at Rails and worker logs for request IDs, validation errors, authentication failures, and job errors around the same timestamp.

Start from a fresh token

If a project was archived and restored, old active tokens were revoked during archive. Generate a new API key before testing ingestion again.

Nothing in Issues

First decide whether the event is being accepted.

An empty Issues list can mean the app never received the event, the token was rejected, the payload failed validation, the event type was not error, or the event landed in Explore › Events, Explore › Charts, Performance, or Monitors instead of Issues. While a project is still waiting for data, the First event step of its setup path (Settings > Setup) checks for any accepted event on its own and lists the usual causes when nothing arrives.

shell

curl -i "$LOGISTER_ENDPOINT/api/v1/ingest_events" \
  -H "Authorization: Bearer $LOGISTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "event": {
      "event_type": "error",
      "level": "error",
      "message": "Troubleshooting test error",
      "fingerprint": "docs-troubleshooting-test",
      "environment": "production",
      "context": {
        "service": "manual-test"
      }
    }
  }'

A successful write returns 201 Created with an accepted status and an id. If it is accepted but Issues still looks empty, check the project you sent to, the selected status filter, search terms, environment or release filters, archived project views, and whether the event was grouped into an existing issue.

Rejected events

Use the status code to narrow the fix.

Status	Likely cause	What to check
`400`	The JSON body is missing the required `event`, `EVENT`, `check_in`, or `CHECK_IN` envelope.	Compare the request with the HTTP API guide. Some runtimes uppercase struct keys, which Logister accepts.
`401`	The token is missing, invalid, revoked, inactive, or belongs to an archived project or one pending deletion.	Use `Authorization: Bearer <token>` or `X-Api-Key: <token>`, generate a fresh project token, and confirm the project is active.
`422`	The envelope exists, but required event, span, or check-in fields did not validate, or the envelope, `context`, or `evidence` was sent as a JSON array instead of an object.	Read the returned `errors` array. For spans, confirm `trace_id`, `span_id`, name/message, and duration fields. For check-ins, keep the slug under 200 characters, start it with a letter or number, and reuse existing slugs: a project holds up to 500 monitors, and new slugs past that limit are refused.
`429`	The API token or authentication-failure source exceeded a public API rate limit.	Honor `Retry-After`. Database-backed app administrators and break-glass admins from `LOGISTER_ADMIN_EMAILS` can tune project overrides or global rate-limit environment variables.

If self-observability is configured with LOGISTER_API_KEY and LOGISTER_ENDPOINT, rejected client submissions are also reported through the Ruby gem as sanitized warning logs.

Background jobs

Run Sidekiq whenever the app needs async work.

Sidekiq delivers email, writes optional ClickHouse copies, schedules digests, dispatches project notifications, and runs operator-triggered archive or prune tasks. The web app can accept events while some async work is delayed, so always check the worker when the UI accepts data but follow-up behavior is missing.

shell

bundle exec sidekiq -C config/sidekiq.yml

Confirm REDIS_URL is identical for web and worker processes.
Confirm the worker has the same secrets and optional service variables as the web process.
Look for retrying jobs when ClickHouse, SMTP, or archive storage credentials are wrong.

Sign-in and access

Check host settings, account lockout, and CLI approvals.

These checks cover pages that refuse to load, people who cannot sign in, and CLI logins waiting for approval.

Symptom	Check
Every page returns `403 Blocked hosts`	The request `Host` is not the host in `LOGISTER_PUBLIC_URL`. Add other names your proxy forwards, such as an internal hostname, to `LOGISTER_ALLOWED_HOSTS` and restart the web process.
The browser console shows `Refused to load the script` or `Content-Security-Policy` errors	A script is not on the allowed list. Check that the Probo Banner script URL in Admin → Installation → Public site is a full `https://` address. While you investigate, `LOGISTER_CSP_REPORT_ONLY=1` reports violations without blocking them.
A user cannot sign in with the right password	After 10 failed attempts the account locks for one hour and Logister emails an unlock link. The sign-in form still says the email or password is invalid. Wait an hour, use the emailed link, or request a new one at `/users/unlock/new`.
Users are signed out after a break	Idle sessions end after `LOGISTER_SESSION_TIMEOUT_MINUTES` (default `120`). Sessions started with Remember me do not time out.
No project alerts	Confirm the user's notification path, worker health, quiet hours, hourly limits, and that the event created the configured alert condition.
A user cannot sign in after several attempts	The account locks for one hour after 10 failed sign-ins. The user can wait, or open Unlock account on the sign-in page and follow the emailed link. Sign-in errors do not say that an account is locked, so check the unlock email first.
A user is asked to sign in again after a break	Idle sessions end after `LOGISTER_SESSION_TIMEOUT_MINUTES` (default 120). Sessions started with Remember me stay signed in.
The sign-up page says sign-up is unavailable	Check Admin → Installation → Authentication → Account sign-up or `LOGISTER_REGISTRATION_MODE`. With allowed domains, the email domain must be listed exactly; subdomains are listed separately.
The CLI keeps waiting after `logister auth login`	Open the verification page, choose at least one project, and press Approve. The page shows when and from which IP address the login started. Deny any request you did not start yourself.
`/health/release` or `/health/clickhouse` returns 401	Send `Authorization: Bearer` with the value of `LOGISTER_HEALTH_TOKEN`, or open the page while signed in as an application admin. Use `/up` for platform health checks.

Read security and access settings for the variables behind these checks.

Email

Check SMTP, sender identity, and worker delivery.

Logister sends confirmation mail, password reset mail, project alerts, digests, monitor notices, workflow notifications, and operational notices through Action Mailer. In production-like environments, configure SMTP and keep Sidekiq running.

Symptom	Check
No auth mail	Confirm SMTP host, port, username, password, TLS/SSL mode, sender address, and provider-side domain verification.
No digests	Confirm recurring jobs are scheduled, Sidekiq is running, and the selected daily or weekly cadence has elapsed.

Read deployment email settings for SMTP variables, or read notification settings for the product paths users configure.

GitHub source lookup

Use attempted lookups when the repo is connected but the file is missing.

A synced GitHub repository is not enough by itself. Source lookup needs a connected project repository mapping, the right runtime path prefix, the right repository source root, and a branch or commit ref that contains the file.

Symptom	Check
Configured repo, but source file not found	Read the attempted repository, path, ref, and reason. A path like `app/app/controllers/...` usually means the source root or runtime prefix is duplicating a path segment.
No repositories available	Install or link the GitHub App installation, accept permission changes, then click Sync repositories.
No source repository mapping	Click Connect for the synced repository in the project's Settings › Integrations.

Read source lookup diagnostics for the focused checklist.

ClickHouse

Separate ClickHouse health from the active dashboard read mode.

ClickHouse is optional, and PostgreSQL remains the durable system of record. A healthy ClickHouse connection does not prove supported dashboard reads moved away from PostgreSQL.

Readiness endpoint

curl -H "Authorization: Bearer $LOGISTER_HEALTH_TOKEN" https://logister.example.com/health/clickhouse

Symptom	Likely cause and check
The endpoint returns `401` with `{"error":"unauthorized"}`.	Health reports need an application admin session or `Authorization: Bearer` with the value of `LOGISTER_HEALTH_TOKEN`. Set the variable on the web process and send it from your monitor. `/up` stays public for liveness checks.
Overview graphs take tens of seconds or minutes while the health endpoint returns `200`.	Inspect `clickhouse_mode` in the response. `dual_write` intentionally keeps supported dashboard reads on PostgreSQL. Pass the stable-window coverage check, then switch to `read_preferred`.
The mode stays on `dual_write` after changing the admin setting.	Check the environment override named beside the field. `LOGISTER_CLICKHOUSE_ENABLED=true` is a legacy dual-write-only setting. Remove it or replace it with `LOGISTER_CLICKHOUSE_MODE=read_preferred`, then restart web and worker processes.
The mode is `read_preferred`, but a graph request is still slow.	Look for `clickhouse dashboard explorer failed` or `clickhouse dashboard rollup failed` in web logs. A failed ClickHouse query falls back to PostgreSQL, which can be slow at high event volume. Fix reachability, credentials, schema, or query errors before retrying.
The endpoint reports missing tables or schema issues.	Run the idempotent schema repair or load `docs/clickhouse_schema.sql`, then rerun the health and coverage checks.
ClickHouse receives no new events.	Check Sidekiq, worker credentials, and retry logs. The web process accepts PostgreSQL events before the queued ClickHouse copy is written.

Archive exports

Archive storage is separate from normal event ingestion.

S3-compatible archive storage is optional and used when operators export compressed JSONL before pruning older hot telemetry. Normal app events are stored in PostgreSQL even when archive storage is not configured.

Set ACTIVE_STORAGE_SERVICE=amazon and the AWS or S3-compatible variables from the deployment guide.
Confirm the bucket exists and credentials can write objects.
Run archive tasks from the same release with access to the same database and storage variables.
Do not run prune tasks until you have verified the archive output you need.

Read data retention for the product model, storage setup, and verification steps.

Support context

Collect a small, useful bundle before asking for help.

Logister version, image tag, or git SHA.
Deployment shape: source, Docker, Compose, Fly, Kamal, or another platform.
Whether PostgreSQL, Redis, Sidekiq, SMTP, ClickHouse, Turnstile, analytics, and archive storage are enabled.
The failing endpoint, status code, response body, and relevant response headers.
Sanitized Rails and worker log lines around the same timestamp.
Never share raw API tokens, credentials, cookies, authorization headers, or private event payloads.

Troubleshoot Logister.

Debug the layer that can explain the symptom.