Verify cleanup before trusting it on important projects.

Maintenance

Use a dry run before you enqueue deletion.

DRY_RUN=true bin/rails "logister:telemetry:retention[PROJECT_UUID]"
DRY_RUN=false CONFIRM=retention \
  bin/rails "logister:telemetry:retention[PROJECT_UUID]"

# Archive without deleting; rerun when continuation_required is true.
bin/rails "logister:telemetry:archive[ingest_events,30,PROJECT_UUID]"
bin/rails "logister:telemetry:archive[trace_spans,30,PROJECT_UUID]"

The confirmed retention command creates or reuses a durable run and enqueues it. It does not keep the shell open until the complete archive finishes. Manual archive commands advance a bounded object slice and resume the same manifest when repeated.

Verify

Check the Data tab first.

1. Use a low-risk project before changing retention on important projects.
2. Open Project > Settings > Data and review Archive Center.
3. Check the durable run phase, object and row progress, heartbeat freshness, and retained error.
4. Use Coverage for each retention scope and Catalog for recent manifests. V2 keys are loaded from object rows in pages of 20.
5. Confirm the run is completed and no verified manifest is awaiting source cleanup before enabling Require archive before deletion more broadly.
6. If your team uses archive storage, confirm the matching .jsonl.gz object exists and can be opened.

Worker topology

Keep archive memory away from core queues.

Small installations can keep using config/sidekiq.yml. Hosted and higher-volume installations should run config/sidekiq-core.yml for normal work and config/sidekiq-archives.yml for the archives queue at concurrency 1. The default core process reserves three projector threads and two threads for other jobs, including maintenance, so continuous ingestion cannot exclude recovery sweeps. Give the core process a database pool of at least 7 and the archive process at least 3.

Telemetry replay

Keep an incomplete batch available for retry.

In 3.6.10, LOGISTER_BATCHED_PROJECTION=true enables source preloading and atomic delivery acknowledgements. After all projector workers run compatible code, enable it separately from deployment. It defaults to false. Each incomplete ClickHouse batch temporarily stores one compressed copy of its exact HTTP body, so a retry can reuse the same bytes and deduplication key. The final acknowledgement removes that copy.

From 3.6.11, LOGISTER_BOUNDED_PROJECTOR=true separately limits projector admission to three drainers and one waiting wake hint. Enable it consistently on intake and workers after the query and saved-payload rollout checks. It defaults off. Workers check a 25-second cooperative budget between operations, return unstarted work without consuming an attempt, renew ownership before external writes, and recover lost hints through PostgreSQL. Dependency failures add a short cooldown. Disable the flag and restart normally to restore the prior scheduling path; keep saved payloads intact.

Retention protects the whole batch while any member remains incomplete. Old batches without saved payloads require their original source members; missing or changed legacy data stops for investigation. Disabling the setting stops new batch records and drains existing ones. Before downgrading below 3.6.10, verify bin/rails runner 'puts TelemetryProjectionBatch.count' returns zero. Keep compatible workers if incomplete payloads remain; do not clear queues or delete payloads to force a downgrade.

Recovery

Recover from PostgreSQL state, not queue absence.

A one-minute recovery sweep fences stale attempts and re-enqueues due queued, waiting, or retrying runs. If a nonterminal run lost its queue entry, invoke the same idempotent sweep:

bin/rails runner \
  'ProjectRetentionRunRecoverySweepJob.perform_now(Time.current.iso8601(6))'

Do not edit attempt tokens, fences, progress, object checkpoints, or source rows by hand. A terminal failed run is a stop sign: preserve its manifest and source rows, repair the cause, and review the integrity error before creating a successor.