LogisterDocumentationRetention jobs
Data retention / Jobs

Verify cleanup before trusting it on important projects.

The Data tab includes Archive Center because users need proof that archive writes and cleanup are working. Use that visible history before requiring archives before deletion on an important project.

Maintenance

Use a dry run before you enqueue deletion.

shell
DRY_RUN=true bin/rails "logister:telemetry:retention[PROJECT_UUID]"
DRY_RUN=false CONFIRM=retention \
  bin/rails "logister:telemetry:retention[PROJECT_UUID]"

# Archive without deleting; rerun when continuation_required is true.
bin/rails "logister:telemetry:archive[ingest_events,30,PROJECT_UUID]"
bin/rails "logister:telemetry:archive[trace_spans,30,PROJECT_UUID]"

The confirmed retention command creates or reuses a durable run and enqueues it. It does not keep the shell open until the complete archive finishes. Manual archive commands advance a bounded object slice and resume the same manifest when repeated.

Verify

Check the Data tab first.

  1. Use a low-risk project before changing retention on important projects.
  2. Open Project > Settings > Data and review Archive Center.
  3. Check the durable run phase, object and row progress, heartbeat freshness, and retained error.
  4. Use Coverage for each retention scope and Catalog for recent manifests. V2 keys are loaded from object rows in pages of 20.
  5. Confirm the run is completed and no verified manifest is awaiting source cleanup before enabling Require archive before deletion more broadly.
  6. If your team uses archive storage, confirm the matching .jsonl.gz object exists and can be opened.

Worker topology

Keep archive memory away from core queues.

Small installations can keep using config/sidekiq.yml. Hosted and higher-volume installations should run config/sidekiq-core.yml for normal work and config/sidekiq-archives.yml for the archives queue at concurrency 1. Give the archive process a database pool of at least 3.

SettingDefaultPurpose
LOGISTER_RETENTION_OBJECTS_PER_ATTEMPT25Bounds upload, verification, and cleanup work in one attempt.
LOGISTER_RETENTION_STALE_SECONDS900Fences a running attempt after its heartbeat expires.
LOGISTER_RETENTION_MAX_FAILURES10Stops automatic retries after repeated failures.
LOGISTER_RETENTION_ENQUEUE_CLAIM_SECONDS300Suppresses duplicate recovery enqueues.

Recovery

Recover from PostgreSQL state, not queue absence.

A one-minute recovery sweep fences stale attempts and re-enqueues due queued, waiting, or retrying runs. If a nonterminal run lost its queue entry, invoke the same idempotent sweep:

shell
bin/rails runner \
  'ProjectRetentionRunRecoverySweepJob.perform_now(Time.current.iso8601(6))'

Do not edit attempt tokens, fences, progress, object checkpoints, or source rows by hand. A terminal failed run is a stop sign: preserve its manifest and source rows, repair the cause, and review the integrity error before creating a successor.

Stop conditions

Stop advancement if worker memory grows with total manifest size, checksums diverge, stale attempts issue effects, cleanup lacks a fresh object verification, or archive work increases projector or notification queue age.