Slack ETL

Slack ETL keeps an indexed, queryable copy of public Slack history in Postgres for agent context and operator workflows. It runs as scheduled Centaur workflows: one workflow keeps recent channel history fresh, one drains deferred historical backfill work, and one turns synced messages into company context documents. See Creating Workflows for the durable workflow model behind these jobs.

The ETL path is separate from Slackbot delivery. Slackbot handles live user turns in Slack threads; Slack ETL reads Slack history with a dedicated user token and writes durable rows into Postgres.

What it runs

The schedules are registered from the workflow files at API startup. Each workflow uses no_delivery, so scheduled runs write to the database without posting to Slack.

Configure Slack access

Create a Slack user token for ETL reads and store it as SLACK_ETL_TOKEN in the same secret source used by tools. The Slack tool declares it as an optional HTTP secret for slack.com; iron-proxy injects the real value when the tool calls Slack.

The token must be able to call:

Slack ETL currently syncs public channels visible to the configured ETL user token. It does not sync private channels, DMs, or Slackbot-only live thread events.

Enable the schedules

Set SLACK_ETL_ENABLED=true on the API service. The other schedules default on once Slack ETL is enabled, but can be tuned independently.

Example exclusion list:

SLACK_ETL_EXCLUDED_CHANNEL_PATTERNS="#eng-*-alerts,*-monitor-*"

Data model

Slack ETL writes normalized Slack data into dedicated tables:

The first incremental run reads a small recent window so useful data appears quickly, then seeds historical backfill jobs for the configured lookback. Later incremental runs resume from each channel checkpoint and re-read a trailing thread window so recent edits and replies are picked up.

The lookback values are read windows, not retention windows. Lowering SLACK_SYNC_BACKFILL_LOOKBACK_DAYS or SLACK_SYNC_THREAD_LOOKBACK_DAYS limits future backfill and refresh work, but it does not delete Slack rows or company context documents that were already synced.

Run it manually

Use a manual run when enabling the feature or testing a configuration change. From inside the API deployment, localhost bypass avoids needing an external API key:

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST \
  http://localhost:8000/workflows/runs \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_name": "slack_sync",
    "input": {"metadata": {"reason": "manual_check"}},
    "eager_start": true
  }' | jq

Then inspect the run:

RUN_ID=wfr_...
 
kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s \
  "http://localhost:8000/workflows/runs/${RUN_ID}" | jq

To drain pending historical work immediately:

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST \
  http://localhost:8000/workflows/runs \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_name": "slack_backfill",
    "input": {"channel_batch_limit": 10},
    "eager_start": true
  }' | jq

To force document projection after rows have synced:

kubectl exec -n centaur deploy/centaur-centaur-api -- curl -s -X POST \
  http://localhost:8000/workflows/runs \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_name": "company_context_documents",
    "input": {},
    "eager_start": true
  }' | jq

Verify

Check the workflow schedules:

kubectl exec -n centaur deploy/centaur-centaur-api -- \
  psql "$DATABASE_URL" -c \
  "SELECT schedule_id, workflow_name, enabled, interval_seconds, next_run_at
   FROM workflow_schedules
   WHERE schedule_id IN ('slack_sync', 'slack_backfill', 'company_context_documents')
   ORDER BY schedule_id;"

Check sync health:

kubectl exec -n centaur deploy/centaur-centaur-api -- \
  psql "$DATABASE_URL" -c \
  "SELECT channel_id, watermark_ts, last_success_at, last_error
   FROM slack_sync_checkpoints
   ORDER BY updated_at DESC
   LIMIT 20;"

Check backfill pressure:

kubectl exec -n centaur deploy/centaur-centaur-api -- \
  psql "$DATABASE_URL" -c \
  "SELECT job_type, status, count(*), min(updated_at) AS oldest_updated_at
   FROM slack_sync_backfill_jobs
   GROUP BY job_type, status
   ORDER BY job_type, status;"

Check document projection:

kubectl exec -n centaur deploy/centaur-centaur-api -- \
  psql "$DATABASE_URL" -c \
  "SELECT source_type, count(*), max(source_updated_at)
   FROM company_context_documents
   WHERE source = 'slack'
   GROUP BY source_type
   ORDER BY source_type;"

Centaur also exports ETL metrics, including cursor lag, sync freshness, active and failed scopes, backfill job counts and age, item counters, document change counters, and Slack projection lag. Use those alongside slack_sync_runs when setting alerts.

Troubleshoot

Keep the ETL token scoped to the channels and workspace data you actually want agents to retrieve. Synced rows and projected documents are deployment-wide context, so treat the token as a deliberate data boundary.