Search quality, source prioritization, OutSystems connector, dark mode & indexing perf

.gitignore

@@ -28,7 +28,6 @@ requestdata.json
 # Playwright MCP session artifacts (console logs, page snapshots, ad-hoc
 # screenshots) written during local UI debugging. Not source.
 .playwright-mcp/
-model-picker-open.png
 
 # Live cluster dumps from `kubectl get -o yaml > …`. NEVER commit:
 # Darwin's ConfigMap currently contains real secrets in plaintext (Slack
@@ -39,3 +38,9 @@ model-picker-open.png
 darwin-kubernetes/temp/
 k8s/overlays/*/secrets.env
 k8s/overlays/*/*.secrets.env
+# Velero Azure SP credentials (source for the cloud-credentials secret)
+k8s/overlays/prod-velero/credentials-velero
+# Velero notifier Slack bot token (source for the slack-notify secret)
+k8s/overlays/prod-velero/slack-notify.env
+# Ad-hoc export of web connector URLs (local only)
+web-connectors.csv

AGENTS.md

@@ -432,6 +432,78 @@ liveness probes by design** (an aggressive one kills slow-but-healthy
 nodes); readiness probes on the Service-backed nodes
 (configserver/query/feed) gate traffic during the slow bootstrap.
 
+### 11. Slow indexing is the per-doc Vespa existence VISIT, not the crawl — and the content-hash dedup already exists
+
+When a connector (especially a big web one like docs.uipath) indexes slowly,
+the bottleneck is almost never the source fetch. For every document,
+`VespaIndex.index()` → `_clear_and_index_vespa_chunks()` →
+`_get_vespa_chunks_by_document_id` (`document_index/vespa/index.py`) hits Vespa's
+**Visit API** (`GET /document/v1/.../docid?selection=document_id=='<id>'&wantedDocumentCount=1000`)
+to find existing chunks before re-writing. That `selection` is a **corpus scan**,
+~**10–11 s per document** on the large prod index — and Vespa content nodes sit
+near-idle while it happens (it's scan/IO-bound, so scaling content nodes does
+NOT help). It's in the shared indexing path, so it slows every connector; large
+multi-doc connectors just make it obvious. The real fix is a keyed lookup
+(`document_id` as a `fast-search` attribute, or a point GET / search query)
+instead of the visit.
+
+**Do NOT "add" a Postgres content-hash dedup to skip this — it already exists.**
+`Document.indexed_content_hash` (`db/models.py`) +
+`get_doc_ids_to_update` (`indexing/indexing_pipeline.py`) skip a doc (no re-embed,
+no Vespa write) when the stored hash equals `doc.get_content_hash()`. The hash is
+written only AFTER a confirmed Vespa write. Why it can still re-index everything:
+
+- It's bypassed when `ignore_time_skip=True`, set on `from_beginning` full runs
+  (`background/indexing/run_indexing.py`).
+- Docs indexed before the hash feature have `indexed_content_hash = NULL`, so the
+  hash check can't fire and it falls back to a `doc_updated_at` timestamp compare.
+- The **web connector never sets `doc_updated_at`**, so that fallback can't skip
+  hash-less web docs either → they re-index every run (each paying the ~11 s
+  visit) UNTIL the run completes and backfills their hash. It is self-healing —
+  once hashes exist, later polls skip unchanged docs and run fast — but a full
+  run that times out before backfilling will keep re-doing the slow work.
+
+(Diagnosed 2026-06 on the docs.uipath automation-suite latest-N connector:
+~2889 of ~3161 docs had NULL hashes.)
+
+---
+
+### 12. NEVER build the web image locally on Apple Silicon
+
+The web image's `next build` step **SIGSEGVs** when built for `linux/amd64`
+under emulation on an arm64 Mac (Next.js build worker dies with `signal:
+SIGSEGV`). Building amd64 under emulation is the only way to produce a
+deployable image locally on Apple Silicon, so there is no working local web
+build there — don't try, and don't burn time "fixing" it. It is not a config /
+dependency / disk-space problem.
+
+Instead, build web on **darwinacr** (native-amd64 ACR build agents) and import
+the result into the prod registry. `k8s/scripts/build-deploy.sh` does this
+**automatically** on Apple Silicon — `build-deploy.sh deploy web` detects the
+host and routes web to `az acr build` + `az acr import`, no flags needed. The
+backend image has no native build step and still builds locally under emulation.
+
+If you ever need the raw commands (script unavailable / debugging):
+
+```bash
+# 1. build on darwinacr (native amd64)
+az acr build --registry darwinacr \
+  --image danswer/danswer-web-server:vha-N \
+  --build-arg NODE_BASE=darwinacr.azurecr.io/library/node:20-alpine \
+  --file web/Dockerfile ./web
+# 2. transfer darwinacr -> prod registry (different subscriptions, so blob-copy
+#    via pull/retag/push, NOT `az acr import`). Pure copy on the Mac, no SIGSEGV.
+az acr login --name darwinacr
+docker pull --platform linux/amd64 darwinacr.azurecr.io/danswer/danswer-web-server:vha-N
+docker tag  darwinacr.azurecr.io/danswer/danswer-web-server:vha-N \
+            sfbrdevhelmweacr.azurecr.io/danswer/danswer-web-server:vha-N
+docker push sfbrdevhelmweacr.azurecr.io/danswer/danswer-web-server:vha-N
+```
+
+(`--file` is relative to the CWD, not the `./web` context — `web/Dockerfile`,
+not `Dockerfile`. `az acr build` on darwinacr needs **PIM Contributor**; the
+prod push uses the `~/.zshrc` ACR_USERNAME/ACR_PASSWORD admin creds.)
+
 ---
 
 ## Common workflows

CONTRIBUTING.md

@@ -112,21 +112,21 @@ playwright install
 #### Dependent Docker Containers
 First navigate to `danswer/deployment/docker_compose`, then start Postgres.
 
-The simplest path is the compose-managed pair (uses a named docker volume for
-Vespa's data; data lives until you `docker volume rm`):
+Start Postgres and Redis via compose under the `-p danswer-stack` project, so
+they share the `danswer-stack_default` network (Vespa is run separately, below,
+on the same network):
 
 ```bash
-docker compose -f docker-compose.dev.yml -p danswer-stack up -d relational_db
+docker compose -f docker-compose.dev.yml -p danswer-stack up -d relational_db redis
 ```
 
-If you'd rather pin Vespa's data + logs to host-mounted directories so you
-can inspect them outside Docker (and survive `docker compose down -v`),
-start Postgres via compose and Vespa via a manual `docker run` on the same
-network. Pick any host paths you like:
+Run Vespa via a manual `docker run` on that same `danswer-stack_default`
+network, with host-mounted data + logs dirs so you can inspect them outside
+Docker (and they survive `docker compose down -v`). Use this rather than the
+compose `index` service (it's unreliable locally); the `--network` flag is what
+keeps the manually-run Vespa on the shared network. Pick any host paths:
 
 ```bash
-docker compose -f docker-compose.dev.yml -p danswer-stack up -d relational_db
-
 export VESPA_VAR_STORAGE="${HOME}/danswer-vespa-data/var"
 export VESPA_LOG_STORAGE="${HOME}/danswer-vespa-data/logs"
 mkdir -p "$VESPA_VAR_STORAGE" "$VESPA_LOG_STORAGE"
@@ -142,14 +142,27 @@ docker run \
   --publish 19071:19071 \
   vespaengine/vespa:8.277.17
 
-# Sanity check: both containers should be on the danswer-stack_default network
+# Sanity check: all containers (Postgres, Redis, Vespa) on danswer-stack_default
 docker ps --format '{{ .ID }} {{ .Names }} {{ json .Networks }}'
 ```
 
 (index refers to Vespa and relational_db refers to Postgres. The hostname
 `index` matters — Danswer reaches Vespa by that DNS name on the shared
 network.)
 
+Redis (caching + per-user rate limiting) comes up with the commands above as
+part of the `danswer-stack` project, so it's already on the shared
+`danswer-stack_default` network. To check it or manage it on its own:
+
+```bash
+docker compose -f docker-compose.dev.yml -p danswer-stack exec redis redis-cli ping   # -> PONG
+docker compose -f docker-compose.dev.yml -p danswer-stack stop redis
+```
+
+The container runs with no auth and publishes `6379` to the host, so a
+host-run backend connects with `REDIS_HOST=localhost`, `REDIS_PORT=6379`,
+`REDIS_PASSWORD=` (empty). (In-compose, the service name is `redis`.)
+
 #### Running Danswer
 To start the frontend, navigate to `danswer/web` and run:
 ```bash
@@ -325,7 +338,18 @@ export MODEL_SERVER_HOST=localhost
 export MODEL_SERVER_PORT=9000
 export INDEXING_MODEL_SERVER_HOST=localhost
 export INDEXING_MODEL_SERVER_PORT=9000
-export REDIS_HOST=cache             # matches the compose service name
+export REDIS_HOST=localhost         # backend runs on the host; reach Redis via the published 6379 port
+
+# Cross-encoder reranking, available locally. The model server (`dmo`) loads the
+# reranker IN-PROCESS (sentence-transformers, CPU) — no extra container. Uses the
+# small default model (mxbai-rerank-xsmall-v1); set RERANK_MODEL_NAME to try a
+# bigger one. Reranking still only runs for assistants / chats that opt in.
+# (Prod serves the reranker via a TEI container instead — see k8s/optional/tei-rerank.)
+export RERANK_ENABLED=true
+export LLM_RELEVANCE_FILTER_ENABLED=true   # LLM relevance filter; independent of rerank
+# Advanced: to mirror prod and offload the reranker to a local TEI container
+# instead of in-process, run TEI yourself and set:
+# export RERANK_SERVER_URL=http://localhost:8086
 
 # ---------------------------------------------------------------------------
 # LLM (Generative AI) — UiPath LLM Gateway via OAuth client credentials

backend/alembic/versions/a8b9c0d1e2f3_persona_display_name.py

@@ -0,0 +1,35 @@
+"""persona: add display_name (user-friendly chat label)
+
+Adds persona.display_name — an optional, admin-editable label shown in the chat
+UI. The immutable `name` stays the identifier; `display_name` is presentational
+only and the chat falls back to `name` when it's blank. Backfills existing rows
+with their `name` so nothing changes visually until an admin edits it. See
+db/models.py::Persona.
+
+Revision ID: a8b9c0d1e2f3
+Revises: f7a8b9c0d1e2
+Create Date: 2026-06-19
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = "a8b9c0d1e2f3"
+down_revision = "f7a8b9c0d1e2"
+branch_labels: None = None
+depends_on: None = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "persona",
+        sa.Column("display_name", sa.String(), nullable=True),
+    )
+    # Backfill: existing assistants keep showing their current name.
+    op.execute("UPDATE persona SET display_name = name WHERE display_name IS NULL")
+
+
+def downgrade() -> None:
+    op.drop_column("persona", "display_name")

backend/alembic/versions/b9c0d1e2f3a4_user_hidden_assistants.py

@@ -0,0 +1,46 @@
+"""user: add hidden_assistants (opt-out assistant visibility)
+
+Adds user.hidden_assistants — the list of assistant (persona) ids a user has
+explicitly hidden from their chat picker. This flips assistant visibility from
+opt-IN (only assistants in `chosen_assistants` were shown) to opt-OUT: every
+accessible assistant is visible by default, so a newly created admin assistant
+appears for all users automatically; a user hides the ones they don't want.
+
+`chosen_assistants` now controls ORDER/default only, not visibility.
+
+No backfill: the chat experience hasn't been rolled out to end users yet, so
+there is no curated state to preserve — every existing user simply starts with
+an empty hidden list (= sees everything), which is the desired behavior. See
+db/models.py::User.
+
+Revision ID: b9c0d1e2f3a4
+Revises: a8b9c0d1e2f3
+Create Date: 2026-06-21
+
+"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+
+# revision identifiers, used by Alembic.
+revision = "b9c0d1e2f3a4"
+down_revision = "a8b9c0d1e2f3"
+branch_labels: None = None
+depends_on: None = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "user",
+        sa.Column(
+            "hidden_assistants",
+            postgresql.ARRAY(sa.Integer()),
+            nullable=False,
+            server_default="{}",
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("user", "hidden_assistants")

backend/alembic/versions/f6a7b8c9d0e1_persona_rerank_enabled.py

@@ -0,0 +1,39 @@
+"""persona: add rerank_enabled (per-assistant cross-encoder reranking opt-in)
+
+Per-assistant toggle for cross-encoder reranking. Only takes effect when
+reranking is globally available (RERANK_ENABLED + a GPU-backed model server);
+default false so existing assistants and the GPU-free local/default setup are
+unchanged. Lets reranking be rolled out incrementally / A-B compared per
+assistant before becoming the default. See db/models.py::Persona and
+search/preprocessing/preprocessing.py.
+
+Revision ID: f6a7b8c9d0e1
+Revises: e5f6a7b8c9d0
+Create Date: 2026-06-03
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = "f6a7b8c9d0e1"
+down_revision = "e5f6a7b8c9d0"
+branch_labels: None = None
+depends_on: None = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "persona",
+        sa.Column(
+            "rerank_enabled",
+            sa.Boolean(),
+            nullable=False,
+            server_default=sa.false(),
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("persona", "rerank_enabled")

backend/alembic/versions/f7a8b9c0d1e2_slack_bot_response_blocklist.py

@@ -0,0 +1,62 @@
+"""slack bot: response blocklist (suppress responses for certain senders)
+
+Creates slack_bot_response_blocklist — senders (by email) whose Slack messages
+should NOT trigger a Darwin response. DB-driven so the list can change without a
+redeploy. Seeds the first entry (jr.bancel@uipath.com). See
+db/models.py::SlackBotResponseBlocklist and
+danswerbot/slack/handlers/handle_message.py.
+
+Revision ID: f7a8b9c0d1e2
+Revises: f6a7b8c9d0e1
+Create Date: 2026-06-17
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = "f7a8b9c0d1e2"
+down_revision = "f6a7b8c9d0e1"
+branch_labels: None = None
+depends_on: None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "slack_bot_response_blocklist",
+        sa.Column("id", sa.Integer(), nullable=False),
+        sa.Column("email", sa.String(), nullable=False),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.func.now(),
+            nullable=False,
+        ),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    # Single unique index — mirrors `mapped_column(String, unique=True, index=True)`.
+    op.create_index(
+        op.f("ix_slack_bot_response_blocklist_email"),
+        "slack_bot_response_blocklist",
+        ["email"],
+        unique=True,
+    )
+
+    # Seed the initial blocked senders (stored lowercase; matched
+    # case-insensitively). Further additions are plain DB inserts — no migration.
+    op.execute(
+        sa.text(
+            "INSERT INTO slack_bot_response_blocklist (email) VALUES "
+            "('jr.bancel@uipath.com'), ('andrei.barbu@uipath.com') "
+            "ON CONFLICT (email) DO NOTHING"
+        )
+    )
+
+
+def downgrade() -> None:
+    op.drop_index(
+        op.f("ix_slack_bot_response_blocklist_email"),
+        table_name="slack_bot_response_blocklist",
+    )
+    op.drop_table("slack_bot_response_blocklist")

backend/danswer/auth/api_key.py

@@ -40,3 +40,35 @@ def validate_api_key(request: Request, db_session: Session = Depends(get_session
     # Cache it for future requests
     cache[api_key_value] = True
     return None
+
+
+def request_has_valid_api_key(request: Request, db_session: Session) -> bool:
+    """Return True if the request carries a valid X-API-Key.
+
+    These keys are service credentials for automation (they intentionally do NOT
+    map to a browser `User`). `current_user` uses this to authorize an api-key
+    request as an anonymous service caller instead of 403'ing it into the SSO
+    flow once AUTH_TYPE enforces auth (e.g. OIDC). Mirrors `validate_api_key`'s
+    lookup + cache exactly, so the two stay consistent.
+
+    NOTE: `db_session` is passed in (not a Depends) because the caller already
+    holds a session.
+    """
+    if _API_KEY_HEADER not in request.headers:
+        return False
+
+    api_key_value = request.headers.get(_API_KEY_HEADER)
+    if not api_key_value:
+        return False
+
+    if api_key_value in cache:
+        return True
+
+    api_key = db_session.scalar(
+        select(ApiKey).where(ApiKey.hashed_api_key == api_key_value)
+    )
+    if api_key is None:
+        return False
+
+    cache[api_key_value] = True
+    return True