feat(site_analytics): app for collecting anonymized visit data for community pages

.env.example

@@ -377,6 +377,26 @@ ARCHIVE_RESYNC_TICK_SECONDS=600
 # keeping headroom for the live sync pipeline.
 ARCHIVE_RESYNC_MIN_RATE_REMAINING=2500
 
+# Site analytics (pageview ingestion)
+# Secret salt for visitor monthly hash (sha256-based; required in production).
+SITE_ANALYTICS_HASH_SALT=
+# Comma-separated slugs of sites allowed to post events; unknown slugs → 400.
+SITE_ANALYTICS_ALLOWED_SITES=
+# Raw pageview retention window in days (default 540 ≈ 18 months).
+SITE_ANALYTICS_RETENTION_DAYS=540
+# Reverse proxies in front of this app. X-Forwarded-For is client-controlled, so only
+# this many entries from the right are trusted (Heroku's router appends exactly one).
+# Set to 0 if the app is exposed directly, to ignore X-Forwarded-For entirely.
+SITE_ANALYTICS_TRUSTED_PROXY_COUNT=1
+# Beat period for the daily aggregation task (seconds); 0 disables the schedule.
+SITE_ANALYTICS_DAILY_AGGREGATE_PERIOD_SECONDS=3600
+# Beat period for the monthly aggregation task (seconds); 0 disables the schedule.
+SITE_ANALYTICS_MONTHLY_AGGREGATE_PERIOD_SECONDS=86400
+# Beat period for the retention pruning task (seconds); 0 disables the schedule.
+SITE_ANALYTICS_PRUNE_PERIOD_SECONDS=86400
+# Reject requests with an empty User-Agent (stricter bot hardening; default off).
+SITE_ANALYTICS_REJECT_EMPTY_UA=0
+
 # CI filter (opt-in allowlist)
 # Set SYNCER_CI_FILTER_MODE=allowlist to enable filtering by the allow lists below.
 # Substrings matched case-insensitively against CheckRun.name and StatusContext.context.

AGENTS.md

@@ -2,7 +2,7 @@
 
 ## Project Structure & Module Organization
 - `src/queueboard/` contains the legacy Python data pipeline: GraphQL queries under `queries/`, HTML assets in `static/`, and scripts like `dashboard.py`, `process.py`, and `suggest_reviewer.py`.
-- `qb_site/` hosts the Django codebase; apps live in `qb_site/{core,syncer,analyzer,api,zulip_bot}/` and share settings from `qb_site/qb_site/settings/`.
+- `qb_site/` hosts the Django codebase; apps live in `qb_site/{core,syncer,analyzer,api,zulip_bot,site_analytics}/` and share settings from `qb_site/qb_site/settings/`.
 - `scripts/` provides operational helpers; `test/` stores fixture JSON for dashboard regression checks; `docs/` captures architecture plans/decisions.
 
 ## Build, Test, and Development Commands
@@ -68,7 +68,7 @@ Notes
 ## Keeping AGENTS.md Files Updated
 - Every directory with significant logic has its own `AGENTS.md` (mirrored as `CLAUDE.md`).
   Current locations: root, `qb_site/`, `qb_site/syncer/`, `qb_site/analyzer/`,
-  `qb_site/zulip_bot/`, `qb_site/console/`, `src/queueboard/`.
+  `qb_site/zulip_bot/`, `qb_site/console/`, `qb_site/site_analytics/`, `src/queueboard/`.
 - When you add, rename, or remove management commands, Celery tasks, key services, or
   directory structure, update the relevant `AGENTS.md` in the same commit/PR.
 - When you add a new app or significant sub-directory, create a matching `AGENTS.md`

docs/queueboard_main_workflow.md

@@ -1,6 +1,32 @@
-Here is the main workflow in the `queueboard` repo, which queries data and generates the dashboard using the code in this repo (`queueboard-core`). For the planned v2 ingestion that replaces these ad‑hoc scripts with a database‑backed syncer, see docs/syncer_ingestion_plan.md.
-Note that in the `queueboard` repo, the JSON files in `data/` and `processed-data/` are persisted from run to run by git pushes in this workflow.
-This is also true of a few auxiliary text files: `closed_prs_to_backfill.txt`, `missing_prs.txt`, `redownload.txt`, `stubborn_prs.txt`.
+This document describes the main GitHub Actions workflow used in the sibling
+[`queueboard`](https://github.com/leanprover-community/queueboard) repo.
+The workflow runs every 8 minutes, fetches fresh PR metadata from a deployed
+instance of `qb_site/` (the Django backend in this repo), generates static
+dashboard HTML, and publishes it to GitHub Pages.
+
+## How it works
+
+1. **Checkout** — checks out `queueboard-core` (this repo) to get scripts,
+   GraphQL query templates, and the `queueboard` Python package.
+2. **Fetch + generate** — calls `python -m queueboard.dashboard --api` three
+   times, once per rule set (different queue-classification rules for
+   experimentation). Each run downloads JSON payloads from the backend API and
+   renders a set of HTML dashboard pages into `gh-pages/<rule-set-dir>/`.
+3. **Deploy** — uploads the `gh-pages/` tree as a Pages artifact and deploys
+   it if the run is on the `master` branch and all three generation steps
+   succeeded.
+
+## Required repository secrets
+
+| Secret | Purpose |
+|---|---|
+| `QUEUEBOARD_API_BASE_URL` | Base URL of the deployed `qb_site` instance (e.g. `https://queueboard.example.com`). Used both to fetch API payloads and as the analytics endpoint host. |
+| `QUEUEBOARD_ANALYTICS_SITE` | Site slug registered in `SITE_ANALYTICS_ALLOWED_SITES` on the server (e.g. `queueboard`). When set, a privacy-preserving analytics snippet is injected into every generated page. Omit to disable analytics. |
+
+If `QUEUEBOARD_ANALYTICS_SITE` is absent (secret not configured), the snippet
+is silently omitted and all other workflow behaviour is unchanged.
+
+## Workflow YAML
 
 ```yaml
 name: Update PR metadata
@@ -55,6 +81,7 @@ jobs:
       id: generate-dashboard-api-rs1
       env:
         QUEUEBOARD_API_BASE_URL: ${{ secrets.QUEUEBOARD_API_BASE_URL }}
+        QUEUEBOARD_ANALYTICS_SITE: ${{ secrets.QUEUEBOARD_ANALYTICS_SITE }}
       run: |
         uv run python -m queueboard.dashboard \
           --api \
@@ -66,6 +93,7 @@ jobs:
       id: generate-dashboard-api-rs2
       env:
         QUEUEBOARD_API_BASE_URL: ${{ secrets.QUEUEBOARD_API_BASE_URL }}
+        QUEUEBOARD_ANALYTICS_SITE: ${{ secrets.QUEUEBOARD_ANALYTICS_SITE }}
       run: |
         uv run python -m queueboard.dashboard \
           --api \
@@ -77,6 +105,7 @@ jobs:
       id: generate-dashboard-api-rs3
       env:
         QUEUEBOARD_API_BASE_URL: ${{ secrets.QUEUEBOARD_API_BASE_URL }}
+        QUEUEBOARD_ANALYTICS_SITE: ${{ secrets.QUEUEBOARD_ANALYTICS_SITE }}
       run: |
         uv run python -m queueboard.dashboard \
           --api \

qb_site/AGENTS.md

@@ -9,12 +9,14 @@
   - `api`: DRF views/serializers for queueboard surfaces.
   - `zulip_bot`: Zulip webhook/command integration and policies.
   - `console`: GitHub-OAuth reviewer console for accepting/declining assignment proposals (design doc 050).
+  - `site_analytics`: privacy-preserving pageview ingestion and aggregation for static/funder-facing sites.
 - Keep new modules inside the owning app (`models/`, `services/`, `tasks/`, `management/commands/`, `tests/`).
 - App-specific guidance:
   - `qb_site/api/AGENTS.md` for public API endpoints, common patterns, and authentication notes.
   - `qb_site/syncer/AGENTS.md` for ingestion, discovery/backfill, and sync admin workflows.
   - `qb_site/analyzer/AGENTS.md` for revision/queue/dependency sweeps and analytics models.
   - `qb_site/zulip_bot/AGENTS.md` for webhook/command/policy/registration behavior.
+  - `qb_site/site_analytics/AGENTS.md` for pageview ingestion, aggregation tasks, and privacy rules.
 
 ## Core Commands
 ```bash

qb_site/api/urls.py

@@ -2,13 +2,15 @@
 
 from django.urls import path
 from api.views import index
+from api.views.analytics_collect import AnalyticsCollectView
 from api.views.queueboard_dependency_graph import QueueboardDependencyGraphView
 from api.views.queueboard_snapshot import QueueboardSnapshotView
 from api.views.reviewer_assignment import AreaStatsView, ReviewerAssignmentsView
 from api.views.reviewer_interests import ReviewerInterestsView
 
 urlpatterns: list = [
     path("", index, name="index"),
+    path("v1/analytics/collect", AnalyticsCollectView.as_view(), name="analytics-collect"),
     path("v1/queueboard/snapshot", QueueboardSnapshotView.as_view(), name="queueboard-snapshot"),
     path(
         "v1/queueboard/dependency-graph",

qb_site/api/views/analytics_collect.py

@@ -0,0 +1,105 @@
+"""POST /api/v1/analytics/collect — lightweight pageview ingestion endpoint."""
+
+from __future__ import annotations
+
+import logging
+
+from django.conf import settings
+from django.utils import timezone
+from rest_framework import status
+from rest_framework.request import Request
+from rest_framework.response import Response
+from rest_framework.views import APIView
+
+from site_analytics.models import AnalyticsPageView
+from site_analytics.services.bot_filter import is_bot
+from site_analytics.services.hashing import SaltUnavailable, compute_visitor_hash, get_client_ip
+
+logger = logging.getLogger(__name__)
+
+# Hard caps to guard against oversized payloads hitting DB column limits.
+_PATH_MAX = 2000
+_REFERRER_MAX = 2000
+_UA_MAX = 1000
+
+# CORS headers added to every response so browsers on third-party/static sites
+# can call this endpoint without a server-side proxy.
+_CORS_HEADERS = {
+    "Access-Control-Allow-Origin": "*",
+    "Access-Control-Allow-Methods": "POST, OPTIONS",
+    "Access-Control-Allow-Headers": "Content-Type",
+    "Access-Control-Max-Age": "86400",
+}
+
+
+def _cors(response: Response) -> Response:
+    for key, value in _CORS_HEADERS.items():
+        response[key] = value
+    return response
+
+
+class AnalyticsCollectView(APIView):
+    """Ingest a single pageview event.
+
+    Intentionally minimal: validate, hash, insert, return 204.
+    All heavier work (aggregation, reporting) happens in periodic tasks.
+
+    CORS headers are always returned so browsers on third-party static sites
+    can call this endpoint directly.
+    """
+
+    authentication_classes: list = []
+    permission_classes: list = []
+
+    def options(self, request: Request, *args: object, **kwargs: object) -> Response:
+        """Handle CORS preflight requests."""
+        return _cors(Response(status=status.HTTP_204_NO_CONTENT))
+
+    def post(self, request: Request, *args: object, **kwargs: object) -> Response:
+        site = (request.data.get("site") or "").strip()
+        path = (request.data.get("path") or "").strip()
+        referrer = (request.data.get("referrer") or "").strip()
+        user_agent = request.META.get("HTTP_USER_AGENT", "").strip()
+
+        if not site:
+            return _cors(Response({"detail": "site is required"}, status=status.HTTP_400_BAD_REQUEST))
+        if not path:
+            return _cors(Response({"detail": "path is required"}, status=status.HTTP_400_BAD_REQUEST))
+
+        allowed_sites = settings.SITE_ANALYTICS_ALLOWED_SITES
+        if site not in allowed_sites:
+            return _cors(Response({"detail": "unknown site"}, status=status.HTTP_400_BAD_REQUEST))
+
+        # Reject empty UA when the stricter hardening flag is enabled.
+        if not user_agent and settings.SITE_ANALYTICS_REJECT_EMPTY_UA:
+            return _cors(Response(status=status.HTTP_204_NO_CONTENT))
+
+        # Silently drop bot traffic rather than returning an error, to avoid
+        # leaking information about detection heuristics.
+        if is_bot(user_agent):
+            return _cors(Response(status=status.HTTP_204_NO_CONTENT))
+
+        now = timezone.now()
+        try:
+            visitor_month_hash = compute_visitor_hash(get_client_ip(request), user_agent)
+        except SaltUnavailable:
+            # Fail closed: dropping the event is strictly better than persisting an
+            # unsalted (reversible) visitor hash. Logged at error level because this
+            # means analytics is silently collecting nothing until a salt exists.
+            logger.error(
+                "site_analytics: dropping pageview for site %r — no hash salt configured. "
+                "Set SITE_ANALYTICS_HASH_SALT or run the site_analytics.rotate_salt task.",
+                site,
+            )
+            return _cors(Response(status=status.HTTP_204_NO_CONTENT))
+
+        AnalyticsPageView.objects.create(
+            site=site,
+            path=path[:_PATH_MAX],
+            referrer=referrer[:_REFERRER_MAX],
+            user_agent=user_agent[:_UA_MAX],
+            occurred_at=now,
+            visitor_month_hash=visitor_month_hash,
+        )
+
+        return _cors(Response(status=status.HTTP_204_NO_CONTENT))

qb_site/qb_site/settings/base.py

@@ -61,6 +61,7 @@ def env_optional_bounded_int(name: str, *, minimum: int, maximum: int) -> int |
     "api",
     "zulip_bot",
     "console",
+    "site_analytics",
 ]
 
 MIDDLEWARE = [
@@ -520,6 +521,24 @@ def env_optional_bounded_int(name: str, *, minimum: int, maximum: int) -> int |
 ARCHIVE_RESYNC_TICK_SECONDS = int(os.getenv("ARCHIVE_RESYNC_TICK_SECONDS", 600))
 ARCHIVE_RESYNC_MIN_RATE_REMAINING = int(os.getenv("ARCHIVE_RESYNC_MIN_RATE_REMAINING", 2500))
 
+# Site analytics settings
+# Fallback salt used until the first rotate_salt task runs and writes a DB salt.
+# Required in production on first deploy; thereafter the DB salt takes precedence.
+SITE_ANALYTICS_HASH_SALT = os.getenv("SITE_ANALYTICS_HASH_SALT", "")
+SITE_ANALYTICS_ALLOWED_SITES: list[str] = [
+    s.strip() for s in os.getenv("SITE_ANALYTICS_ALLOWED_SITES", "").split(",") if s.strip()
+]
+SITE_ANALYTICS_RETENTION_DAYS = int(os.getenv("SITE_ANALYTICS_RETENTION_DAYS", 540))
+# Number of reverse proxies in front of this app. X-Forwarded-For is client-controlled,
+# so only this many entries from the right of the chain are trustworthy (Heroku's router
+# appends one). Set to 0 when the app is exposed directly, to ignore the header entirely.
+SITE_ANALYTICS_TRUSTED_PROXY_COUNT = int(os.getenv("SITE_ANALYTICS_TRUSTED_PROXY_COUNT", 1))
+SITE_ANALYTICS_DAILY_AGGREGATE_PERIOD_SECONDS = int(os.getenv("SITE_ANALYTICS_DAILY_AGGREGATE_PERIOD_SECONDS", 3600))
+SITE_ANALYTICS_MONTHLY_AGGREGATE_PERIOD_SECONDS = int(os.getenv("SITE_ANALYTICS_MONTHLY_AGGREGATE_PERIOD_SECONDS", 86400))
+SITE_ANALYTICS_PRUNE_PERIOD_SECONDS = int(os.getenv("SITE_ANALYTICS_PRUNE_PERIOD_SECONDS", 86400))
+# Reject requests with an empty User-Agent header (stricter bot hardening).
+SITE_ANALYTICS_REJECT_EMPTY_UA = env_bool(os.getenv("SITE_ANALYTICS_REJECT_EMPTY_UA"), False)
+
 # CI filter (opt-in allowlist mode)
 # Set mode to 'allowlist' to enable filtering by the following substrings; otherwise all contexts are ingested.
 SYNCER_CI_FILTER_MODE = os.getenv("SYNCER_CI_FILTER_MODE", "all").lower()
@@ -755,3 +774,24 @@ def env_optional_bounded_int(name: str, *, minimum: int, maximum: int) -> int |
             "fanout": True,
         },
     }
+if SITE_ANALYTICS_DAILY_AGGREGATE_PERIOD_SECONDS > 0:
+    CELERY_BEAT_SCHEDULE["site_analytics_aggregate_daily"] = {
+        "task": "site_analytics.aggregate_daily_metrics",
+        "schedule": SITE_ANALYTICS_DAILY_AGGREGATE_PERIOD_SECONDS,
+    }
+if SITE_ANALYTICS_MONTHLY_AGGREGATE_PERIOD_SECONDS > 0:
+    CELERY_BEAT_SCHEDULE["site_analytics_aggregate_monthly"] = {
+        "task": "site_analytics.aggregate_monthly_metrics",
+        "schedule": SITE_ANALYTICS_MONTHLY_AGGREGATE_PERIOD_SECONDS,
+    }
+if SITE_ANALYTICS_PRUNE_PERIOD_SECONDS > 0:
+    CELERY_BEAT_SCHEDULE["site_analytics_prune_pageviews"] = {
+        "task": "site_analytics.prune_old_pageviews",
+        "schedule": SITE_ANALYTICS_PRUNE_PERIOD_SECONDS,
+        "kwargs": {"retention_days": SITE_ANALYTICS_RETENTION_DAYS},
+    }
+# Rotate the visitor-hash salt at midnight UTC on the 1st of each month.
+CELERY_BEAT_SCHEDULE["site_analytics_rotate_salt"] = {
+    "task": "site_analytics.rotate_salt",
+    "schedule": crontab(minute=0, hour=0, day_of_month=1),
+}

qb_site/site_analytics/AGENTS.md

@@ -0,0 +1,54 @@
+# Site Analytics Guidelines
+
+## Scope
+- `qb_site/site_analytics/` implements privacy-preserving pageview ingestion and aggregation for static/funder-facing sites.
+- Raw events in `AnalyticsPageView`; aggregate reporting in `AnalyticsDailyMetric` and `AnalyticsMonthlyMetric` (added in A3/A4).
+- Design record: `docs/design-decisions/031-analytics-ingestion-design.md`.
+
+## Module Layout
+- `models/pageview.py` — `AnalyticsPageView` raw event rows (immutable after insert).
+- `models/daily_metric.py` — `AnalyticsDailyMetric` (added in A3).
+- `models/monthly_metric.py` — `AnalyticsMonthlyMetric` (added in A4).
+- `models/salt.py` — `SiteAnalyticsSalt` single-row table holding the current month's hash salt.
+- `services/` — hashing, bot filtering, aggregation logic.
+- `checks.py` — Django system checks (registered in `apps.py:ready()`).
+- `tasks/` — periodic Celery tasks for aggregation, pruning, and salt rotation.
+- `tests/` — unit and integration tests.
+- API ingestion view: `qb_site/api/views/analytics_collect.py` (added in A2).
+
+## Key Settings (all env-overridable)
+- `SITE_ANALYTICS_HASH_SALT` — fallback salt used until the first `rotate_salt` task runs and writes a DB salt. Required on first deploy; thereafter the `SiteAnalyticsSalt` DB row takes precedence.
+- `SITE_ANALYTICS_ALLOWED_SITES` — comma-separated site slugs; unknown slugs rejected with `400`.
+- `SITE_ANALYTICS_RETENTION_DAYS` — raw pageview retention window (default 540 days / ~18 months).
+- `SITE_ANALYTICS_TRUSTED_PROXY_COUNT` — reverse-proxy hops in front of the app (default 1, matching Heroku's router). Controls how many `X-Forwarded-For` entries from the right are trusted; 0 ignores the header entirely.
+- `SITE_ANALYTICS_DAILY_AGGREGATE_PERIOD_SECONDS` — beat period for daily aggregation task (default 3600).
+- `SITE_ANALYTICS_MONTHLY_AGGREGATE_PERIOD_SECONDS` — beat period for monthly aggregation task (default 86400).
+- `SITE_ANALYTICS_PRUNE_PERIOD_SECONDS` — beat period for retention pruning task (default 86400).
+
+## Task Surface
+Celery task names (as registered via `@shared_task(name=…)`):
+
+- `site_analytics.aggregate_daily_metrics` — idempotent upsert of daily pageview/unique-visitor counts (added in A3).
+- `site_analytics.aggregate_monthly_metrics` — idempotent upsert of monthly metrics; recomputes current + previous month (added in A4).
+- `site_analytics.prune_old_pageviews` — deletes raw rows older than `SITE_ANALYTICS_RETENTION_DAYS` (added in A4).
+- `site_analytics.rotate_salt` — generates a new random visitor-hash salt and discards the previous one; runs at midnight UTC on the 1st of each month.
+
+## Privacy Invariants
+- Raw IP addresses are never stored.
+- **Ingestion fails closed without a salt.** `compute_visitor_hash` raises `SaltUnavailable` when neither a `SiteAnalyticsSalt` row nor `SITE_ANALYTICS_HASH_SALT` is set, and the collect view drops the event (204 + error log) rather than persist an unsalted hash — `sha256(ip | ua)` with no secret is brute-forceable over the IPv4 space, so it would be a recoverable identifier, not a pseudonymous one. Collecting nothing is the correct failure mode.
+- A deploy-time system check (`site_analytics.E001`, in `checks.py`) fails `manage.py check`/`migrate` when `SITE_ANALYTICS_ALLOWED_SITES` is non-empty but no salt is set. It is gated on allowed-sites because analytics is opt-in, and reads settings only — never the DB, since `migrate` runs checks before `SiteAnalyticsSalt` exists. Gunicorn does not run system checks on boot, so the runtime guarantee is `SaltUnavailable`, not this check.
+- `visitor_month_hash = sha256(ip | normalized_user_agent | salt)` where `salt` is the current month's randomly generated value from `SiteAnalyticsSalt`.
+- The salt is replaced at month start and the old value deleted, so hashes from different months are unlinkable even with knowledge of the current salt (forward secrecy).
+- IP is extracted from `X-Forwarded-For` taking `SITE_ANALYTICS_TRUSTED_PROXY_COUNT` entries from the **right** (proxies append; the leftmost entries are client-supplied and spoofable), falling back to `REMOTE_ADDR`. Set the count to 0 when the app is exposed directly.
+- Changing hashing semantics requires an explicit migration/versioning note in the design doc.
+
+## Backup Policy
+- `site_analytics_analyticspageview` → TRUNCATE (raw rows contain visitor hashes; excluded from public backup).
+- `site_analytics_analyticsdailymetric`, `site_analytics_analyticsmonthlymetric` → RETAIN (aggregate-only, safe to share).
+- Update `scripts/backup_policy.py` whenever adding or removing tables.
+
+## Testing
+```bash
+uv run python qb_site/manage.py test site_analytics
+bash scripts/repo_check_compose.sh
+```

qb_site/site_analytics/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md