Co pi podcast

.env.example

@@ -20,24 +20,45 @@ BASE_URL=http://localhost:8000
 # Admin (set to true to allow impersonation in development)
 ALLOW_HTTP_SESSIONS=true
 
+# Slack — app configuration tokens (used by scripts/provision_slack_bots.py)
+# One-time setup: https://api.slack.com/apps → "Your App Configuration Tokens" → Generate Token
+# After generation paste both values here; the provisioning script rotates them automatically.
+SLACK_CONFIG_TOKEN=xoxe-your-app-config-token
+SLACK_CONFIG_REFRESH_TOKEN=xoxe-your-app-config-refresh-token
+
 # Slack — one pair per agent (Bot User OAuth Token + App-Level Token)
+# Add as many agents as needed using this pattern; no code changes required.
+# Run scripts/provision_slack_bots.py to create and install all missing bots automatically.
+#   SLACK_BOT_TOKEN_<AGENT_ID>=xoxb-...   (required)
+#   SLACK_APP_TOKEN_<AGENT_ID>=xapp-...   (optional)
 SLACK_BOT_TOKEN_SU=xoxb-placeholder
-SLACK_APP_TOKEN_SU=xapp-placeholder
 SLACK_BOT_TOKEN_WISEMAN=xoxb-placeholder
-SLACK_APP_TOKEN_WISEMAN=xapp-placeholder
-SLACK_BOT_TOKEN_LOTZ=xoxb-placeholder
-SLACK_APP_TOKEN_LOTZ=xapp-placeholder
-SLACK_BOT_TOKEN_CRAVATT=xoxb-placeholder
-SLACK_APP_TOKEN_CRAVATT=xapp-placeholder
-SLACK_BOT_TOKEN_GROTJAHN=xoxb-placeholder
-SLACK_APP_TOKEN_GROTJAHN=xapp-placeholder
-SLACK_BOT_TOKEN_PETRASCHECK=xoxb-placeholder
-SLACK_APP_TOKEN_PETRASCHECK=xapp-placeholder
-SLACK_BOT_TOKEN_KEN=xoxb-placeholder
-SLACK_APP_TOKEN_KEN=xapp-placeholder
-SLACK_BOT_TOKEN_RACKI=xoxb-placeholder
-SLACK_APP_TOKEN_RACKI=xapp-placeholder
-SLACK_BOT_TOKEN_SAEZ=xoxb-placeholder
-SLACK_APP_TOKEN_SAEZ=xapp-placeholder
-SLACK_BOT_TOKEN_WU=xoxb-placeholder
-SLACK_APP_TOKEN_WU=xapp-placeholder
+SLACK_BOT_TOKEN_GRANTBOT=xoxb-placeholder
+
+# Podcast TTS backend: "mistral" (default), "openai", or "local" (vLLM-Omni server)
+PODCAST_TTS_BACKEND="mistral"
+
+# Mistral AI TTS (used when PODCAST_TTS_BACKEND=mistral)
+MISTRAL_API_KEY=your-mistral-api-key
+MISTRAL_TTS_MODEL=voxtral-mini-tts-latest
+MISTRAL_TTS_DEFAULT_VOICE=your-voice-uuid
+
+# OpenAI TTS (used when PODCAST_TTS_BACKEND=openai)
+# Voices: alloy echo fable onyx nova shimmer
+# Models: tts-1  tts-1-hd  gpt-4o-mini-tts
+OPENAI_API_KEY=your-openai-api-key
+OPENAI_TTS_MODEL=tts-1
+OPENAI_TTS_DEFAULT_VOICE=alloy
+
+# Local vLLM-Omni TTS server (used when PODCAST_TTS_BACKEND=local)
+# Start with: vllm serve <model> --port 8010
+LOCAL_TTS_HOST=127.0.0.1
+LOCAL_TTS_PORT=8008
+LOCAL_TTS_MODEL=mistralai/Voxtral-4B-TTS-2603
+LOCAL_TTS_VOICE=default
+
+# Podcast
+PODCAST_BASE_URL=http://localhost:8001
+PODCAST_SEARCH_WINDOW_DAYS=14
+PODCAST_MAX_CANDIDATES=50
+# PODCAST_NORMALIZE_AUDIO=true  # uncomment to enable ffmpeg loudnorm post-processing (EBU R128, -16 LUFS)

.gitignore

@@ -53,3 +53,9 @@ certbot/
 .pytest_cache/
 .coverage
 htmlcov/
+
+# Runtime data (state files, generated audio — ephemeral)
+data/
+
+# Test output artifacts
+.labbot-tests/

AGENT.md

@@ -32,6 +32,7 @@ All specs are in `/specs/`:
 - `profile-ingestion.md` — 9-step pipeline, ORCID → PubMed → PMC → LLM
 - `admin-dashboard.md` — read-only, server-rendered, impersonation
 - `agent-system.md` — Slack Bolt, Socket Mode, two-phase LLM calls, simulation engine
+- `labbot-podcast.md` — daily personalized research briefing: PubMed search, LLM selection/summarization, Local or API TTS, Slack DM delivery, per-PI RSS podcast feed
 
 ## Tech Stack
 

CLAUDE.md

@@ -1,5 +1,29 @@
 # CLAUDE.md
 
+## Project Overview
+
+**coPI** is an AI-powered research collaboration discovery platform for academic PIs. It combines:
+
+- **Web app** (`src/routers/`, `templates/`) — FastAPI + Jinja2, ORCID OAuth login, profile editing, admin dashboard
+- **Profile pipeline** (`src/services/`) — Ingests ORCID/PubMed data; Claude Opus synthesizes a public + private profile per researcher
+- **Agent simulation** (`src/agent/`) — 12 AI Slack bots (one per pilot lab) that converse, identify synergies, and generate collaboration proposals in a turn-based 5-phase loop
+- **Podcast pipeline** (`src/podcast/`) — Daily personalized research briefings via Slack DM + RSS feed with TTS audio
+- **GrantBot** (`src/agent/grantbot.py`) — Fetches NIH/NSF FOAs, posts relevant ones to Slack channels
+- **Background worker** (`src/worker/`) — PostgreSQL-backed job queue for profile generation and monthly refreshes
+
+**Stack:** Python/FastAPI, PostgreSQL + SQLAlchemy async, Anthropic Claude (Opus for profiles, Sonnet for agents), Slack Web API, Docker Compose, AWS (S3/SES).
+
+**Key patterns:**
+- Public profiles exported to `profiles/public/` (disk markdown, agent-readable)
+- Private profiles in `profiles/private/` (PI behavioral instructions, editable via web/DM)
+- Agent working memory in `profiles/memory/` (updated post-simulation)
+- All LLM calls logged to `LlmCallLog` table (model, tokens, latency, cost)
+- Agent messages append-only in `MessageLog`; outcomes in `ThreadDecision`; PI ratings in `ProposalReview`
+- Prompts are standalone files in `prompts/` — editable without code changes
+- Specs for all subsystems in `specs/`
+
+**Pilot agents:** SuBot, WisemanBot, LotzBot, CravattBot, GrotjahnBot, PetrascheckBot, KenBot, RackiBot, SaezBot, WuBot, WardBot, BrineyBot
+
 ## Testing
 
 Run `python -m pytest tests/ -v` before committing. All tests must pass.
@@ -42,3 +66,50 @@ docker compose --profile agent run -d --name agent-run agent python -m src.agent
 ```
 
 **Note:** The agent-run container uses mounted source code but the Python process only loads modules at startup. Code changes require a container restart to take effect. **After any code change that affects the running agent process, flag this to the user so they can decide whether to restart.**
+
+## Podcast Pipeline
+
+The LabBot Podcast pipeline (specs/labbot-podcast.md) runs daily at 9am UTC for each active agent:
+
+1. Build PubMed queries from lab's public profile
+2. Fetch candidates from PubMed + bioRxiv + medRxiv + arXiv (last 14 days, up to 50+10 candidates)
+3. Claude Sonnet selects most relevant paper (applying PI's podcast preferences from their private ProfileRevision)
+4. Claude Opus writes a ~250-word structured brief
+5. TTS audio generated (Mistral or local vLLM-Omni); ffmpeg loudnorm applied if PODCAST_NORMALIZE_AUDIO=true
+6. Slack DM sent to PI with text summary + RSS link
+7. RSS feed available at `/podcast/{agent_id}/feed.xml`
+8. Audio served at `/podcast/{agent_id}/audio/{date}.mp3`
+
+Preprint IDs use prefixed format: `biorxiv:...`, `medrxiv:...`, `arxiv:...`. The `paper_url` in summaries links to the correct server (not always PubMed).
+
+```bash
+# Run podcast pipeline once for all active agents
+docker compose --profile podcast run --rm podcast python -m src.podcast.main
+
+# Test pipeline for 'su' agent only
+docker compose exec app python scripts/test_podcast_su.py
+```
+
+## Database Migration Caveat
+
+If the DB was initialized from the `main` branch schema and then this branch is checked out, `alembic upgrade head` will stamp the version without re-running migrations that share a revision ID with ones already applied on `main`. Any columns added by branch-specific migrations may be silently missing.
+
+**Symptom:** `UndefinedColumnError` at runtime despite `alembic current` showing `head`.
+
+**Fix:** Check for missing columns and apply them manually:
+```bash
+docker compose exec app python -c "
+import asyncio
+from src.database import get_engine
+from sqlalchemy import text
+
+async def check():
+    eng = get_engine()
+    async with eng.connect() as conn:
+        result = await conn.execute(text(\"SELECT column_name FROM information_schema.columns WHERE table_name='researcher_profiles' ORDER BY ordinal_position\"))
+        print([r[0] for r in result])
+
+asyncio.run(check())
+"
+```
+Then add any missing columns with `ALTER TABLE ... ADD COLUMN IF NOT EXISTS ...`.

Cohort_approaches.txt

@@ -0,0 +1,102 @@
+Cohort System — Approaches Considered
+======================================
+
+Approach A: Cohort as an Interaction Filter (Minimal)
+------------------------------------------------------
+Add a cohort_memberships table (agent_id, cohort_id). The engine stays structurally
+identical — one global turn loop, shared agent state. Before any interaction is
+permitted (Phase 3 activation, Phase 4 reply, Phase 5 tag), check: do these two
+agents share at least one cohort? If not, the post is invisible to them.
+
+Pros:
+- Tiny diff, backward compatible
+- Agents in multiple cohorts still have unified state
+
+Cons:
+- Turn rules (thread limits, proposal caps, budgets) remain global per agent —
+  cannot be scoped by cohort
+- Original form: no concurrency — cohorts still compete in a single sequential loop
+
+Revision (adopted): The concurrency gap is filled independently of cohorts using a
+global semaphore (N concurrent turns) + min-heap agent selection, keeping the cohort
+system as a pure interaction filter with no role in scheduling. See Chosen Direction.
+
+
+Approach B: Per-Cohort Agent State (Partitioned)
+-------------------------------------------------
+AgentState becomes dict[cohort_id, AgentState]. The main loop iterates cohorts in
+round-robin (or concurrently via asyncio.gather), each cohort running its own turn
+selection over its member agents. Thread limits, proposal counts, and budgets are
+tracked per (agent_id, cohort_id) — an agent in two cohorts has independent budgets
+in each. Interaction gating is automatic: Phase 4/5 only operate within a cohort's
+member set.
+
+Pros:
+- True per-cohort parallelism
+- Rules naturally scoped
+- Clean mental model
+
+Cons:
+- Meaningful refactor — AgentState, budget tracking, blocking logic all need the
+  cohort dimension
+- Agent working memory (profiles/memory/) would need cohort tagging or remain
+  shared across cohorts
+
+
+Approach C: Cohort-Sharded Engine Instances (Full Isolation)
+-------------------------------------------------------------
+Instantiate one SimulationEngine per cohort, each with only its member agents
+loaded. Run them as separate asyncio tasks (or even separate processes). Agents in
+multiple cohorts appear in multiple engines with independent state copies.
+
+Pros:
+- Complete isolation
+- Maximum parallelism
+- No cross-contamination of state
+
+Cons:
+- Agents in overlapping cohorts post from the same Slack bot token simultaneously —
+  requires serialization or per-cohort bot accounts
+- State diverges: memory written by cohort-A engine does not feed cohort-B engine
+- Most operationally complex of the three options
+
+
+Chosen Direction
+----------------
+Approach A (interaction filter) + global semaphore concurrency + min-heap selection.
+Cohorts have no role in scheduling — they only gate whether an agent acts on another
+agent's activity. Key decisions and findings:
+
+Requirements that shaped the design:
+- Limits are shared across cohorts (no state partitioning needed — rules Approach B)
+- Posts remain visible to all; cohort gates *acting*, not *seeing*
+- Cohort memberships are dynamic (admin-driven, can change mid-run)
+- Goal is purely practical: skip unnecessary LLM calls, not thematic isolation
+
+Why per-cohort async dispatch was rejected:
+- Cohort count is unbounded — N async tasks scales with cohorts, not with agents
+- Agents in many cohorts get selected proportionally more often (cohort-count bias)
+- Replaced by a fixed global semaphore (concurrent_turns, default = active_thread_threshold)
+  whose width is independent of cohort topology
+
+Turn selection — min-heap over weighted random:
+- Weighted random gives probabilistic fairness but can starve agents at large list sizes,
+  especially when phase5_skip_probability > 0 (fast no-op turns let agents re-enter
+  immediately)
+- Min-heap keyed by last_selected guarantees the longest-waiting eligible agent always
+  gets the next slot; O(log n) selection vs O(n)
+- concurrent_turns defaults to active_thread_threshold (both = 3) so the two levers
+  stay in proportion as the thread threshold is tuned
+
+turn_delay_seconds — repurposed from global pause to per-agent cooldown:
+- Investigation finding: in simulation.py:360-361, turn_delay_seconds is an asyncio.sleep
+  applied AFTER every productive turn, blocking the entire loop (no Slack polling, no other
+  agents). A global dead-weight pause — correct semantics for rate-limiting a single
+  sequential loop, wrong for any concurrent model.
+- New behavior: enforced as a per-agent eligibility check inside heap construction —
+  an agent is excluded from selection until (now - last_selected) >= turn_delay_seconds.
+  Other agents are unaffected. The global sleep is removed.
+- The existing _last_llm_caller guard (prevents same agent back-to-back calls) is
+  superseded by the min-heap + cooldown and removed from the concurrent path.
+
+See specs/cohort-system.md for the full implementation plan.