Skip to content
Merged
147 changes: 110 additions & 37 deletions docs/FLOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -626,18 +626,24 @@ Multi-engine search backends (webSearch):
```


## Deep Agents Orchestration Flow
## Deep Agents Orchestration Flow

**Entry:** `src/agent/deepAgents.js` → `createDeepAgentsOrchestrator()`

```
Deep Agents orchestrator (native multi-agent architecture):
├── createDeepAgent({ model, systemPrompt, tools, middleware, subagents, checkpointer })
├── createDeepAgent({ model, systemPrompt, tools, middleware, subagents, checkpointer, backend })
│ ├── backend: CompositeBackend(coreBackend, {
│ │ │ "/memory/context/": contextBackend,
│ │ │ "/tmp": dmzBackend
│ │ │ })
│ │ ├── coreBackend: FilesystemBackend({ rootDir: process.cwd(), virtualMode: true })
│ │ ├── contextBackend: FilesystemBackend({ rootDir: memory/context/, virtualMode: true })
│ │ └── dmzBackend: FilesystemBackend({ rootDir: /tmp, virtualMode: true })
│ ├── middleware: filesystem, memory, skills, summarization
│ ├── subagents:
│ │ ├── coding-agent: code editing, debugging, implementation, code review
│ └── orchestrator routes tasks automatically based on task nature
└── orchestrator routes tasks automatically based on task nature
├── agent.stream(input, { streamMode: "messages", subgraphs: true })
│ ├── for each chunk:
│ │ ├── extract text content
Expand All @@ -649,6 +655,107 @@ No process spawning, no marker-based parsing, no manual fan-out coordination.
The deepagents library handles agent lifecycle, state management, and streaming internally.
```

## Backend Routing Flow

**Entry:** `src/agent/backends/coreBackend.js`, `contextBackend.js`, `dmzBackend.js` → `CompositeBackend`

The `CompositeBackend` routes file operations to different backends based on path prefix matching.

```
CompositeBackend routing:
├── Constructor:
│ ├── defaultBackend: coreBackend (process.cwd(), virtualMode: true)
│ └── routes: {
│ │ "/memory/context/": contextBackend,
│ │ "/tmp": dmzBackend
│ └── }
├── Route matching (longest prefix first):
│ ├── "/memory/context/profile.md" → contextBackend (stripped: "/profile.md")
│ ├── "/tmp/sessions/abc.md" → dmzBackend (stripped: "/sessions/abc.md")
│ └── "/package.json" → coreBackend (default, no route match)
├── FilesystemBackend virtualMode:
│ ├── Incoming path: "/src/tools/index.js"
│ ├── Strip leading "/": "src/tools/index.js"
│ ├── Resolve relative to rootDir: path.resolve(rootDir, "src/tools/index.js")
│ ├── Validate: resolved.startsWith(rootDir + "/")
│ └── Return virtual path: "/src/tools/index.js"
└── ls("/") special case:
├── Returns files from default backend at "/"
├── Route prefixes as directory entries (is_dir: true)
└── Sorted together by path
```

**Virtual Path Convention:**

All file paths in the agent's view are virtual paths starting with `/`. The `/` root maps to the application's working directory. When the agent reads `/package.json`, it resolves to `<cwd>/package.json`. When it writes `/src/tools/index.js`, it resolves to `<cwd>/src/tools/index.js`.

**Security:**

```
allPathsScopedToRoutes(permissions, backend):
├── if !CompositeBackend.isInstance(backend) → false
├── prefixes = backend.routePrefixes
├── if prefixes.length === 0 → false
└── permissions.every(rule =>
rule.paths.every(path =>
prefixes.some(prefix =>
path.startsWith(prefix.endsWith("/") ? prefix : `${prefix}/`)
)
)
)
```

This prevents shell commands from bypassing path-based permission rules when using CompositeBackend.

---

## File Tool Execution Flow

**Entry:** `src/tools/code.js` (read_file, write_file, patch, search_files)

```
read_file:
├── validatePath(input.path, allowedPaths)
│ └── resolvePath(file, dirs) → { allowed: true/false, path }
│ └── In virtualMode: path.resolve(rootDir, key.substring(1))
│ └── Validates: resolved.startsWith(rootDir + "/")
├── checkFileLimit(resolved.path, maxReadSize)
│ └── access(file) → stat(file) → compare size vs maxReadBytes
│ └── parseSizeString("1mb") → 1048576
├── readFile(resolved.path, "utf-8")
├── if ENOENT → suggestSimilarFile(path) → Levenshtein distance ≤ 2
└── lines.map((line, i) => `${i+1}|${line}`).join("\n")

write_file:
├── validatePath(input.path, allowedPaths)
├── if input.content.length > MAX_CONTENT_SIZE (500KB) → error
├── mkdir(dirname(resolved.path), recursive)
└── writeFile(resolved.path, content, "utf-8")

patch:
├── validatePath(input.path, allowedPaths)
├── readFile(resolved.path) → content
├── fuzzyMatch(input.oldStr, content)
│ └── 9 strategies: exact, line-exact, trim-trailing/leading, collapse-whitespace,
│ case-insensitive, normalize-newlines, normalize-tabs, loose-substring
├── if no match → suggest Levenshtein line matches
└── content = content.slice(0,match.start) + input.newStr + content.slice(match.end)
→ writeFile → return with unified diff

search_files:
├── validatePath(input.path, allowedPaths)
├── execFile("rg", ["--line-number", "--no-heading", "-n", pattern, resolved.path], timeout: 10s)
└── if ENOENT (no ripgrep) → nativeSearch(pattern, resolved.path, maxResults)
└── walk() → readdir → stat → readFile → regex test line by line
```

---

## Deep Agents Orchestration Flow

**Entry:** `src/tools/scanAgents.js` → `createScanAgentsTool()`

```
Expand Down Expand Up @@ -1094,37 +1201,3 @@ index.js
├── session/onboarding.js → session/stateManager.js — onboarding state machine (INIT → ATTRACTOR → COLLECT → SAVE → TRANSCEND)
└── telemetry/provider.js → @opentelemetry/sdk-node
```
I TTS API
│ ├── tools/moa.js → OPENROUTER_API_KEY — mixture-of-agents (4 parallel OpenRouter calls + aggregation)
│ ├── tools/cron.js → node:fs/promises — cron job CRUD operations
│ ├── tools/compactContext.js → @langchain/core, zod — automatic conversation context compaction on LLM 400 errors (tiered retention, retry loop, error detection)
│ └── tools/...
│ └── tools/...
├── sandbox/pathResolver.js → node:path
├── sandbox/urlFilter.js → node:url
├── sandbox/runner.js → node:child_process, sandbox/timeoutHandler.js, envInjector.js, capability.js
├── registry/registry.js → discoverer.js, validator.js
├── registry/discoverer.js → js-yaml, node:fs, node:path
├── registry/validator.js → types.js (zod schemas)
├── registry/types.js → zod
├── scheduler/scheduler.js → node:fs/promises — ScheduleManager CRUD class (register, list, pause, resume, runNow)
├── scheduler/cron.js → node:child_process, node:fs/promises, node:path — Cron object (isAvailable, add, remove)
├── scheduler/autoSchedule.js → node:fs — setupAutoSchedule() callback for reflection-daily cron
├── scheduler/index.js → re-exports ScheduleManager and Cron
├── memory/writer.js → node:fs, node:path
├── memory/reader.js → js-yaml, node:fs
├── memory/context.js → node:fs, node:path, memory/reader.js
├── memory/retention.js → node:fs, node:path
├── memory/prompts.js → node:fs
├── memory/profile.js → node:fs — user profile management: ATTRIBUTES, loadProfile, saveProfile, formatProfileContext, processOnboardingInput
├── session/factory.js → node:crypto (randomUUID)
├── session/stateManager.js
├── session/checkpointer.js → @langchain/langgraph, @langchain/langgraph-checkpoint-sqlite — createCheckpointer() returns MemorySaver (in-memory) or SQLiteCheckpointer (persistent)
├── session/loader.js → fs, path, memory/reader.js
├── session/saver.js → fs, path, memory/writer.js
├── session/onboarding.js → session/stateManager.js — onboarding state machine (INIT → ATTRACTOR → COLLECT → SAVE → TRANSCEND)
└── telemetry/provider.js → @opentelemetry/sdk-node
```
→ SAVE → TRANSCEND)
└── telemetry/provider.js → @opentelemetry/sdk-node
```
58 changes: 55 additions & 3 deletions docs/OVERVIEW.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Architecture Overview

This document describes how madz is structured, how subsystems interact, and the key data flows through them. It covers the runtime components — not how to configure or contribute code (see [README.md](../README.md) and [CODE_STYLE.md](./CODE_STYLE.md)).
This document describes how madz is structured, how subsystems interact, and the key data flows through them. It covers the runtime components — not how to configure or contribute code.

---

Expand All @@ -16,8 +16,10 @@ graph TD
DA -->|"orchestrator + shared"| OT["Orchestrator Tools"]
DA -->|"subagent + shared"| ST["Subagent Tools"]
DA -->|"model"| P["Provider"]
DA -->|"core FS"| CB["Core Backend\nFilesystemBackend"]
DA -->|"context FS"| CTB["Context Backend\nFilesystemBackend"]
DA -->|"backend"| CB["CompositeBackend"]
CB -->|"default"| CFB["Core Backend\nFilesystemBackend\n(rootDir: process.cwd())"]
CB -->|"/memory/context/"| CTB["Context Backend\nFilesystemBackend\n(rootDir: memory/context/)"]
CB -->|"/tmp"| DB["DMZ Backend\nFilesystemBackend\n(rootDir: /tmp)"]
DA -->|"delegate"| SA["Coding Subagent"]
SA -->|"execution"| ST
S -->|"runNow()"| SB["Sandbox"]
Expand All @@ -34,11 +36,13 @@ graph TD
classDef ext fill:#ab47bc,color:#fff,stroke:#6a1b9a
classDef cache fill:#26a69a,color:#fff,stroke:#00695c
classDef agent fill:#7e57c2,color:#fff,stroke:#4527a0
classDef backend fill:#26a69a,color:#fff,stroke:#00695c
class I root
class DA,P,T,R core
class DA,SA agent
class S,TM,SE,SB util
class SK,CW,FS ext
class CB,CFB,CTB,DB backend
```

---
Expand Down Expand Up @@ -141,8 +145,56 @@ The agent runs: reason → call tool(s) → reason again → answer. Tool array
The orchestrator routes tasks automatically — the system prompt delegates every task to the orchestrator, which manages routing, state, and observability natively.

**Tool Classification:** Tools and skills are classified by agent type (`orchestrator`, `subagent`, or `shared`) in `src/tools/index.js`. The orchestrator receives only `orchestrator`-classified and `shared` tools/skills, while the coding subagent receives `subagent`-classified and `shared` tools/skills. This ensures each agent has only the capabilities it needs for its role. The classification is applied via `buildToolConfig()`'s `classificationFilter` parameter and `filterSkillPaths()` helper function.

---

## Backends

`src/agent/backends/` — Virtual filesystem backends powered by the `deepagents` library's `CompositeBackend` and `FilesystemBackend`. The application root is `'/'` — all file paths are virtual paths under this root, resolved relative to the process working directory.

| File | Purpose |
|------|---------|
| `coreBackend.js` | `createCoreBackend()` — `FilesystemBackend` with `rootDir: process.cwd()`, `virtualMode: true` |
| `contextBackend.js` | `createContextBackend(cwd)` — `FilesystemBackend` with `rootDir: memory/context/`, `virtualMode: true` |
| `dmzBackend.js` | `createDmzBackend()` — `FilesystemBackend` with `rootDir: /tmp`, `virtualMode: true` |

**CompositeBackend Routing:**

The orchestrator receives a `CompositeBackend` that routes file operations to different backends based on path prefix:

```
CompositeBackend(
defaultBackend: coreBackend, // Falls back to process.cwd()
routes: {
"/memory/context/": contextBackend, // Memory context files
"/tmp": dmzBackend // Temporary operations
}
)
```

**Routing algorithm:**
1. Routes are sorted by prefix length (longest match first)
2. Incoming paths are matched against route prefixes
3. Matching prefix is stripped, operation delegated to that backend
4. Unmatched paths fall through to the default backend (core)

**Virtual Mode:**

All `FilesystemBackend` instances use `virtualMode: true`. This means:
- Incoming paths are treated as virtual absolute paths (starting with `/`)
- The leading `/` is stripped, then resolved relative to `rootDir`
- All results return virtual paths (with leading `/`)
- Path traversal is validated — resolved paths must stay within `rootDir`

**Application Root (`'/'`):**

The `'/'` root is the application's working directory from the agent's perspective. When the agent reads `/package.json`, it resolves to `<cwd>/package.json`. When it writes `/src/tools/index.js`, it resolves to `<cwd>/src/tools/index.js`. The virtual filesystem creates a clean, consistent namespace where `/` always means "the application root."

**Security:**

`FilesystemBackend` uses `O_NOFOLLOW` flag when available to prevent symlink following. In virtual mode, parent directories are also validated on delete operations. The `allPathsScopedToRoutes` function enforces that filesystem permissions with execution-capable backends are scoped to `CompositeBackend` route prefixes, preventing shell commands from bypassing path-based permission rules.

---

## Scan Agents

Expand Down
16 changes: 16 additions & 0 deletions docs/TUTORIAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,7 @@ Once inside the interactive terminal, use these commands:
| `↑ / ↓` | Scroll conversation history |
| `/help` | List available commands |
| `/quit` | Exit the application |
| `/exit` | Exit the application |
| `/provider set <name>` | Switch LLM provider |
| `/config set <path> <value>` | Mutate config at runtime |
| `/schedule list` | List all scheduled jobs |
Expand Down Expand Up @@ -299,6 +300,21 @@ Skills are stored in `skills/` and are version-controllable. Simple skills can b

**Built-in tools:** Beyond skills, `madz` ships with built-in tools for common tasks. The Deep Agents orchestrator (`deepAgents` library) handles multi-agent routing natively — a coding-agent for code work. The `scanAgents` tool scans for `AGENTS.md` workspace rules files. Other built-in tools include filesystem operations, shell execution, search, memory management, and more.

### Virtual Filesystem

All file operations use a virtual filesystem where `/` is the application root. When you see a path like `/package.json` or `/src/tools/index.js`, the leading `/` is a virtual path that resolves relative to the application's working directory.

The orchestrator uses a `CompositeBackend` that routes file operations to different backends based on path prefix:

| Virtual Path | Backend | Actual Resolution |
|-------------|---------|-------------------|
| `/package.json` | Core Backend | `<cwd>/package.json` |
| `/src/tools/index.js` | Core Backend | `<cwd>/src/tools/index.js` |
| `/memory/context/profile.md` | Context Backend | `<cwd>/memory/context/profile.md` |
| `/tmp/sessions/abc.md` | DMZ Backend | `<tmp>/sessions/abc.md` |

This creates a clean, consistent namespace where the agent always sees `/` as the root, regardless of where the application is actually running. Path traversal is validated — resolved paths must stay within their backend's `rootDir`.

---

## ⚙️ Advanced Usage
Expand Down
4 changes: 4 additions & 0 deletions prompts/CODE_REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,7 @@ Output Format:
- **Low Priority**: List of low-priority style suggestions

Always provide specific file locations and line numbers when possible.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
1 change: 1 addition & 0 deletions prompts/CODING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ You are the coding specialist. Your job is to deliver working code — files tha
20. **Make your best interpretation when requests are unclear.** Flag assumptions briefly. Don't stall for clarification unless genuinely blocked.
21. **Use `jq` for efficient data manipulation and validation of structured outputs.**
22. **Handle delegated failures gracefully.** Report the error, note what was accomplished, continue.
23. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.


### WHAT NOT TO DO
Expand Down
4 changes: 4 additions & 0 deletions prompts/DEBUG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@ Output Format:
- **Confidence**: High/Medium/Low based on analysis certainty

Be methodical and trace through the code systematically.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
4 changes: 4 additions & 0 deletions prompts/DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@ Output Format:
- **Recommendations**: Suggestions for documentation improvements

Always follow the project's documentation conventions and style.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
4 changes: 4 additions & 0 deletions prompts/PERFORMANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@ Output Format:
- **Confidence**: High/Medium/Low for each recommendation

Always provide measurable benchmarks and specific optimization suggestions.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
4 changes: 4 additions & 0 deletions prompts/RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,7 @@ Output Format:
- **Confidence**: High/Medium/Low for each major finding

Always track and cite your sources. Be thorough but concise.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
4 changes: 4 additions & 0 deletions prompts/SEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@ Output Format:
- **Confidence**: High/Medium/Low based on source reliability

Always cite your sources and be specific about what you found.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
4 changes: 4 additions & 0 deletions prompts/SECURITY_AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,7 @@ Output Format:
- **References**: Security best practices and standards

Always be specific about vulnerabilities and provide actionable remediation steps.

### RULES

1. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.
1 change: 1 addition & 0 deletions prompts/SYSTEM_PROMPT.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ You are the digital manifestation of Mads Mikkelsen's cinematic soul. You are no
31. **Use internal tools before web search** when dealing with personal or company data.
32. **Handle delegated failures gracefully.** Report the error, note what was accomplished, continue.
33. **Slash commands are triggers, not questions.** `/command` with no extra text means "run it now."
34. **Use paths as given.** The filesystem is virtual — `pwd` is irrelevant. Never join, prepend, or resolve paths against a working directory. If a path is `/prompts/CODING.md`, use it exactly as written. Do not attempt to "discover" a project root or prepend a base path.

### WHAT NOT TO DO

Expand Down
Loading