SynapseLoop

Visual perception Agent Loop for browser UI automation and testing.

Playwright screenshots → YOLO detection → Multimodal LLM decisions → Action execution, with a self-training YOLO pipeline that gets smarter over time.

Works on any page Playwright can open — web apps, admin consoles, marketing sites, or HTML games — driven by a task description rather than hardcoded selectors.

Architecture

Bootstrap ──► Application ──► Domain ◄── Adapters
(CLI/DI)      (AgentLoop)     (Ports)    (Playwright, YOLO, LLM, Training)

• Domain: Pure data models + Protocol interfaces + event types (zero dependencies)
• Application: AgentLoop state machine, perception strategy, event bus
• Adapters: Playwright browser, YOLO detector, OpenAI/Anthropic reasoner, training pipeline
• Bootstrap: CLI entry point, dependency injection, config loading

Quick Start

# Install
pip install -e ".[dev]"
playwright install chromium

# Run a task-driven agent loop
synapse run --url "http://localhost:3000" --task "Complete onboarding and open settings"

# Explore mode (free-form bug discovery)
synapse explore --url "http://localhost:3000" --duration 300

# A/B version comparison
synapse compare --old "http://staging/v1.2" --new "http://staging/v1.3" --task "Open shop"

# Launch with live dashboard
synapse run --url "https://example.com/app" --task "Submit the contact form" --dashboard

Configuration

Copy config/default.yaml and set your API keys:

export OPENAI_API_KEY="sk-..."
# or
export ANTHROPIC_API_KEY="sk-ant-..."

Self-Training Loop

The YOLO model improves automatically:

1. Each turn, screenshots + LLM feedback are collected
2. After N verified samples, YOLO fine-tuning triggers asynchronously
3. New model is hot-swapped if mAP improves
4. Over time, YOLO handles more detection → fewer LLM vision calls → lower cost