CrawlSmart

Crawl any website, build a cited Q&A knowledge base over it, and ask questions — all inside your own Databricks workspace.

An AI agent crawls a site you point it at, lands the pages as a Delta table + a UC Volume of markdown, indexes them in Mosaic AI Vector Search, and serves cited, deep-linkable answers through a Knowledge Assistant (or built-in RAG). A React + FastAPI app on Databricks Apps ties it together: watch the crawl as a live graph, ask grounded questions, and check answer quality in an eval pane.

 crawl (httpx + trafilatura, agentic link planner via FMAPI)
   └─> Delta web_silver + UC Volume of .md
         └─> Mosaic AI Vector Search (Delta-Sync, databricks-gte-large-en)
               └─> AgentBricks Knowledge Assistant  ── or ──  built-in RAG
                     └─> FastAPI + React app (cited Q&A, live graph, eval)

Install — honest promise

Not a literal one click. It's one wrapper command running the inherent Databricks CLI sequence (bundle deploy → setup_job → start app), and it's UI-free once your workspace meets the prerequisites below.

Prerequisites

On your machine

• Databricks CLI ≥ 0.239 (databricks version) — for the Apps + schema/volume bundle resources. (Vector Search is provisioned by the setup job via the SDK.)
• Node 18+ (the installer builds the React SPA).

In the target workspace (admin, one-time) — the installer prechecks these and fails fast with guidance; it can't grant them for you:

• Unity Catalog + Serverless compute + Model Serving enabled.
• A region where your chat model and databricks-gte-large-en are served (e.g. NOT us-west-2 / eu-west-3).
• An existing catalog you can CREATE SCHEMA in.
• For the Knowledge Assistant option only: AgentBricks enabled and a serverless budget policy with a nonzero budget assigned to the installer.

Quickstart

git clone https://github.com/Praneeth16/crawlsmart && cd crawlsmart
databricks auth login --host https://<your-workspace>.cloud.databricks.com   # one OAuth; saves a profile
make install CATALOG=<your_catalog> PROFILE=<your_profile>

make install will:

1. databricks bundle deploy — create the schema, volume, crawl + setup jobs, and the App (with its service principal).
2. databricks bundle run setup_job — preflight, grant the app SP, create the Vector Search endpoint + Delta-Sync index, and (default) a managed Knowledge Assistant over it. By default the app opens blank (no seed crawl); pass seed=true to pre-crawl seed_url for a demo-ready first load.
3. Start the App on compute and print its URL.

Open the URL, enter a site URL and Start crawl (the app opens blank by default), then ask away.

What gets provisioned

Piece	How
UC schema + volume	bundle resource (resources/uc.yml)
Crawl job + eval job + setup job	bundle resource (resources/jobs.yml)
Databricks App + service principal	bundle resource (resources/app.yml)
Vector Search endpoint + Delta-Sync index	scripts/setup.py (needs the seeded source table first)
App-SP grants (UC SELECT, READ VOLUME, job run, endpoint query)	scripts/setup.py
Knowledge Assistant (qa_engine=ka)	scripts/setup.py via the SDK (no bundle resource type exists)

Configuration

All knobs are bundle variables (databricks.yml); override with --var, or set the matching env on make install:

Variable	Default	Notes
catalog	main	install target (must exist)
schema	crawlsmart	created
volume	docs_md	created
vs_endpoint	crawlsmart_vs	STANDARD; set to an existing endpoint to avoid a new billable one
chat_model	databricks-meta-llama-3-3-70b-instruct	answerer / KA fallback
embedding_model	databricks-gte-large-en	KA-attachable embeddings only
planner_model	databricks-meta-llama-3-3-70b-instruct	crawl follow/skip planner
suggest_model	databricks-gpt-5-4-mini	contextual question suggestions
seed	false	true pre-crawls seed_url on install for a demo-ready first load; default opens blank
seed_url	Databricks agent-framework docs	only used when seed=true
qa_engine	ka	ka = Knowledge Assistant; rag = built-in Vector Search RAG

The App self-heals: in ka mode it uses grounded Vector Search RAG until (and unless) the Knowledge Assistant endpoint is live, so it always answers.

Evaluation

The Evaluation tab runs a real, reproducible quality check on the crawled corpus — not a fixture. "Run evaluation" triggers a Databricks Job (jobs/eval.py) that auto-generates a golden question set from the crawled page titles, answers each through the same grounded retrieval pipeline the app serves, and scores groundedness and answer relevance with an LLM-as-judge (your own FMAPI model). The run is logged to an MLflow experiment in your workspace (/Users/<you>/crawlsmart-eval) and a row is appended to {catalog}.{schema}.eval_runs; the app reads back the latest summary on load. It runs as a Job (not inside the request) because N questions × several model calls is too much to do synchronously. Numbers differ run-to-run because they're real.

Scope & limitations

• Single shared corpus per install. There is one Vector Search index/corpus per deployment; each crawl overwrites it. Any app user crawling a new site replaces the knowledge base for everyone. This is intended for a single team / demo. For multi-tenant use, deploy one install per team (separate schema).
• The crawler only fetches public http(s) URLs (private/link-local/metadata hosts are blocked) and honors a site's robots.txt blanket disallow.

Crawl engine

Default is the http engine (httpx + trafilatura) — serverless-safe, ideal for docs/static sites. JS-heavy sites need the browser engine (Crawl4AI + Chromium), which requires a classic cluster (jobs/init-chromium.sh) and is not available on serverless-only workspaces.

Teardown

make destroy CATALOG=<your_catalog> PROFILE=<your_profile>

Removes the bundle-managed schema, volume, jobs, and app. The Vector Search endpoint/index and Knowledge Assistant are created by setup_job (not bundle-managed) — delete them manually if no longer needed. A STANDARD Vector Search endpoint bills continuously until deleted. The eval Job also creates an {catalog}.{schema}.eval_runs table and a /Users/<you>/crawlsmart-eval MLflow experiment (cheap to leave, tidy to remove).

Troubleshooting

• Answers say "I don't know" / no citations — the index hasn't finished its first sync, or the seed crawl returned thin content. Re-run setup_job or crawl a content-rich site.
• Region 404 on chat/embeddings — your workspace region doesn't serve the model; switch region or pick a served model via --var chat_model=....
• App shows "Frontend not built" — frontend/dist wasn't synced; re-run make install (it builds the SPA).
• App stopped — databricks bundle run crawlsmart_app -t prod.

Layout

backend/      FastAPI app (SSE crawl stream, RAG/KA chat, page + suggest APIs)
frontend/     Vite + React + Tailwind SPA (built into frontend/dist)
jobs/         crawl.py — the crawl pipeline (Job)
scripts/      preflight.py + setup.py — the one-shot provisioner
resources/    bundle resource definitions (uc / jobs / app)
databricks.yml  bundle root (variables, targets, engine: direct)
install.sh / Makefile   the install wrapper