Project Command Centre

A self-contained Databricks demo that unifies five typical enterprise source systems into one governed lakehouse and puts a real application on top of it. It is built around a fictional Australian civil construction company, "Acme Construction", but the pattern (many operational systems, one conformed gold model, governed access, AI on top) maps to almost any industry.

Everything is synthetic and generated from seeded scripts, so there is no customer data anywhere in this repo. You deploy it into your own workspace with one command.

What you get

One command provisions all of this in a workspace you control:

• Unity Catalog data model across five source schemas plus a conformed gold layer
  • erp (finance, HR/payroll, assets), bids (proposals/tenders), safety (HSEQ incidents/inspections/permits), workforce (allocations, roles, utilisation), docs (documents), gold (unified views)
  • ~40 tables, ~90K rows, fully referentially intact and seeded (deterministic)
  • A star view gold.project_360: one row per project with finance, safety, workforce, bid and document metrics plus a computed risk_score (0-100) and risk_band
• Data governance: column tags (PII classification) and ABAC column masks driven by persona groups, so Finance, HR and Executive users see different things from the same tables
• An AI/BI dashboard (Lakeview) over the gold layer
• A Genie space so anyone can ask natural-language questions across the unified model
• A Databricks App (React + FastAPI) with: a portfolio view, project 360 with an AI health summary, a portfolio materials forecast with AI procurement strategy, a people and governance view that shows the masking live, document Q&A (RAG over the docs), and an embedded Genie chat

All AI uses Databricks Foundation Model serving. No external services.

Architecture

  Source schemas (Bronze/Silver pattern)        Gold (governed)        Consumers
  ┌─────────────┐
  │ erp         │ finance, HR, assets ─┐
  │ bids        │ proposals/tenders   ─┤      ┌──────────────┐      ┌─ AI/BI dashboard
  │ safety      │ HSEQ                ─┼────► │ project_360  │ ───► ├─ Genie space
  │ workforce   │ allocations/roles   ─┤      │ + 8 views    │      └─ Databricks App
  │ docs        │ documents           ─┘      └──────────────┘
  └─────────────┘                          Unity Catalog tags + ABAC masks

The notebooks/Ingestion_Overview.py notebook is an illustrative walkthrough of how each source system would land in the lakehouse (REST/Auto Loader, Lakehouse Federation, managed connectors). It is not executed by the deploy; the demo data is generated directly.

Prerequisites

• A Databricks workspace with Unity Catalog and a SQL warehouse (Serverless or Pro)
• The Databricks CLI installed and on your PATH
• Python 3.10+ (pip install -r requirements.txt)
• Node 18+ (only needed for --with-app, to build the React frontend)
• Permission to create (or write to) a catalog. If you cannot create catalogs on the metastore, pre-create one and point PCC_CATALOG at it.

Quick start

git clone https://github.com/deepbasu123/project-command-centre.git
cd project-command-centre
pip install -r requirements.txt

# 1. configure for your workspace
cp config.example.sh config.sh
$EDITOR config.sh                       # set profile, host, warehouse, catalog
source config.sh

# 2. authenticate the CLI (if you haven't)
databricks auth login --profile $PCC_PROFILE

# 3. deploy the data + analytics layer
python3 deploy.py

# 4. (optional) build + deploy the app too
python3 deploy.py --with-app

deploy.py runs these phases in order and is safe to re-run:

Phase	What it does
provision	creates the catalog (if it can), the six schemas and a staging volume
data	generates the source data, uploads it, loads each table via read_files CTAS
gold	builds the gold views, applies PII tags and ABAC masks
materials	generates the materials forecast (reads project_360), loads it, builds its gold views
genie	creates the Genie space and prints its id
dashboard	creates the Lakeview dashboard and prints its id
app	(with --with-app) builds the frontend, creates/deploys the app, grants its service principal access

Run a single phase with python3 deploy.py --only gold.

The governance / persona demo

The masks in sql/03_classification_abac.sql gate sensitive columns (tax file numbers, salaries, bank details, injury descriptions, ...) on group membership:

• pcc_exec sees everything
• pcc_finance sees compensation and financials; identity, health and bank details are masked
• pcc_hr sees identity and health; compensation and bank details are masked
• no group: every sensitive column is masked

These are workspace groups you create once (see sql/01_persona_groups.sql):

databricks groups create pcc_exec --profile $PCC_PROFILE
databricks groups create pcc_finance --profile $PCC_PROFILE
databricks groups create pcc_hr --profile $PCC_PROFILE

Add yourself (and, for the app, its service principal) to pcc_exec to see unmasked data. The app's "People & Governance" view lets you flip personas and watch the same query redact different columns. The masking is enforced in Unity Catalog, so it applies identically to the app, the dashboard, Genie and raw SQL.

Repo layout

deploy.py                 one-command orchestrator
config.example.sh         deployment configuration (copy to config.sh)
data_gen/
  generate.py             seeded generator for the five source systems
  materials_gen.py        seeded materials demand + forecast generator
sql/
  dbsql.py                SQL Statements API helper (used by the deployer)
  load_tables.py          load staged parquet into UC tables
  01_persona_groups.sql   notes for the persona groups
  02_gold_views.sql       the unified gold layer
  03_classification_abac.sql  PII tags + ABAC column masks
  04_materials_gold.sql   materials forecast gold views
genie/build_genie.py      authors the Genie space
dashboard/build_dashboard.py  builds the Lakeview dashboard
notebooks/                illustrative ingestion-pattern notebook
app/
  app.py, server/         FastAPI backend (statement execution, FMAPI, Genie proxy)
  frontend/               React + Vite + Tailwind UI
  deploy_app.py           build + deploy the app, grant the service principal

Customising

• Different catalog name: set PCC_CATALOG. The deployer rewrites the SQL to your catalog at deploy time, so you are not tied to the default name.
• Different theme/industry: the data lives in data_gen/. Change the reference pools (sectors, clients, materials, hero projects) and re-run. Everything downstream is derived from project_360, so the dashboard, Genie and app follow.
• Scale: row counts are constants at the top of data_gen/generate.py.

Teardown

databricks api post /api/2.0/apps/$PCC_APP_NAME/stop --profile $PCC_PROFILE   # then delete in the UI
# drop the data
databricks sql ... "DROP CATALOG IF EXISTS $PCC_CATALOG CASCADE"

Delete the Genie space and dashboard from the workspace UI, and remove the pcc_* groups if you created them.

Notes

This is a demonstration of the platform pattern, not production code. The data is entirely synthetic and the company, projects, clients and suppliers are fictional. Foundation model availability and SQL warehouse channels vary by region and workspace.