EcoGrid-Demand-Ops

An enterprise-grade, framework-driven Machine Learning Operations (MLOps) pipeline built for regional green energy grid-load prediction and continuous sensor telemetry tracking.

This repository replaces fragile manual scripts with an automated workflow managed by Prefect, tracks model performance tracking metrics via MLflow, validates incoming data health with Evidently, and packages the final app inside a multi-stage production Docker container.

Engineering & Pipeline Highlights

• Framework-Driven Orchestration: Powered by the Prefect pipeline engine to handle functional dependency graphs, structured logging, error handling, and automated task retry limits.
• Proactive Silent Degradation Alerts: Implements an Evidently structural data drift gate check to intercept shifting climate or weather anomalies before they corrupt production inference weights.
• Experiment Lineage Logging: Automatically tracks training hyperparameter logs, metric data iterations, and out-of-sample RMSE history within an MLflow registry.
• Production Packaging: Uses multi-stage Docker configurations to isolate heavy training dependencies from the lightweight, low-latency FastAPI edge-inference runtime wrapper.

System Tooling Architecture

graph TD
    A[Prefect Core Engine Flow Init] -->|Task 1: Run Sensor Monitor| B(Evidently Analytics)
    B -->|Calculate Dataset Feature Drift Matrix| C{Data Drift Identified?}
    C -->|YES: Halt Retraining & Alert Team| D[Pipeline Emergency Break]
    C -->|NO: Safe Pipeline Gate Pass| E[Task 2: Train Random Forest Regressor]
    E -->|Scikit-Learn Fits Fresh Metrics| F[MLflow Experiment Ledger]
    F -->|Export Serialized Binary Weights| G[FastAPI Performance Server]
    G -->|Continuous Container Health| H[Dockerized Edge Deployment]

Loading

MLOps Stack Specifications

• Pipeline Infrastructure: Prefect
• Metric Registry Ledger: MLflow
• Monitoring Matrix Engine: Evidently (Data Drift Preset)
• Application Infrastructure: FastAPI / Uvicorn Server / Pydantic V2
• Validation & Testing Runner: PyTest / Flake8 Linter
• Container Layer: Docker (Multi-Stage Build Configuration)

Running the MLOps Pipeline Framework

1. Project Workspace Setup

Clone this workspace repository and install the production dependency tree:

git clone 

cd EcoGrid-Demand-Ops
pip install -r requirements.txt

2. Start the Telemetry Orchestration Server Dashboard

Spin up your local Prefect orchestration database server profile:

prefect server start

Open your web browser and route to http://127.0.0.1:4200 to view interactive task execution graphs, system logs, and success telemetry in real-time.

3. Trigger the Orchestrated Workflow Flow

In a separate terminal window, execute the central orchestrator module script:

python -m src.orchestrator

This task checks data drift parameters first. If clean, it auto-spins the training module loop and maps parameters to your local MLflow instance (mlflow ui).

4. Build and Launch the Production Container

Build the optimized, multi-stage isolated container:

docker build -t ecogrid-demand-ops:latest .

Spin up the final live inference edge microservice server locally:

docker run -d -p 8000:8000 ecogrid-demand-ops:latest

5. Send an Edge Query Telemetry Payload

curl -X POST "http://127.0.0" \
     -H "Content-Type: application/json" \
     -d '{
       "ambient_temperature": 24.5,
       "solar_irradiance_kw": 610.0,
       "wind_speed_mps": 11.2,
       "grid_frequency_hz": 50.02
     }'

Quality Assurance & Test Suites

Run the automated validation routines verifying core server inference processing stability:

pytest tests/ -v