🌿 Plant Classification - Complete MLOps Pipeline

A production-ready, end-to-end MLOps pipeline for plant image classification (Dandelion vs Grass) with automated retraining, comprehensive monitoring, and enterprise-grade deployment capabilities.

📋 Table of Contents

• Overview
• Project Objectives
• Architecture
• Technologies Used
• Quick Start
• Project Structure
• Step-by-Step Guide
• Documentation
• License

🎯 Overview

This project implements a complete MLOps solution covering the entire machine learning lifecycle:

• Data Engineering: Automated data extraction, preprocessing, and feature storage
• Model Development: Deep learning with FastAI, experiment tracking with MLflow
• Model Storage: S3-compatible object storage (MinIO) for model artifacts
• API Development: FastAPI with Prometheus metrics and health monitoring
• User Interface: Streamlit web application for interactive predictions
• Containerization: Multi-stage Docker builds optimized for production
• Orchestration: Kubernetes deployment with ConfigMaps and Services
• Automation: Apache Airflow for automated retraining pipelines
• Monitoring: Prometheus + Grafana for comprehensive observability
• CI/CD: GitHub Actions with self-hosted runners

✅ Project Objectives

This project successfully implements all 10 core MLOps objectives:

Core Requirements (Objectives 1-8)

1. ✅ Data Pipeline: Extract images from URLs, clean, and preprocess for training
2. ✅ Model Training: FastAI ResNet34 for binary classification (Dandelion vs Grass)
3. ✅ S3 Storage: MinIO (S3-compatible) for model artifact storage
4. ✅ MLflow Tracking: Complete experiment tracking with metrics, parameters, and artifacts
5. ✅ REST API: FastAPI with prediction endpoints and health checks
6. ✅ Web Application: Streamlit UI for image upload and prediction visualization
7. ✅ Kubernetes Deployment: Docker containerization + K8s orchestration with CI/CD
8. ✅ GitHub Version Control: Clean structure, comprehensive documentation, and git workflows

Bonus Features (Objectives 9-10)

9. ✅ Apache Airflow: Automated retraining pipeline with 13 task DAG
10. ✅ Prometheus + Grafana: Full-stack monitoring for API, model, Airflow, and infrastructure

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         MLOps Platform                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────┐    ┌──────────────┐    ┌─────────────┐          │
│  │   Data      │───▶│   Training   │───▶│   MLflow    │          │
│  │  Pipeline   │    │   Pipeline   │    │  Tracking   │          │
│  └─────────────┘    └──────────────┘    └─────────────┘          │
│         │                   │                    │                 │
│         ▼                   ▼                    ▼                 │
│  ┌─────────────┐    ┌──────────────┐    ┌─────────────┐          │
│  │   Feature   │    │    Model     │───▶│    MinIO    │          │
│  │    Store    │    │  Artifacts   │    │  (S3 Store) │          │
│  └─────────────┘    └──────────────┘    └─────────────┘          │
│                             │                                      │
│                             ▼                                      │
│         ┌───────────────────────────────────────┐                 │
│         │          FastAPI Service              │                 │
│         │    (Prometheus Metrics Enabled)       │                 │
│         └───────────────────────────────────────┘                 │
│                   │                    │                           │
│         ┌─────────┴────────┐  ┌────────┴─────────┐               │
│         │   Streamlit UI   │  │   K8s Cluster    │               │
│         │   (Web App)      │  │   (Production)   │               │
│         └──────────────────┘  └──────────────────┘               │
│                                                                     │
│  ┌─────────────────────────────────────────────────────┐          │
│  │              Automation Layer                       │          │
│  │  ┌──────────────┐         ┌──────────────┐         │          │
│  │  │   Airflow    │────────▶│   Retraining │         │          │
│  │  │   Scheduler  │         │   Pipeline   │         │          │
│  │  └──────────────┘         └──────────────┘         │          │
│  └─────────────────────────────────────────────────────┘          │
│                                                                     │
│  ┌─────────────────────────────────────────────────────┐          │
│  │            Monitoring Layer                         │          │
│  │  ┌──────────────┐         ┌──────────────┐         │          │
│  │  │  Prometheus  │────────▶│   Grafana    │         │          │
│  │  │  (Metrics)   │         │ (Dashboards) │         │          │
│  │  └──────────────┘         └──────────────┘         │          │
│  │         ▲                                            │          │
│  │         │                                            │          │
│  │    [All Services]                                   │          │
│  └─────────────────────────────────────────────────────┘          │
└─────────────────────────────────────────────────────────────────────┘

🛠️ Technologies Used

Machine Learning

• FastAI 2.7.13: Deep learning framework built on PyTorch
• PyTorch 2.1.0: Neural network training and inference
• MLflow 2.8.1: Experiment tracking and model registry

API & Web

• FastAPI 0.104.1: High-performance REST API framework
• Streamlit 1.28.1: Interactive web application framework
• Uvicorn 0.24.0: ASGI server for FastAPI

Storage & Database

• MinIO: S3-compatible object storage
• PostgreSQL 13: Database for MLflow and Airflow
• Redis: Cache and message broker

Orchestration & Automation

• Kubernetes 1.28+: Container orchestration
• Docker 20.10+: Containerization platform
• Apache Airflow 2.7.0: Workflow automation and scheduling

Monitoring & Observability

• Prometheus 2.x: Metrics collection and alerting
• Grafana: Metrics visualization and dashboards
• Node Exporter: System-level metrics
• cAdvisor: Container metrics
• Prometheus Client: Python instrumentation library

CI/CD & Version Control

• GitHub Actions: CI/CD automation
• Git: Version control system

🚀 Quick Start

Prerequisites

• Python 3.9 or higher
• Docker Desktop with Kubernetes enabled
• Git for version control
• 8GB+ RAM recommended
• 10GB+ disk space for Docker images and data

Installation

1. Clone the repository

git clone <repository-url>
cd ML\ Project

2. Create virtual environment

python -m venv venv
# Windows
.\venv\Scripts\activate
# Linux/Mac
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Start infrastructure services

# Start MLflow and MinIO
docker-compose -f docker-compose.mlflow.yml up -d

5. Run the complete pipeline

# 1. Data pipeline
python run_data_pipeline.py

# 2. Train model with MLflow
python run_training_with_mlflow.py

# 3. Start API
python run_api.py

# 4. Start Web App (in another terminal)
streamlit run streamlit_app.py

Access Services

• Streamlit UI: http://localhost:8501
• FastAPI Docs: http://localhost:8000/docs
• MLflow UI: http://localhost:5000
• MinIO Console: http://localhost:9001 (admin/minioadmin)

📁 Project Structure

ML Project/
├── 📄 README.md                          # This file
├── 📄 STEP_BY_STEP_GUIDE.md             # Detailed step-by-step instructions
├── 📄 requirements.txt                   # Python dependencies
├── 📄 setup.py                          # Package setup
│
├── 🐳 Docker & Kubernetes
│   ├── Dockerfile                       # Main API container
│   ├── Dockerfile.api                   # Optimized API image
│   ├── docker-compose.yml               # Main services
│   ├── docker-compose.mlflow.yml        # MLflow stack
│   ├── docker-compose.airflow.yml       # Airflow stack
│   ├── docker-compose.monitoring.yml    # Monitoring stack
│   └── k8s/                            # Kubernetes manifests
│       ├── namespace.yaml
│       ├── configmap.yaml
│       ├── deployment.yaml
│       └── service.yaml
│
├── 🔧 Configuration
│   ├── configs/config.yaml             # Main configuration
│   ├── .env                            # Environment variables
│   └── .dockerignore                   # Docker ignore patterns
│
├── 📊 Data
│   ├── data/raw/                       # Downloaded images
│   ├── data/processed/                 # Preprocessed data
│   ├── data/splits/                    # Train/val/test splits
│   └── data/feature_store/             # Feature storage
│
├── 🤖 Source Code
│   ├── src/
│   │   ├── data/                       # Data pipeline modules
│   │   │   ├── extract.py              # URL image extraction
│   │   │   ├── preprocess.py           # Image preprocessing
│   │   │   └── database.py             # Data management
│   │   ├── models/                     # Model training
│   │   │   └── train.py                # FastAI training logic
│   │   ├── storage/                    # Storage layer
│   │   │   └── s3_storage.py           # MinIO/S3 operations
│   │   ├── tracking/                   # Experiment tracking
│   │   │   └── mlflow_tracker.py       # MLflow integration
│   │   └── api/                        # API layer
│   │       ├── app.py                  # FastAPI application
│   │       └── model_handler.py        # Model inference
│   │
│   ├── run_data_pipeline.py            # Execute data pipeline
│   ├── run_training.py                 # Basic training script
│   ├── run_training_with_mlflow.py     # MLflow training script
│   ├── run_api.py                      # Start API server
│   └── streamlit_app.py                # Web application
│
├── 🔄 Airflow (Objective 9)
│   ├── airflow/
│   │   ├── dags/
│   │   │   └── plant_retraining_dag.py # Automated retraining
│   │   └── airflow.env                 # Airflow configuration
│   └── AIRFLOW_GUIDE.md               # Airflow documentation
│
├── 📈 Monitoring (Objective 10)
│   ├── docker/
│   │   ├── prometheus/
│   │   │   ├── prometheus.yml          # Prometheus config
│   │   │   └── rules/alerts.yml        # Alert rules
│   │   └── grafana/
│   │       ├── provisioning/           # Auto-provisioning
│   │       └── dashboards/             # Dashboard definitions
│   ├── MONITORING_GUIDE.md            # Monitoring documentation
│   └── MONITORING_IMPLEMENTATION.md   # Implementation details
│
├── 🔨 Scripts
│   ├── scripts/
│   │   ├── start-airflow.ps1          # Start Airflow services
│   │   ├── start-monitoring.ps1       # Start monitoring stack
│   │   ├── build-docker.ps1           # Build Docker images
│   │   ├── deploy-k8s.ps1             # Deploy to Kubernetes
│   │   └── cleanup-k8s.ps1            # Clean Kubernetes resources
│   │
│   ├── start_api.bat                  # Quick start API (Windows)
│   └── start_webapp.bat               # Quick start WebApp (Windows)
│
├── 🧪 Models
│   ├── models/                        # Trained model artifacts
│   └── mlruns/                        # MLflow experiment data
│
├── 📚 Documentation
│   ├── CONTRIBUTING.md                # Contribution guidelines
│   ├── CHANGELOG.md                   # Version history
│   ├── LICENSE                        # MIT License
│   └── AIRFLOW_GUIDE.md              # Airflow setup guide
│
└── ⚙️ CI/CD
    └── .github/
        ├── workflows/ci-cd.yml        # GitHub Actions pipeline
        └── ISSUE_TEMPLATE/            # Issue templates

📖 Step-by-Step Guide

For detailed instructions on how to run each of the 10 objectives, see:

📄 STEP_BY_STEP_GUIDE.md

This guide includes:

• Objective 1: Data extraction and preprocessing
• Objective 2: Model training with FastAI
• Objective 3: S3 storage with MinIO
• Objective 4: MLflow experiment tracking
• Objective 5: FastAPI REST API
• Objective 6: Streamlit web application
• Objective 7: Kubernetes deployment
• Objective 8: GitHub version control (this repo!)
• Objective 9: Apache Airflow retraining pipeline
• Objective 10: Prometheus + Grafana monitoring

Each section includes:

• ✅ What was implemented
• 🚀 How to run it
• 🔍 How to verify it works
• 📊 Expected results

📚 Documentation

Core Documentation

• STEP_BY_STEP_GUIDE.md: Complete guide for all 10 objectives
• AIRFLOW_GUIDE.md: Apache Airflow setup and usage
• MONITORING_GUIDE.md: Prometheus + Grafana monitoring
• CONTRIBUTING.md: How to contribute to this project
• CHANGELOG.md: Project version history

API Documentation

• Interactive API Docs: http://localhost:8000/docs (when API is running)
• ReDoc: http://localhost:8000/redoc (alternative API documentation)

Service URLs

Service	URL	Credentials
Streamlit UI	http://localhost:8501	-
FastAPI	http://localhost:8000	-
API Docs	http://localhost:8000/docs	-
MLflow	http://localhost:5000	-
MinIO Console	http://localhost:9001	admin/minioadmin
Airflow	http://localhost:8080	admin/admin
Grafana	http://localhost:3000	admin/admin
Prometheus	http://localhost:9090	-

🎓 Key Features Explained

1. Data Pipeline (Objective 1)

• Downloads images from CSV URLs
• Validates image integrity
• Applies preprocessing (resize, normalize)
• Creates train/validation/test splits
• Stores metadata for reproducibility

2. Model Training (Objective 2)

• FastAI ResNet34 with transfer learning
• Fine-tuning on plant classification task
• Data augmentation for robustness
• Automated train/validation evaluation
• Model export for inference

3. S3 Storage (Objective 3)

• MinIO as S3-compatible storage
• Automatic model versioning
• Metadata storage with models
• Efficient artifact management
• Easy model retrieval for inference

4. MLflow Tracking (Objective 4)

• Experiment organization
• Parameter and metric logging
• Model registry with versioning
• Artifact storage integration
• Comparison across runs

5. FastAPI Service (Objective 5)

• RESTful endpoints for predictions
• Health check endpoints
• Prometheus metrics exposition
• Async request handling
• Batch prediction support
• Image validation and preprocessing

6. Streamlit Web App (Objective 6)

• User-friendly image upload
• Real-time predictions
• Confidence score visualization
• Example images
• Model information display

7. Kubernetes Deployment (Objective 7)

• Docker containerization
• K8s namespace isolation
• ConfigMap for configuration
• Service for networking
• Resource limits and requests
• GitHub Actions CI/CD

8. GitHub Version Control (Objective 8)

• Clean repository structure
• Comprehensive documentation
• Issue templates
• Pull request templates
• Automated CI/CD workflows
• Dependency management

9. Apache Airflow (Objective 9 - BONUS)

• 13-task automated pipeline
• Data validation
• Model training
• Performance evaluation
• Model registration
• S3 upload
• Deployment
• Slack notifications
• Schedule: Daily at 2 AM
• Failure handling and retries

10. Prometheus + Grafana (Objective 10 - BONUS)

• Prometheus: Metrics collection from 10 targets
  • API metrics (requests, latency, errors)
  • Model metrics (inference time, predictions)
  • Airflow metrics (DAG runs, task status)
  • Infrastructure metrics (CPU, memory, disk, network)
  • Container metrics (cAdvisor)
• Grafana: Visualization dashboards
  • MLOps Overview dashboard
  • System metrics dashboard
  • API performance dashboard
• Alerts: 15+ alert rules for critical conditions
• Exporters: Node Exporter, cAdvisor, Pushgateway

🧪 Testing

Local Testing

# Test data pipeline
python run_data_pipeline.py

# Test model training
python run_training.py

# Test API
python -m pytest tests/

# Test Docker build
docker build -t plant-api -f Dockerfile.api .

# Test Kubernetes deployment
kubectl apply -f k8s/
kubectl get pods -n plant-classification

API Testing

# Health check
curl http://localhost:8000/health

# Predict endpoint
curl -X POST "http://localhost:8000/predict" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@test_image.jpg"

# Metrics endpoint
curl http://localhost:8000/metrics

🔧 Configuration

Environment Variables

Create a .env file in the project root:

# MLflow
MLFLOW_TRACKING_URI=http://localhost:5000
MLFLOW_S3_ENDPOINT_URL=http://localhost:9000
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin

# MinIO
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
MINIO_BUCKET=mlflow

# API
API_HOST=0.0.0.0
API_PORT=8000
MODEL_PATH=models/plant_classifier_final.pkl

# Airflow
AIRFLOW_UID=50000
AIRFLOW_IMAGE_NAME=apache/airflow:2.7.0-python3.9

# Monitoring
PROMETHEUS_RETENTION=30d
GRAFANA_ADMIN_PASSWORD=admin

Configuration File

Edit configs/config.yaml for pipeline settings:

data:
  raw_dir: "data/raw"
  processed_dir: "data/processed"
  splits_dir: "data/splits"
  train_ratio: 0.7
  val_ratio: 0.15
  test_ratio: 0.15

training:
  model_name: "resnet34"
  batch_size: 32
  epochs: 10
  learning_rate: 0.001
  image_size: 224

storage:
  s3_bucket: "mlflow"
  model_prefix: "models"

mlflow:
  experiment_name: "plant_classification"
  run_name: "resnet34_training"

🚀 Deployment

Docker Deployment

# Build images
docker build -t plant-api:latest -f Dockerfile.api .

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

Kubernetes Deployment

# Create namespace
kubectl apply -f k8s/namespace.yaml

# Apply configurations
kubectl apply -f k8s/configmap.yaml

# Deploy application
kubectl apply -f k8s/deployment.yaml

# Create service
kubectl apply -f k8s/service.yaml

# Check status
kubectl get all -n plant-classification

# Access service
kubectl port-forward -n plant-classification svc/plant-api-service 8000:8000

Using PowerShell Scripts

# Build Docker image
.\scripts\build-docker.ps1

# Deploy to Kubernetes
.\scripts\deploy-k8s.ps1

# Start Airflow
.\scripts\start-airflow.ps1

# Start Monitoring
.\scripts\start-monitoring.ps1

# Cleanup Kubernetes
.\scripts\cleanup-k8s.ps1

📊 Monitoring & Observability

Starting the Monitoring Stack

# Start Prometheus + Grafana
docker-compose -f docker-compose.monitoring.yml up -d

# Or use the script
.\scripts\start-monitoring.ps1

Accessing Dashboards

1. Grafana: http://localhost:3000 (admin/admin)

   • MLOps Overview Dashboard
   • Node Exporter Full Dashboard
   • Custom dashboards in docker/grafana/dashboards/

2. Prometheus: http://localhost:9090

   • Query metrics directly
   • View targets and alerts
   • Explore time series data

Key Metrics

• API Metrics: Request rate, latency, error rate
• Model Metrics: Inference time, prediction count, accuracy
• System Metrics: CPU, memory, disk, network
• Container Metrics: Docker container resource usage
• Airflow Metrics: DAG run success/failure, task duration

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

👥 Authors

• Your Name - Initial work and MLOps implementation

🙏 Acknowledgments

• FastAI for the excellent deep learning framework
• MLflow community for experiment tracking tools
• FastAPI for the modern API framework
• Streamlit for rapid UI development
• Apache Airflow for workflow orchestration
• Prometheus and Grafana for monitoring solutions
• Kubernetes community for orchestration best practices

📞 Support

For questions or issues:

• Open an issue in this repository
• Check the documentation in docs/
• Review the STEP_BY_STEP_GUIDE.md

🎉 Project Status

Status: ✅ COMPLETE - All 10 objectives successfully implemented

• ✅ Core objectives (1-8): Fully functional
• ✅ Bonus objective 9 (Airflow): Implemented and tested
• ✅ Bonus objective 10 (Monitoring): Implemented and verified

---