diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 000000000..3da493f8e --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,13 @@ +## Goal + + +## Changes +- + +## Testing + + +## Checklist +- [ ] Title is a clear sentence (<= 70 chars) +- [ ] Commits are signed (`git log --show-signature`) +- [ ] `submissions/labN.md` updated diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 000000000..348e41c85 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,69 @@ +name: Release image + +on: + push: + tags: + - "v*" + +concurrency: + group: release-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read + packages: write + +jobs: + publish: + name: Build and push QuickNotes image + runs-on: ubuntu-24.04 + timeout-minutes: 20 + + steps: + - name: Checkout repository + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2, resolved 2026-07-07 + + - name: Compute image tags + id: meta + shell: bash + run: | + set -euo pipefail + image="ghcr.io/${GITHUB_REPOSITORY}/quicknotes" + image="$(printf '%s' "$image" | tr '[:upper:]' '[:lower:]')" + version="${GITHUB_REF_NAME}" + + { + echo "image=${image}" + echo "version=${version}" + echo "version_tag=${image}:${version}" + echo "latest_tag=${image}:latest" + } >> "$GITHUB_OUTPUT" + + printf 'Publishing %s and %s\n' "${image}:${version}" "${image}:latest" + + - name: Login to GitHub Container Registry + shell: bash + env: + GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + printf '%s' "$GHCR_TOKEN" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin + + - name: Build and push image + shell: bash + run: | + set -euo pipefail + docker buildx create --use --name quicknotes-release-builder || docker buildx use quicknotes-release-builder + docker buildx inspect --bootstrap + docker buildx build \ + --platform linux/amd64 \ + --tag "${{ steps.meta.outputs.version_tag }}" \ + --tag "${{ steps.meta.outputs.latest_tag }}" \ + --push \ + ./app + + - name: Print published image digest + shell: bash + run: | + set -euo pipefail + docker buildx imagetools inspect "${{ steps.meta.outputs.version_tag }}" diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 000000000..de3f670d0 --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,98 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : +# +# Lab 5 — QuickNotes in a Vagrant VM. +# Boots an Ubuntu 24.04 LTS VM, installs Go 1.24.x, builds QuickNotes from the +# synced ./app folder, and runs it as a systemd service on guest port 8080. +# The host reaches it on http://127.0.0.1:18080. + +GO_VERSION = "1.24.5" + +Vagrant.configure("2") do |config| + # Ubuntu 24.04 LTS. The bento box ships VirtualBox Guest Additions, which we + # need for the default (virtualbox) synced-folder type. + config.vm.box = "bento/ubuntu-24.04" + + # Identify the VM clearly (shows up in the shell prompt and `vagrant ssh`). + config.vm.hostname = "quicknotes-vm" + + # Forward host 18080 -> guest 8080, bound to loopback only so the app is not + # exposed to the local network. + config.vm.network "forwarded_port", guest: 8080, host: 18080, host_ip: "127.0.0.1" + + # Mount the application source into the guest (default type: virtualbox). + config.vm.synced_folder "./app", "/opt/quicknotes/app" + + # Cap resources at 2 vCPU / 1024 MB RAM (over-provisioning is an antipattern). + config.vm.provider "virtualbox" do |vb| + vb.name = "quicknotes-vm" + vb.memory = 1024 + vb.cpus = 2 + end + + # Provisioning runs on first `vagrant up`; re-run with `vagrant provision`. + # The script is idempotent, so `vagrant up --provision` is safe to repeat. + config.vm.provision "shell", inline: <<-SHELL + #!/usr/bin/env bash + set -euo pipefail + + GO_VERSION="#{GO_VERSION}" + case "$(uname -m)" in + x86_64) + GO_ARCH="amd64" + ;; + aarch64|arm64) + GO_ARCH="arm64" + ;; + *) + echo "Unsupported guest architecture: $(uname -m)" >&2 + exit 1 + ;; + esac + GO_TARBALL="go${GO_VERSION}.linux-${GO_ARCH}.tar.gz" + + export DEBIAN_FRONTEND=noninteractive + apt-get update -y + apt-get install -y curl + + # Install the pinned Go version only if it is not already present. + if ! /usr/local/go/bin/go version 2>/dev/null | grep -q "go${GO_VERSION} "; then + echo "Installing Go ${GO_VERSION}..." + curl -fsSL "https://go.dev/dl/${GO_TARBALL}" -o "/tmp/${GO_TARBALL}" + rm -rf /usr/local/go + tar -C /usr/local -xzf "/tmp/${GO_TARBALL}" + fi + + # Make Go available on PATH for interactive logins (e.g. `vagrant ssh`). + echo 'export PATH=$PATH:/usr/local/go/bin' > /etc/profile.d/go.sh + + # Build QuickNotes from the synced source into a stable location. + install -d /var/lib/quicknotes + /usr/local/go/bin/go build -C /opt/quicknotes/app -o /usr/local/bin/quicknotes . + + # Run QuickNotes as a service so it survives reboots and starts on boot. + cat > /etc/systemd/system/quicknotes.service <<'UNIT' +[Unit] +Description=QuickNotes API +After=network-online.target +Wants=network-online.target + +[Service] +Environment=ADDR=:8080 +Environment=DATA_PATH=/var/lib/quicknotes/notes.json +Environment=SEED_PATH=/opt/quicknotes/app/seed.json +WorkingDirectory=/var/lib/quicknotes +ExecStart=/usr/local/bin/quicknotes +Restart=on-failure + +[Install] +WantedBy=multi-user.target +UNIT + + systemctl daemon-reload + systemctl enable quicknotes + systemctl restart quicknotes + + echo "Provisioning done. QuickNotes should be live on guest :8080." + SHELL +end diff --git a/ansible/files/quicknotes b/ansible/files/quicknotes new file mode 100755 index 000000000..be6613c27 Binary files /dev/null and b/ansible/files/quicknotes differ diff --git a/ansible/inventory.ini b/ansible/inventory.ini new file mode 100644 index 000000000..91e9ab7ad --- /dev/null +++ b/ansible/inventory.ini @@ -0,0 +1,2 @@ +[quicknotes_vm] +quicknotes-vm ansible_host=127.0.0.1 ansible_port=2222 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/default/virtualbox/private_key ansible_python_interpreter=/usr/bin/python3 ansible_ssh_common_args='-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa' diff --git a/ansible/playbook.yaml b/ansible/playbook.yaml new file mode 100644 index 000000000..17d31d062 --- /dev/null +++ b/ansible/playbook.yaml @@ -0,0 +1,168 @@ +- name: Deploy QuickNotes to the Lab 5 VM + hosts: quicknotes_vm + become: true + gather_facts: false + + vars: + quicknotes_user: quicknotes + quicknotes_group: quicknotes + quicknotes_home: /nonexistent + quicknotes_shell: /usr/sbin/nologin + quicknotes_binary_src: files/quicknotes + quicknotes_binary_dest: /usr/local/bin/quicknotes + quicknotes_service_name: quicknotes + quicknotes_service_unit_path: /etc/systemd/system/quicknotes.service + quicknotes_data_dir: /var/lib/quicknotes + quicknotes_data_path: /var/lib/quicknotes/notes.json + quicknotes_seed_path: /opt/quicknotes/app/seed.json + listen_addr: ":8080" + quicknotes_restart_sec: 4 + ansible_pull_enabled: true + ansible_pull_repo_url: https://github.com/Hidancloud/DevOps-Intro.git + ansible_pull_repo_branch: feature/lab7 + ansible_pull_checkout_dir: /var/lib/ansible-pull/quicknotes + ansible_pull_inventory_path: /etc/ansible/quicknotes-local.ini + ansible_pull_playbook_path: ansible/playbook.yaml + ansible_pull_service_name: ansible-pull-quicknotes.service + ansible_pull_timer_name: ansible-pull-quicknotes.timer + + tasks: + - name: Create the quicknotes system group + ansible.builtin.group: + name: "{{ quicknotes_group }}" + system: true + + - name: Create the quicknotes system user + ansible.builtin.user: + name: "{{ quicknotes_user }}" + group: "{{ quicknotes_group }}" + system: true + create_home: false + home: "{{ quicknotes_home }}" + shell: "{{ quicknotes_shell }}" + + - name: Ensure the quicknotes data directory exists + ansible.builtin.file: + path: "{{ quicknotes_data_dir }}" + state: directory + owner: "{{ quicknotes_user }}" + group: "{{ quicknotes_group }}" + mode: "0750" + + - name: Install the QuickNotes binary + ansible.builtin.copy: + src: "{{ quicknotes_binary_src }}" + dest: "{{ quicknotes_binary_dest }}" + owner: root + group: root + mode: "0755" + notify: restart quicknotes + + - name: Install the systemd unit + ansible.builtin.template: + src: templates/quicknotes.service.j2 + dest: "{{ quicknotes_service_unit_path }}" + owner: root + group: root + mode: "0644" + register: quicknotes_service_unit + notify: restart quicknotes + + - name: Reload systemd when the unit changes + ansible.builtin.systemd: + daemon_reload: true + when: quicknotes_service_unit.changed + + - name: Enable and start the quicknotes service + ansible.builtin.systemd: + name: "{{ quicknotes_service_name }}" + enabled: true + state: started + + - name: Install ansible-pull dependencies + ansible.builtin.apt: + name: + - ansible + - git + state: present + update_cache: true + cache_valid_time: 3600 + when: ansible_pull_enabled + + - name: Ensure ansible config directory exists + ansible.builtin.file: + path: /etc/ansible + state: directory + owner: root + group: root + mode: "0755" + when: ansible_pull_enabled + + - name: Install ansible-pull local inventory + ansible.builtin.template: + src: templates/quicknotes-local.ini.j2 + dest: "{{ ansible_pull_inventory_path }}" + owner: root + group: root + mode: "0644" + register: ansible_pull_local_inventory + when: ansible_pull_enabled + + - name: Ensure ansible-pull checkout directory exists + ansible.builtin.file: + path: "{{ ansible_pull_checkout_dir }}" + state: directory + owner: root + group: root + mode: "0755" + when: ansible_pull_enabled + + - name: Install ansible-pull service + ansible.builtin.template: + src: templates/ansible-pull-quicknotes.service.j2 + dest: "/etc/systemd/system/{{ ansible_pull_service_name }}" + owner: root + group: root + mode: "0644" + register: ansible_pull_service_unit + when: ansible_pull_enabled + + - name: Install ansible-pull timer + ansible.builtin.template: + src: templates/ansible-pull-quicknotes.timer.j2 + dest: "/etc/systemd/system/{{ ansible_pull_timer_name }}" + owner: root + group: root + mode: "0644" + register: ansible_pull_timer_unit + when: ansible_pull_enabled + + - name: Reload systemd when ansible-pull units change + ansible.builtin.systemd: + daemon_reload: true + when: + - ansible_pull_enabled + - ansible_pull_service_unit.changed or ansible_pull_timer_unit.changed + + - name: Prime ansible-pull once so the timer has a last-run anchor + ansible.builtin.systemd: + name: "{{ ansible_pull_service_name }}" + state: started + when: + - ansible_pull_enabled + - ansible_connection != 'local' + - ansible_pull_local_inventory.changed or ansible_pull_service_unit.changed or ansible_pull_timer_unit.changed + + - name: Enable and start the ansible-pull timer + ansible.builtin.systemd: + name: "{{ ansible_pull_timer_name }}" + enabled: true + state: started + when: ansible_pull_enabled + + handlers: + - name: restart quicknotes + ansible.builtin.systemd: + name: "{{ quicknotes_service_name }}" + state: restarted + daemon_reload: true diff --git a/ansible/templates/ansible-pull-quicknotes.service.j2 b/ansible/templates/ansible-pull-quicknotes.service.j2 new file mode 100644 index 000000000..24f6c54bc --- /dev/null +++ b/ansible/templates/ansible-pull-quicknotes.service.j2 @@ -0,0 +1,8 @@ +[Unit] +Description=Reconcile QuickNotes from Git via ansible-pull +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/ansible-pull -U {{ ansible_pull_repo_url }} -C {{ ansible_pull_repo_branch }} -d {{ ansible_pull_checkout_dir }} -i {{ ansible_pull_inventory_path }} {{ ansible_pull_playbook_path }} diff --git a/ansible/templates/ansible-pull-quicknotes.timer.j2 b/ansible/templates/ansible-pull-quicknotes.timer.j2 new file mode 100644 index 000000000..e3768fc03 --- /dev/null +++ b/ansible/templates/ansible-pull-quicknotes.timer.j2 @@ -0,0 +1,10 @@ +[Unit] +Description=Run ansible-pull for QuickNotes every 5 minutes + +[Timer] +OnBootSec=1min +OnUnitActiveSec=5min +Unit={{ ansible_pull_service_name }} + +[Install] +WantedBy=timers.target diff --git a/ansible/templates/quicknotes-local.ini.j2 b/ansible/templates/quicknotes-local.ini.j2 new file mode 100644 index 000000000..3758c7116 --- /dev/null +++ b/ansible/templates/quicknotes-local.ini.j2 @@ -0,0 +1,2 @@ +[quicknotes_vm] +quicknotes-vm ansible_connection=local ansible_python_interpreter=/usr/bin/python3 diff --git a/ansible/templates/quicknotes.service.j2 b/ansible/templates/quicknotes.service.j2 new file mode 100644 index 000000000..9a229d6da --- /dev/null +++ b/ansible/templates/quicknotes.service.j2 @@ -0,0 +1,19 @@ +[Unit] +Description=QuickNotes API +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User={{ quicknotes_user }} +Group={{ quicknotes_group }} +WorkingDirectory={{ quicknotes_data_dir }} +Environment=ADDR={{ listen_addr }} +Environment=DATA_PATH={{ quicknotes_data_path }} +Environment=SEED_PATH={{ quicknotes_seed_path }} +ExecStart={{ quicknotes_binary_dest }} +Restart=on-failure +RestartSec={{ quicknotes_restart_sec }} + +[Install] +WantedBy=multi-user.target diff --git a/app/Dockerfile b/app/Dockerfile new file mode 100644 index 000000000..2af69a552 --- /dev/null +++ b/app/Dockerfile @@ -0,0 +1,44 @@ +# syntax=docker/dockerfile:1 + +# ─── builder stage ─── +# Official Go image, pinned to a 1.24 patch (not :latest). +FROM golang:1.24.5 AS builder +WORKDIR /src + +# Copy dependency manifest first so the module layer is cached independently +# of the source. QuickNotes has no third-party deps, so there is no go.sum. +COPY go.mod ./ +RUN go mod download + +# Now copy the source — changes here don't invalidate the layer above. +COPY . . + +# Build a static, stripped, reproducible binary. CGO off so distroless-static +# (which has no libc / dynamic linker) can run it. +RUN CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /quicknotes . + +# A tiny healthcheck helper — distroless has no shell/wget/curl, so we ship a +# minimal static binary that hits /health and exits 0/1. +RUN CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /healthcheck ./cmd/healthcheck + +# Pre-create the data dir. A fresh named volume inherits this dir's ownership, +# so the nonroot user (65532) can write notes.json. +RUN mkdir -p /data + +# ─── runtime stage ─── +# Distroless static + nonroot: no shell, no package manager, runs as UID 65532. +FROM gcr.io/distroless/static:nonroot + +COPY --from=builder /quicknotes /quicknotes +COPY --from=builder /healthcheck /healthcheck +COPY --from=builder --chown=65532:65532 /data /data +COPY seed.json /seed.json + +# Sensible defaults so `docker run` works without extra env (compose overrides). +ENV ADDR=":8080" \ + DATA_PATH="/data/notes.json" \ + SEED_PATH="/seed.json" + +EXPOSE 8080 +USER nonroot +ENTRYPOINT ["/quicknotes"] diff --git a/app/cmd/healthcheck/main.go b/app/cmd/healthcheck/main.go new file mode 100644 index 000000000..77af75b50 --- /dev/null +++ b/app/cmd/healthcheck/main.go @@ -0,0 +1,29 @@ +// Command healthcheck is a tiny static binary used as the container +// healthcheck. Distroless images have no shell, wget, or curl, so we ship this +// instead. It performs a single GET /health and exits 0 on HTTP 200, else 1. +package main + +import ( + "net/http" + "os" + "time" +) + +func main() { + addr := os.Getenv("HEALTHCHECK_URL") + if addr == "" { + addr = "http://127.0.0.1:8080/health" + } + + client := &http.Client{Timeout: 2 * time.Second} + resp, err := client.Get(addr) + if err != nil { + os.Exit(1) + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + os.Exit(1) + } + os.Exit(0) +} diff --git a/cloud/cloudflare/README.md b/cloud/cloudflare/README.md new file mode 100644 index 000000000..4103a58ed --- /dev/null +++ b/cloud/cloudflare/README.md @@ -0,0 +1,43 @@ +# Cloudflare Tunnel Bonus Notes + +The bonus uses a Cloudflare quick tunnel, so no Cloudflare account, domain, or +secret token is stored in this repository. Quick tunnels produce ephemeral +`trycloudflare.com` URLs that change on every restart. + +## Run QuickNotes locally + +```bash +docker run --rm --name quicknotes-lab10 -p 8080:8080 \ + ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +``` + +If the GHCR image is not public yet, use the local image while waiting for the +GitHub package visibility step: + +```bash +docker build -t quicknotes:lab10 ./app +docker run --rm --name quicknotes-lab10 -p 8080:8080 quicknotes:lab10 +``` + +## Start a quick tunnel + +```bash +cloudflared tunnel --url http://localhost:8080 --protocol http2 --no-autoupdate +``` + +The `--protocol http2` flag avoids relying on outbound QUIC/UDP. In this lab +environment, the first QUIC quick tunnel registered but the generated hostname +did not resolve reliably; forcing HTTP/2 produced a stable reachable quick URL. + +Copy the generated `https://.trycloudflare.com` URL and verify it from a +different network, for example a phone on cellular data: + +```bash +curl -v https://.trycloudflare.com/health +``` + +## Measure warm latency + +```bash +cloud/scripts/measure-curl-latency.sh https://.trycloudflare.com/health 50 +``` diff --git a/cloud/huggingface-space/Dockerfile b/cloud/huggingface-space/Dockerfile new file mode 100644 index 000000000..ecf6e2256 --- /dev/null +++ b/cloud/huggingface-space/Dockerfile @@ -0,0 +1,4 @@ +# This Space intentionally runs the exact release artifact from GHCR instead of +# rebuilding QuickNotes. The deployment consumes the same immutable image that +# CI published for Lab 10, which keeps release and production behavior aligned. +FROM ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 diff --git a/cloud/huggingface-space/README.md b/cloud/huggingface-space/README.md new file mode 100644 index 000000000..4df1ddf97 --- /dev/null +++ b/cloud/huggingface-space/README.md @@ -0,0 +1,27 @@ +--- +title: QuickNotes Lab 10 +emoji: "📝" +colorFrom: blue +colorTo: green +sdk: docker +app_port: 8080 +pinned: false +short_description: QuickNotes API deployed from the Lab 10 GHCR release image. +--- + +# QuickNotes Lab 10 + +This Hugging Face Space runs the immutable Lab 10 QuickNotes release image: + +```text +ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +``` + +The Space uses the Docker SDK and sets `app_port: 8080` because QuickNotes +listens on port 8080. The public API endpoints are: + +```text +/health +/notes +/metrics +``` diff --git a/cloud/scripts/measure-curl-latency.sh b/cloud/scripts/measure-curl-latency.sh new file mode 100755 index 000000000..efb4b8465 --- /dev/null +++ b/cloud/scripts/measure-curl-latency.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [ "${1:-}" = "" ]; then + echo "usage: $0 [runs]" >&2 + exit 2 +fi + +url="$1" +runs="${2:-5}" + +tmp="$(mktemp)" +sorted="$(mktemp)" +trap 'rm -f "$tmp" "$sorted"' EXIT + +for i in $(seq 1 "$runs"); do + curl -o /dev/null -s -w '%{time_total}\n' "$url" | tee -a "$tmp" +done + +sort -n "$tmp" > "$sorted" +count="$(wc -l < "$sorted" | tr -d ' ')" +p50_index=$(( (count + 1) * 50 / 100 )) +p95_index=$(( (count + 1) * 95 / 100 )) +[ "$p50_index" -lt 1 ] && p50_index=1 +[ "$p95_index" -lt 1 ] && p95_index=1 +[ "$p50_index" -gt "$count" ] && p50_index="$count" +[ "$p95_index" -gt "$count" ] && p95_index="$count" + +p50="$(sed -n "${p50_index}p" "$sorted")" +p95="$(sed -n "${p95_index}p" "$sorted")" + +printf 'runs=%s\n' "$count" +printf 'p50=%ss\n' "$p50" +printf 'p95=%ss\n' "$p95" diff --git a/cloud/teardown.md b/cloud/teardown.md new file mode 100644 index 000000000..7ef7f0150 --- /dev/null +++ b/cloud/teardown.md @@ -0,0 +1,19 @@ +# Lab 10 Teardown + +## Hugging Face Space + +The Space is free, but it can be deleted after grading if desired: + +1. Open the Space on Hugging Face. +2. Go to Settings. +3. Choose Delete this Space. +4. Type the Space name to confirm. + +## Cloudflare Quick Tunnel + +Quick tunnels are ephemeral. Stop the local tunnel with `Ctrl-C`; the public +`trycloudflare.com` URL becomes invalid. Stop the local QuickNotes container too: + +```bash +docker rm -f quicknotes-lab10 +``` diff --git a/compose.yaml b/compose.yaml new file mode 100644 index 000000000..f077625ac --- /dev/null +++ b/compose.yaml @@ -0,0 +1,64 @@ +services: + quicknotes: + build: ./app + image: quicknotes:lab6 + ports: + - "8080:8080" + environment: + ADDR: ":8080" + DATA_PATH: "/data/notes.json" + SEED_PATH: "/seed.json" + volumes: + - quicknotes-data:/data + healthcheck: + test: ["CMD", "/healthcheck"] + interval: 10s + timeout: 3s + retries: 3 + start_period: 5s + restart: unless-stopped + + prometheus: + image: prom/prometheus:v3.5.0 + command: + - --config.file=/etc/prometheus/prometheus.yml + ports: + - "9090:9090" + volumes: + - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro + - ./monitoring/prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro + depends_on: + quicknotes: + condition: service_healthy + restart: unless-stopped + + blackbox-exporter: + image: prom/blackbox-exporter:v0.27.0 + command: + - --config.file=/etc/blackbox_exporter/blackbox.yml + ports: + - "9115:9115" + volumes: + - ./monitoring/blackbox/blackbox.yml:/etc/blackbox_exporter/blackbox.yml:ro + depends_on: + quicknotes: + condition: service_healthy + restart: unless-stopped + + grafana: + image: grafana/grafana:13.0.3-slim + environment: + GF_SECURITY_ADMIN_USER: quicknotes_admin + GF_SECURITY_ADMIN_PASSWORD: qn-lab8-monitoring + GF_USERS_ALLOW_SIGN_UP: "false" + ports: + - "3000:3000" + volumes: + - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro + - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro + depends_on: + - prometheus + restart: unless-stopped + +volumes: + quicknotes-data: diff --git a/docs/runbook/high-error-rate.md b/docs/runbook/high-error-rate.md new file mode 100644 index 000000000..185960b11 --- /dev/null +++ b/docs/runbook/high-error-rate.md @@ -0,0 +1,22 @@ +# High Error Rate + +## What this alert means + +QuickNotes is returning 4xx and 5xx responses for more than 5% of requests, sustained for at least 5 minutes. + +## Triage steps + +1. Confirm the alert is still active by checking Prometheus `/alerts` and Grafana panels for traffic and error ratio over the last 15 minutes. +2. Compare healthy versus failing requests with `curl` against `/health`, `/notes`, and one deliberately malformed `POST /notes` to determine whether the issue is broad or isolated to a route. +3. Inspect the QuickNotes container logs with `docker compose logs --tail=200 quicknotes` and correlate timestamps with the error spike. +4. Check whether the backing data file or mounted volume is writable and present, because write failures can surface as 5xx on note creation. + +## Mitigations + +1. Roll back the most recent change to the QuickNotes image or monitoring-related config if the spike started immediately after a deploy. +2. Temporarily reduce bad traffic by blocking malformed clients, rate-limiting the offending caller, or disabling the specific integration sending invalid payloads. +3. Restart the `quicknotes` container if the service is wedged but the root cause is still under investigation. + +## Post-incident + +Write a short blameless postmortem using the Lecture 1 template: timeline, customer impact, root cause, contributing factors, and concrete follow-up actions. diff --git a/monitoring/blackbox/blackbox.yml b/monitoring/blackbox/blackbox.yml new file mode 100644 index 000000000..e52285932 --- /dev/null +++ b/monitoring/blackbox/blackbox.yml @@ -0,0 +1,9 @@ +modules: + http_2xx: + prober: http + timeout: 10s + http: + method: GET + valid_status_codes: [200] + no_follow_redirects: false + preferred_ip_protocol: ip4 diff --git a/monitoring/grafana/dashboards/golden-signals.json b/monitoring/grafana/dashboards/golden-signals.json new file mode 100644 index 000000000..be1c792ad --- /dev/null +++ b/monitoring/grafana/dashboards/golden-signals.json @@ -0,0 +1,98 @@ +{ + "annotations": {"list": []}, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "description": "QuickNotes does not expose a request-duration histogram yet, so this panel uses request throughput as a temporary latency proxy.", + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "unit": "reqps"}, "overrides": []}, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}, + "id": 1, + "options": {"legend": {"displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "single", "sort": "none"}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "editorMode": "code", + "expr": "sum(rate(quicknotes_http_requests_total[1m]))", + "legendFormat": "requests/sec proxy", + "range": true, + "refId": "A" + } + ], + "title": "Latency (proxy until histogram exists)", + "type": "timeseries" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "unit": "reqps"}, "overrides": []}, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}, + "id": 2, + "options": {"legend": {"displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "single", "sort": "none"}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "editorMode": "code", + "expr": "sum(rate(quicknotes_http_requests_total[5m]))", + "legendFormat": "traffic", + "range": true, + "refId": "A" + } + ], + "title": "Traffic", + "type": "timeseries" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 5}]}, "unit": "percent"}, "overrides": []}, + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}, + "id": 3, + "options": {"legend": {"displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "single", "sort": "none"}}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "editorMode": "code", + "expr": "100 * sum(rate(quicknotes_http_responses_by_code_total{code=~\"4..|5..\"}[5m])) / clamp_min(sum(rate(quicknotes_http_requests_total[5m])), 0.000001)", + "legendFormat": "error ratio", + "range": true, + "refId": "A" + } + ], + "title": "Errors", + "type": "timeseries" + }, + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 20}, {"color": "red", "value": 50}]}, "unit": "short"}, "overrides": []}, + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}, + "id": 4, + "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "center", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"}, + "targets": [ + { + "datasource": {"type": "prometheus", "uid": "prometheus"}, + "editorMode": "code", + "expr": "quicknotes_notes_total", + "legendFormat": "notes stored", + "range": true, + "refId": "A" + } + ], + "title": "Saturation", + "type": "stat" + } + ], + "refresh": "15s", + "schemaVersion": 41, + "tags": ["devops-intro", "quicknotes", "golden-signals"], + "templating": {"list": []}, + "time": {"from": "now-15m", "to": "now"}, + "timepicker": {}, + "timezone": "browser", + "title": "QuickNotes Golden Signals", + "uid": "quicknotes-golden-signals", + "version": 1, + "weekStart": "" +} diff --git a/monitoring/grafana/provisioning/alerting/.gitkeep b/monitoring/grafana/provisioning/alerting/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/monitoring/grafana/provisioning/dashboards/dashboard.yml b/monitoring/grafana/provisioning/dashboards/dashboard.yml new file mode 100644 index 000000000..de2713f80 --- /dev/null +++ b/monitoring/grafana/provisioning/dashboards/dashboard.yml @@ -0,0 +1,11 @@ +apiVersion: 1 + +providers: + - name: quicknotes-dashboards + orgId: 1 + folder: QuickNotes + type: file + disableDeletion: false + editable: false + options: + path: /var/lib/grafana/dashboards diff --git a/monitoring/grafana/provisioning/datasources/datasource.yml b/monitoring/grafana/provisioning/datasources/datasource.yml new file mode 100644 index 000000000..00f99157b --- /dev/null +++ b/monitoring/grafana/provisioning/datasources/datasource.yml @@ -0,0 +1,10 @@ +apiVersion: 1 + +datasources: + - name: Prometheus + uid: prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true + editable: false diff --git a/monitoring/grafana/provisioning/plugins/.gitkeep b/monitoring/grafana/provisioning/plugins/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/monitoring/prometheus/alerts.yml b/monitoring/prometheus/alerts.yml new file mode 100644 index 000000000..65ce08f3e --- /dev/null +++ b/monitoring/prometheus/alerts.yml @@ -0,0 +1,17 @@ +groups: + - name: quicknotes + rules: + - alert: QuickNotesHighErrorRate + expr: | + ( + sum(rate(quicknotes_http_responses_by_code_total{code=~"4..|5.."}[5m])) + / + clamp_min(sum(rate(quicknotes_http_requests_total[5m])), 0.000001) + ) > 0.05 + for: 5m + labels: + severity: page + annotations: + summary: "QuickNotes error ratio is above 5%" + description: "4xx + 5xx responses have exceeded 5% of total traffic for 5 minutes." + runbook: "docs/runbook/high-error-rate.md" diff --git a/monitoring/prometheus/prometheus.yml b/monitoring/prometheus/prometheus.yml new file mode 100644 index 000000000..6887c87f9 --- /dev/null +++ b/monitoring/prometheus/prometheus.yml @@ -0,0 +1,29 @@ +global: + scrape_interval: 15s + +rule_files: + - /etc/prometheus/alerts.yml + +scrape_configs: + - job_name: blackbox-internal + metrics_path: /probe + params: + module: [http_2xx] + static_configs: + - targets: + - http://quicknotes:8080/health + relabel_configs: + - source_labels: [__address__] + target_label: __param_target + - source_labels: [__address__] + target_label: __address__ + replacement: blackbox-exporter:9115 + - source_labels: [__param_target] + target_label: instance + - target_label: job + replacement: blackbox-internal + + - job_name: quicknotes + static_configs: + - targets: + - quicknotes:8080 diff --git a/monitoring/results/checkhost/metadata.json b/monitoring/results/checkhost/metadata.json new file mode 100644 index 000000000..70f9a858a --- /dev/null +++ b/monitoring/results/checkhost/metadata.json @@ -0,0 +1,16 @@ +{ + "target": "https://intensity-biological-beverages-perfectly.trycloudflare.com/health", + "regions": [ + "DE", + "SG", + "US" + ], + "window": { + "start": "2026-06-30T15:01:43Z", + "end": "2026-06-30T15:35:35Z", + "duration_seconds": 2032, + "interval_seconds": 60, + "samples": 31 + }, + "samples": 31 +} \ No newline at end of file diff --git a/monitoring/results/checkhost/summary.json b/monitoring/results/checkhost/summary.json new file mode 100644 index 000000000..b8b270eb1 --- /dev/null +++ b/monitoring/results/checkhost/summary.json @@ -0,0 +1,178 @@ +{ + "target": "https://intensity-biological-beverages-perfectly.trycloudflare.com/health", + "regions": [ + "DE", + "SG", + "US" + ], + "window": { + "start": "2026-06-30T15:01:43Z", + "end": "2026-06-30T15:35:35Z", + "duration_seconds": 2032, + "interval_seconds": 60, + "samples": 31 + }, + "selected_nodes": { + "DE": { + "node": "DE-FFM-KRK", + "country": "Germany", + "city": "Frankfurt am Main", + "continent": "EU", + "monitoring_allowed": 1 + }, + "SG": { + "node": "SG-SIN-FDCServers", + "country": "Singapore", + "city": "Singapore", + "continent": "AS", + "monitoring_allowed": 0 + }, + "US": { + "node": "US-DAL-Leora", + "country": "United States", + "city": "Dallas", + "continent": "NA", + "monitoring_allowed": 0 + } + }, + "nodes": { + "DE": { + "node": "DE-FFM-KRK", + "country": "Germany", + "city": "Frankfurt am Main", + "continent": "EU", + "monitoring_allowed": 1, + "checks": 31, + "errors": 0, + "statuses": [ + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200 + ], + "p50_seconds": 0.389, + "p95_seconds": 0.619 + }, + "SG": { + "node": "SG-SIN-FDCServers", + "country": "Singapore", + "city": "Singapore", + "continent": "AS", + "monitoring_allowed": 0, + "checks": 31, + "errors": 0, + "statuses": [ + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200 + ], + "p50_seconds": 0.99, + "p95_seconds": 1.549 + }, + "US": { + "node": "US-DAL-Leora", + "country": "United States", + "city": "Dallas", + "continent": "NA", + "monitoring_allowed": 0, + "checks": 31, + "errors": 0, + "statuses": [ + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200, + 200 + ], + "p50_seconds": 0.418, + "p95_seconds": 0.478 + } + }, + "overall": { + "checks": 93, + "errors": 0, + "p50_seconds": 0.428, + "p95_seconds": 1.284 + } +} \ No newline at end of file diff --git a/monitoring/scripts/checkhost-multi-region.sh b/monitoring/scripts/checkhost-multi-region.sh new file mode 100755 index 000000000..cc86fb7b1 --- /dev/null +++ b/monitoring/scripts/checkhost-multi-region.sh @@ -0,0 +1,229 @@ +#!/usr/bin/env bash +set -euo pipefail + +public_url="${1:?usage: checkhost-multi-region.sh [duration_seconds] [interval_seconds] [output_dir]}" +duration_seconds="${2:-1800}" +interval_seconds="${3:-60}" +output_dir="${4:-monitoring/results/checkhost}" +regions="${CHECKHOST_REGIONS:-DE,SG,US}" +append_mode="${CHECKHOST_APPEND:-0}" + +mkdir -p "$output_dir" +raw_file="$output_dir/raw.jsonl" +summary_file="$output_dir/summary.json" +metadata_file="$output_dir/metadata.json" +if [ "$append_mode" != "1" ]; then + : > "$raw_file" +fi + +start_epoch=$(date +%s) +end_epoch=$(( start_epoch + duration_seconds )) + +request_check() { + local response report_url result curl_error + + if ! response=$(curl -fsS -X POST https://api.check-host.cc/http \ + -H 'Content-Type: application/json' \ + -d "{\"target\":\"${public_url}\"}" 2>&1); then + jq -cn --arg status "request_failed" --arg error "$response" '{status:0, success:false, error:$error, stage:$status, data:{}}' + return 0 + fi + + report_url=$(printf '%s' "$response" | jq -r '.apiURL // empty') + if [ -z "$report_url" ]; then + jq -cn --arg status "missing_api_url" --arg error "$response" '{status:0, success:false, error:$error, stage:$status, data:{}}' + return 0 + fi + + result='' + curl_error='' + for _ in $(seq 1 20); do + sleep 3 + if result=$(curl -fsS "$report_url" 2>&1); then + if printf '%s' "$result" | jq -e '.success == true and (.data | type == "object") and ((.data | length) > 0)' >/dev/null 2>&1; then + printf '%s' "$result" + return 0 + fi + else + curl_error="$result" + fi + done + + jq -cn --arg status "report_unavailable" --arg error "$curl_error" '{status:0, success:false, error:$error, stage:$status, data:{}}' +} + +while [ "$(date +%s)" -lt "$end_epoch" ]; do + ts=$(date -u '+%Y-%m-%dT%H:%M:%SZ') + request=$(curl -fsS -X POST https://api.check-host.cc/http \ + -H 'Content-Type: application/json' \ + -d "{\"target\":\"${public_url}\"}" 2>/dev/null || true) + + if [ -n "$request" ] && printf '%s' "$request" | jq -e '.apiURL' >/dev/null 2>&1; then + report_url=$(printf '%s' "$request" | jq -r '.apiURL') + result='' + curl_error='' + for _ in $(seq 1 20); do + sleep 3 + if result=$(curl -fsS "$report_url" 2>&1); then + if printf '%s' "$result" | jq -e '.success == true and (.data | type == "object") and ((.data | length) > 0)' >/dev/null 2>&1; then + break + fi + else + curl_error="$result" + fi + done + if [ -z "$result" ] || ! printf '%s' "$result" | jq -e '.success == true and (.data | type == "object") and ((.data | length) > 0)' >/dev/null 2>&1; then + result=$(jq -cn --arg error "$curl_error" '{status:0, success:false, error:$error, stage:"report_unavailable", data:{}}') + fi + else + if [ -z "$request" ]; then + request=$(jq -cn '{status:0, success:false, error:"request_failed", data:{}}') + fi + result=$(jq -cn --arg error "request bootstrap failed" '{status:0, success:false, error:$error, stage:"request_failed", data:{}}') + fi + + jq -cn \ + --arg ts "$ts" \ + --arg url "$public_url" \ + --arg regions "$regions" \ + --argjson request "$request" \ + --argjson result "$result" \ + '{timestamp:$ts, url:$url, regions:($regions|split(",")), request:$request, result:$result}' >> "$raw_file" + + now=$(date +%s) + if [ "$now" -lt "$end_epoch" ]; then + sleep "$interval_seconds" + fi +done + +python3 - <<'PY' "$raw_file" "$summary_file" "$metadata_file" "$public_url" "$regions" "$interval_seconds" +import json, math, statistics, sys +from datetime import datetime, timezone +from pathlib import Path + +raw_path = Path(sys.argv[1]) +summary_path = Path(sys.argv[2]) +metadata_path = Path(sys.argv[3]) +public_url = sys.argv[4] +regions = [x for x in sys.argv[5].split(',') if x] +interval_seconds = int(sys.argv[6]) + +entries = [json.loads(line) for line in raw_path.read_text().splitlines() if line.strip()] + +def parse_ts(value): + return datetime.strptime(value, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=timezone.utc) + +def percentile(sorted_values, p): + if not sorted_values: + return None + idx = max(0, math.ceil((len(sorted_values) - 1) * p)) + return sorted_values[idx] + +start_ts = parse_ts(entries[0]["timestamp"]) if entries else None +end_ts = parse_ts(entries[-1]["timestamp"]) if entries else None + +summary = { + "target": public_url, + "regions": regions, + "window": { + "start": entries[0]["timestamp"] if entries else None, + "end": entries[-1]["timestamp"] if entries else None, + "duration_seconds": int((end_ts - start_ts).total_seconds()) if start_ts and end_ts else 0, + "interval_seconds": interval_seconds, + "samples": len(entries), + }, + "selected_nodes": {}, + "nodes": {}, + "overall": {}, +} + +selected_nodes = {} +for region in regions: + chosen = None + for entry in entries: + data = entry.get("result", {}).get("data", {}) + candidates = [] + for node_name, node in data.items(): + if node.get("countryCode") != region: + continue + checks = node.get("checks") or [] + if not checks: + continue + candidates.append((0 if node.get("monitoring_allowed") == 1 else 1, node_name, node)) + if candidates: + _, chosen_name, chosen_node = sorted(candidates, key=lambda item: (item[0], item[1]))[0] + chosen = { + "node": chosen_name, + "country": chosen_node.get("country"), + "city": chosen_node.get("city"), + "continent": chosen_node.get("continent"), + "monitoring_allowed": chosen_node.get("monitoring_allowed"), + } + break + if chosen: + selected_nodes[region] = chosen + +summary["selected_nodes"] = selected_nodes + +all_latencies = [] +all_errors = 0 +all_checks = 0 + +for region, chosen in selected_nodes.items(): + node_name = chosen["node"] + latencies = [] + errors = 0 + checks_count = 0 + statuses = [] + + for entry in entries: + data = entry.get("result", {}).get("data", {}) + node = data.get(node_name) + if not node: + continue + checks = node.get("checks") or [] + if not checks: + errors += 1 + checks_count += 1 + continue + first = checks[0] + checks_count += 1 + status = first.get("http_status") + statuses.append(status) + latency_ms = first.get("connectiontime") + if status != 200 or first.get("status") != 1: + errors += 1 + if latency_ms is not None: + latencies.append(float(latency_ms) / 1000.0) + + latencies.sort() + all_latencies.extend(latencies) + all_errors += errors + all_checks += checks_count + summary["nodes"][region] = { + **chosen, + "checks": checks_count, + "errors": errors, + "statuses": statuses, + "p50_seconds": statistics.median(latencies) if latencies else None, + "p95_seconds": percentile(latencies, 0.95), + } + +all_latencies.sort() +summary["overall"] = { + "checks": all_checks, + "errors": all_errors, + "p50_seconds": statistics.median(all_latencies) if all_latencies else None, + "p95_seconds": percentile(all_latencies, 0.95), +} + +summary_path.write_text(json.dumps(summary, indent=2)) +metadata_path.write_text(json.dumps({ + "target": public_url, + "regions": regions, + "window": summary["window"], + "samples": len(entries), +}, indent=2)) +PY + +echo "$summary_file" diff --git a/monitoring/scripts/generate-traffic.sh b/monitoring/scripts/generate-traffic.sh new file mode 100755 index 000000000..a3522f4ca --- /dev/null +++ b/monitoring/scripts/generate-traffic.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail + +base_url="${1:-http://localhost:8080}" +count="${2:-200}" + +for i in $(seq 1 "$count"); do + curl -fsS "$base_url/health" >/dev/null + curl -fsS "$base_url/notes" >/dev/null + curl -fsS -X POST "$base_url/notes" \ + -H 'Content-Type: application/json' \ + -d "{\"title\":\"note-$i\",\"body\":\"traffic\"}" >/dev/null + if (( i % 10 == 0 )); then + curl -sS "$base_url/notes/999999" >/dev/null || true + fi + sleep 0.1 +done diff --git a/monitoring/scripts/trigger-high-error-rate.sh b/monitoring/scripts/trigger-high-error-rate.sh new file mode 100755 index 000000000..5503bb925 --- /dev/null +++ b/monitoring/scripts/trigger-high-error-rate.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -euo pipefail + +base_url="${1:-http://localhost:8080}" +duration_seconds="${2:-330}" +end_epoch=$(( $(date +%s) + duration_seconds )) + +while [ "$(date +%s)" -lt "$end_epoch" ]; do + curl -fsS "$base_url/health" >/dev/null || true + curl -fsS -X POST "$base_url/notes" \ + -H 'Content-Type: application/json' \ + -d '{"title":"healthy","body":"ok"}' >/dev/null || true + curl -sS -X POST "$base_url/notes" \ + -H 'Content-Type: application/json' \ + -d '{"title":' >/dev/null || true + curl -sS "$base_url/notes/not-an-int" >/dev/null || true + sleep 1 +done diff --git a/submissions/lab10-checkhost-cloudflare/metadata.json b/submissions/lab10-checkhost-cloudflare/metadata.json new file mode 100644 index 000000000..eb881bd43 --- /dev/null +++ b/submissions/lab10-checkhost-cloudflare/metadata.json @@ -0,0 +1,16 @@ +{ + "target": "https://headed-approval-feels-image.trycloudflare.com/health", + "regions": [ + "DE", + "SG", + "US" + ], + "window": { + "start": "2026-07-07T20:11:29Z", + "end": "2026-07-07T20:11:29Z", + "duration_seconds": 0, + "interval_seconds": 1, + "samples": 1 + }, + "samples": 1 +} \ No newline at end of file diff --git a/submissions/lab10-checkhost-cloudflare/summary.json b/submissions/lab10-checkhost-cloudflare/summary.json new file mode 100644 index 000000000..fda8353c0 --- /dev/null +++ b/submissions/lab10-checkhost-cloudflare/summary.json @@ -0,0 +1,88 @@ +{ + "target": "https://headed-approval-feels-image.trycloudflare.com/health", + "regions": [ + "DE", + "SG", + "US" + ], + "window": { + "start": "2026-07-07T20:11:29Z", + "end": "2026-07-07T20:11:29Z", + "duration_seconds": 0, + "interval_seconds": 1, + "samples": 1 + }, + "selected_nodes": { + "DE": { + "node": "DE-FFM-KRK", + "country": "Germany", + "city": "Frankfurt am Main", + "continent": "EU", + "monitoring_allowed": 1 + }, + "SG": { + "node": "SG-SIN-FDCServers", + "country": "Singapore", + "city": "Singapore", + "continent": "AS", + "monitoring_allowed": 0 + }, + "US": { + "node": "US-DAL-Leora", + "country": "United States", + "city": "Dallas", + "continent": "NA", + "monitoring_allowed": 0 + } + }, + "nodes": { + "DE": { + "node": "DE-FFM-KRK", + "country": "Germany", + "city": "Frankfurt am Main", + "continent": "EU", + "monitoring_allowed": 1, + "checks": 1, + "errors": 0, + "statuses": [ + 200 + ], + "p50_seconds": 0.439, + "p95_seconds": 0.439 + }, + "SG": { + "node": "SG-SIN-FDCServers", + "country": "Singapore", + "city": "Singapore", + "continent": "AS", + "monitoring_allowed": 0, + "checks": 1, + "errors": 0, + "statuses": [ + 200 + ], + "p50_seconds": 0.935, + "p95_seconds": 0.935 + }, + "US": { + "node": "US-DAL-Leora", + "country": "United States", + "city": "Dallas", + "continent": "NA", + "monitoring_allowed": 0, + "checks": 1, + "errors": 0, + "statuses": [ + 200 + ], + "p50_seconds": 0.387, + "p95_seconds": 0.387 + } + }, + "overall": { + "checks": 3, + "errors": 0, + "p50_seconds": 0.439, + "p95_seconds": 0.935 + } +} \ No newline at end of file diff --git a/submissions/lab10-ghcr-public-package.png b/submissions/lab10-ghcr-public-package.png new file mode 100644 index 000000000..38834195d Binary files /dev/null and b/submissions/lab10-ghcr-public-package.png differ diff --git a/submissions/lab10-github-release-success.png b/submissions/lab10-github-release-success.png new file mode 100644 index 000000000..6fc73c2a6 Binary files /dev/null and b/submissions/lab10-github-release-success.png differ diff --git a/submissions/lab10-hf-space-running-root-404.png b/submissions/lab10-hf-space-running-root-404.png new file mode 100644 index 000000000..a3d8efa06 Binary files /dev/null and b/submissions/lab10-hf-space-running-root-404.png differ diff --git a/submissions/lab10.md b/submissions/lab10.md new file mode 100644 index 000000000..12d4e1d20 --- /dev/null +++ b/submissions/lab10.md @@ -0,0 +1,602 @@ +# Lab 10 - Cloud Computing: Ship QuickNotes to a Real Cloud + +## Objective + +Publish the QuickNotes container image to GitHub Container Registry from a tag-triggered GitHub Actions release workflow, prepare a Hugging Face Docker Space deployment that runs the immutable release image, and compare the hosted model with a Cloudflare quick tunnel. + +## Environment + +| Component | Version / value | +|-----------|-----------------| +| Branch | `feature/lab10` | +| Release tag | `v0.1.0` | +| Registry image | `ghcr.io/hidancloud/devops-intro/quicknotes` | +| Local Docker | Docker 29.1.5 | +| Local Go | go1.26.4 darwin/arm64 | +| Cloudflare tunnel | cloudflared 2026.6.1 | +| Benchmark tool | hyperfine 1.20.0 | + +## Evidence Status + +The repository-side implementation is complete: the release workflow, Hugging Face Space artifacts, Cloudflare tunnel notes, measurement helper, and this submission are included in the branch. + +Repository, registry, and hosted deployment evidence already collected: + +1. GitHub Actions release run succeeded: `https://github.com/Hidancloud/DevOps-Intro/actions/runs/28892339659`. +2. GHCR package is public and anonymously pullable. +3. The published `v0.1.0` image was pulled and smoke-tested locally from GHCR. +4. Hugging Face Space is live: `https://hiidancloud-inno-devops-quicknotes.hf.space`. + +Manual evidence still useful before final submission: + +1. Optional HF Space logs screenshot. + +No secrets, private keys, tokens, or production data are stored in this repository. + +--- + +## Task 1 - CI-Automated Push to GHCR + +### Implementation + +Workflow file: `.github/workflows/release.yml` + +```yaml +name: Release image + +on: + push: + tags: + - "v*" + +concurrency: + group: release-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read + packages: write + +jobs: + publish: + name: Build and push QuickNotes image + runs-on: ubuntu-24.04 + timeout-minutes: 20 + + steps: + - name: Checkout repository + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2, resolved 2026-07-07 + + - name: Compute image tags + id: meta + shell: bash + run: | + set -euo pipefail + image="ghcr.io/${GITHUB_REPOSITORY}/quicknotes" + image="$(printf '%s' "$image" | tr '[:upper:]' '[:lower:]')" + version="${GITHUB_REF_NAME}" + + { + echo "image=${image}" + echo "version=${version}" + echo "version_tag=${image}:${version}" + echo "latest_tag=${image}:latest" + } >> "$GITHUB_OUTPUT" + + printf 'Publishing %s and %s\n' "${image}:${version}" "${image}:latest" + + - name: Login to GitHub Container Registry + shell: bash + env: + GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + printf '%s' "$GHCR_TOKEN" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin + + - name: Build and push image + shell: bash + run: | + set -euo pipefail + docker buildx create --use --name quicknotes-release-builder || docker buildx use quicknotes-release-builder + docker buildx inspect --bootstrap + docker buildx build \ + --platform linux/amd64 \ + --tag "${{ steps.meta.outputs.version_tag }}" \ + --tag "${{ steps.meta.outputs.latest_tag }}" \ + --push \ + ./app + + - name: Print published image digest + shell: bash + run: | + set -euo pipefail + docker buildx imagetools inspect "${{ steps.meta.outputs.version_tag }}" +``` + +The workflow uses a tag trigger, builds from `./app`, publishes both immutable and mutable tags, and grants only `contents: read` plus `packages: write`. The only third-party action is `actions/checkout`, pinned to a full 40-character commit SHA. + +### Local validation output + +Go tests: + +```text +$ cd app && go test ./... +ok quicknotes 0.802s +? quicknotes/cmd/healthcheck [no test files] +``` + +YAML parsing: + +```text +$ ruby -e 'require "yaml"; YAML.load_file(".github/workflows/release.yml"); YAML.load_file("cloud/huggingface-space/README.md"); puts "YAML parsed"' +YAML parsed +``` + +GitHub Actions workflow lint: + +```text +$ actionlint .github/workflows/release.yml +(no output, exit 0) +``` + +Local image build and smoke test: + +```text +$ docker build -t quicknotes:lab10 ./app +#21 naming to docker.io/library/quicknotes:lab10 done +#21 DONE 0.0s + +$ docker run -d --name quicknotes-lab10-smoke -p 8080:8080 quicknotes:lab10 +7b5f344962163156a4d9df3839e6c74db7fd7895b5c43c69beaad49d8d96c71c + +$ curl -s http://localhost:8080/health +{"notes":4,"status":"ok"} +``` + +Local latency helper sanity check: + +```text +$ ./cloud/scripts/measure-curl-latency.sh http://localhost:8080/health 5 +0.002701 +0.002259 +0.002234 +0.002092 +0.002058 +runs=5 +p50=0.002234s +p95=0.002701s +``` + +### Release commands + +The intended release commands are: + +```bash +git tag -a -s v0.1.0 -m "Lab 10 release" +git push origin feature/lab10 +git push origin v0.1.0 +``` + +Local signed release evidence: + +```text +$ git log --show-signature -1 --format=fuller +commit 9c9bd0b102caee06c05da898953597650ae2a29f +Good "git" signature for hidancloud@yandex.ru with ED25519 key SHA256:0suWfmEHZ/Xt+yrRNKc2HZQbjzw33ZGHOnxmXKllv54 +Author: Arseny Pinigin +AuthorDate: Tue Jul 7 14:12:45 2026 +0300 +Commit: Arseny Pinigin +CommitDate: Tue Jul 7 14:12:45 2026 +0300 + + feat(lab10): add cloud release workflow + + Signed-off-by: Arseny Pinigin + +$ git tag -v v0.1.0 +Good "git" signature for hidancloud@yandex.ru with ED25519 key SHA256:0suWfmEHZ/Xt+yrRNKc2HZQbjzw33ZGHOnxmXKllv54 +object 9c9bd0b102caee06c05da898953597650ae2a29f +type commit +tag v0.1.0 +tagger Arseny Pinigin 1783422772 +0300 + +Lab 10 release +``` + +After the workflow succeeds, verify the image from an unauthenticated Docker session: + +```bash +docker logout ghcr.io || true +docker pull ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +docker run --rm -p 8080:8080 ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +curl -s http://localhost:8080/health +``` + +### Registry URL + +```text +ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +ghcr.io/hidancloud/devops-intro/quicknotes:latest +``` + +### Green release run + +```text +https://github.com/Hidancloud/DevOps-Intro/actions/runs/28892339659 +``` + +GitHub API confirmation: + +```text +$ curl -s 'https://api.github.com/repos/Hidancloud/DevOps-Intro/actions/workflows/release.yml/runs?per_page=3' | jq -r '.workflow_runs[] | [.id, .status, .conclusion, .html_url, .head_branch, .head_sha, .display_title, .run_started_at] | @tsv' +28892339659 completed success https://github.com/Hidancloud/DevOps-Intro/actions/runs/28892339659 v0.1.0 9c9bd0b102caee06c05da898953597650ae2a29f Release image 2026-07-07T19:19:23Z +``` + +The GitHub UI also shows `Release image #1` as successful with a total duration of 50 seconds. The only annotation is a non-failing GitHub runner warning that Node.js 20 actions are being forced to run on Node.js 24. + +![GitHub Actions release workflow success](lab10-github-release-success.png) + +### Public pull evidence + +This was tested after logging out from `ghcr.io`. Because this host is Apple Silicon and the release workflow intentionally published `linux/amd64` for GitHub/Hugging Face compatibility, the local verification uses `--platform linux/amd64`. On a normal `linux/amd64` clean machine, the same pull works without the platform override. + +The GitHub Packages UI shows the QuickNotes package as public: + +![GHCR public QuickNotes package](lab10-ghcr-public-package.png) + +```text +$ docker logout ghcr.io || true +$ docker pull --platform linux/amd64 ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +v0.1.0: Pulling from hidancloud/devops-intro/quicknotes +Digest: sha256:889abfe27fdffdfbe0fd8a513c5ea1eb33e305ead4e3371908139264f6105b88 +Status: Downloaded newer image for ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 + +$ docker image inspect ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 --format 'ID={{.Id}} Size={{.Size}} Architecture={{.Architecture}} OS={{.Os}}' +ID=sha256:889abfe27fdffdfbe0fd8a513c5ea1eb33e305ead4e3371908139264f6105b88 Size=5704983 Architecture=amd64 OS=linux + +$ docker run --platform linux/amd64 -d --name quicknotes-ghcr-smoke -p 18080:8080 ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +acfce2c4ef2bf85106a14dd6a811e37e00f51c0e4c88c3342d09dbb547e22d2f + +$ curl -s http://localhost:18080/health +{"notes":4,"status":"ok"} +``` + +### Design questions + +**a) OIDC vs `GITHUB_TOKEN`.** + +For pushing to GHCR from the same GitHub repository, `GITHUB_TOKEN` with `packages: write` is enough because GitHub can authorize the workflow directly against the repository package namespace. I would reach for OIDC when GitHub Actions must access an external cloud provider such as AWS, GCP, Azure, or Vault without storing long-lived cloud credentials as repository secrets. OIDC gives a short-lived, audience-bound identity token to the job, and the cloud provider exchanges it for temporary permissions based on claims such as repository, branch, environment, or workflow. That reduces secret sprawl and makes credential theft less useful because there is no static token to reuse later. + +**b) `latest` tag vs immutable version tag.** + +The immutable `:v0.1.0` tag is the release artifact used for reproducibility, rollback, audit, and deployments that must not drift. The `:latest` tag is still useful as a convenience pointer for humans, demos, smoke tests, and environments that intentionally follow the newest stable release. In production I would deploy the immutable tag and publish `latest` as a discoverability pointer, not as the source of truth for controlled rollouts. + +**c) Why only `packages: write`.** + +This is least privilege: the workflow token should have only the permission needed to publish the image. Compared with broad `write-all` permissions, this prevents a compromised build step from pushing commits, modifying issues, changing pull requests, creating releases, or writing unrelated repository state. If a malicious Docker build dependency or action step executes arbitrary code, the narrow token limits the blast radius to package publishing instead of handing it the whole repository. + +--- + +## Task 2 - Hugging Face Spaces Deployment + +### Space artifact layout + +The repository contains the files to copy or push into the Hugging Face Space Git repository: + +```text +cloud/huggingface-space/ +├── Dockerfile +└── README.md +``` + +### Space Dockerfile + +```dockerfile +# This Space intentionally runs the exact release artifact from GHCR instead of +# rebuilding QuickNotes. The deployment consumes the same immutable image that +# CI published for Lab 10, which keeps release and production behavior aligned. +FROM ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +``` + +### Space README with YAML frontmatter + +````markdown +--- +title: QuickNotes Lab 10 +emoji: "📝" +colorFrom: blue +colorTo: green +sdk: docker +app_port: 8080 +pinned: false +short_description: QuickNotes API deployed from the Lab 10 GHCR release image. +--- + +# QuickNotes Lab 10 + +This Hugging Face Space runs the immutable Lab 10 QuickNotes release image: + +```text +ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +``` + +The Space uses the Docker SDK and sets `app_port: 8080` because QuickNotes +listens on port 8080. The public API endpoints are: + +```text +/health +/notes +/metrics +``` +```` + +### Deployment evidence + +Hugging Face Space repository: + +```text +https://huggingface.co/spaces/HiidanCloud/inno-devops-quicknotes +``` + +Public Space URL: + +```text +https://hiidancloud-inno-devops-quicknotes.hf.space +``` + +The Space UI shows the Space as running. The browser view of the root path returns `404 page not found`, which is expected for this API-only service because QuickNotes exposes `/health`, `/notes`, and `/metrics`, not a homepage. + +![Hugging Face Space running with API-only root 404](lab10-hf-space-running-root-404.png) + +Space repo commit: + +```text +$ git -C /tmp/inno-devops-quicknotes-hf log --oneline --decorate -2 +f4f172d (HEAD -> main, origin/main, origin/HEAD) Deploy QuickNotes release image +d7e74f8 initial commit +``` + +The contents of `cloud/huggingface-space/` were pushed to the Space repository: + +```bash +git clone https://huggingface.co/spaces/HiidanCloud/inno-devops-quicknotes /tmp/inno-devops-quicknotes-hf +cp cloud/huggingface-space/Dockerfile cloud/huggingface-space/README.md /tmp/inno-devops-quicknotes-hf/ +cd /tmp/inno-devops-quicknotes-hf +git add Dockerfile README.md +git commit -m "Deploy QuickNotes release image" +git push +``` + +### Public health check + +```text +$ curl -v https://hiidancloud-inno-devops-quicknotes.hf.space/health +* Host hiidancloud-inno-devops-quicknotes.hf.space:443 was resolved. +* Connected to hiidancloud-inno-devops-quicknotes.hf.space (35.175.32.85) port 443 +* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 +* Server certificate: +* subject: CN=hf.space +* subjectAltName: host "hiidancloud-inno-devops-quicknotes.hf.space" matched cert's "*.hf.space" +* using HTTP/2 +> GET /health HTTP/2 +> Host: hiidancloud-inno-devops-quicknotes.hf.space +< HTTP/2 200 +< content-type: application/json +< x-proxied-replica: lo00sud0-z7xbb +< link: ;rel="canonical" +{"notes":4,"status":"ok"} +``` + +The `/notes` endpoint also returns the seeded QuickNotes data: + +```text +[{ + "id":4, + "title":"Endpoint cheat-sheet", + "body":"GET /notes GET /notes/{id} POST /notes DELETE /notes/{id} GET /health GET /metrics", + "created_at":"2026-01-15T10:15:00Z" +}, ...] +``` + +### Warm latency + +Five consecutive warm requests: + +```text +$ for i in 1 2 3 4 5; do curl -w '%{time_total}\n' -o /dev/null -s https://hiidancloud-inno-devops-quicknotes.hf.space/health; done +0.868571 +0.813725 +0.754585 +0.797764 +0.802027 +``` + +Warm p50 from the required five-request sample: `0.802027s`. + +Additional 50-run sample for p50/p95 comparison: + +```text +$ ./cloud/scripts/measure-curl-latency.sh https://hiidancloud-inno-devops-quicknotes.hf.space/health 50 +runs=50 +p50=0.801118s +p95=0.989296s +``` + +### Cold latency note + +Cold-start sampling was intentionally not completed. The lab text mentions waiting about 35 minutes, but the current Hugging Face documentation says free `cpu-basic` Spaces cannot configure a custom sleep time and are automatically paused after 48 hours of inactivity. Collecting three true cold samples would therefore require roughly six days of idle windows, which is out of proportion for this lab turn. + +Source: + +If the Space is allowed to sleep naturally, the exact command to run after each idle period is: + +```bash +curl -w '%{time_total}\n' -o /dev/null -s https://hiidancloud-inno-devops-quicknotes.hf.space/health +``` + +Cold samples table, not attempted for the reason above: + +| Sample | Idle time before request | `time_total` | +|--------|--------------------------|-------------:| +| 1 | 48h inactivity required on free `cpu-basic` | Not attempted | +| 2 | 48h inactivity required on free `cpu-basic` | Not attempted | +| 3 | 48h inactivity required on free `cpu-basic` | Not attempted | + +### Design questions + +**d) HF Spaces sleep vs Cloud Run scale to zero.** + +Both platforms remove idle compute and wake it on demand, but they optimize for different products. Cloud Run is a production serverless container platform, so it invests heavily in fast scheduling, request routing, image caching, concurrency controls, and predictable cold starts. Hugging Face Spaces is optimized for free public demos and ML apps; a sleeping Space may need to rehydrate a container, restore the runtime environment, and sometimes pull larger images or model assets. That makes HF wake-ups slower, but it keeps the free tier practical for many public demos. + +**e) Why `app_port: 8080`.** + +Hugging Face Docker Spaces default to port `7860`, which matches common Gradio and demo-app conventions in the Hugging Face ecosystem. QuickNotes listens on port `8080` and the lab explicitly says not to change the application. The `app_port: 8080` frontmatter tells the Spaces router which internal container port should receive public traffic. + +**f) Pulling from GHCR vs building inside the Space.** + +Pulling the GHCR image deploys the exact release artifact produced by CI. That is better for reproducibility and rollback because the Space runs `v0.1.0`, not whatever source happens to build later. It can also make Space builds smaller and faster because the Space Dockerfile is only a `FROM` line. The trade-off is that debugging may require looking at GitHub Actions and GHCR instead of only HF logs, and the Space depends on GHCR availability and package visibility. Building inside the Space is more self-contained and easier to inspect in HF logs, but it duplicates CI build logic and risks drift between release and deployment. + +--- + +## Bonus Task - Cloudflare Tunnel and Cross-Platform Comparison + +### Tool installation + +```text +$ brew install cloudflared hyperfine +==> Pouring cloudflared--2026.6.1.arm64_tahoe.bottle.tar.gz +==> Summary +/opt/homebrew/Cellar/cloudflared/2026.6.1: 10 files, 38.3MB +==> Pouring hyperfine--1.20.0.arm64_tahoe.bottle.tar.gz +/opt/homebrew/Cellar/hyperfine/1.20.0: 14 files, 1.2MB + +$ cloudflared --version +cloudflared version 2026.6.1 (built 2026-06-18T13:39:02Z) + +$ hyperfine --version +hyperfine 1.20.0 +``` + +### Local QuickNotes container + +```text +$ docker run --platform linux/amd64 -d --name quicknotes-lab10 -p 8080:8080 ghcr.io/hidancloud/devops-intro/quicknotes:v0.1.0 +67747b9a2f01e699f7c1d7790a392894ce085ebb30ca760e50ef1cbf1b721ee3 + +$ curl -s http://localhost:8080/health +{"notes":4,"status":"ok"} +``` + +### Quick tunnel + +```text +$ cloudflared tunnel --url http://localhost:8080 --protocol http2 --no-autoupdate +2026-07-07T20:11:03Z INF Your quick Tunnel has been created! Visit it at: +https://headed-approval-feels-image.trycloudflare.com +2026-07-07T20:11:03Z INF Version 2026.6.1 +2026-07-07T20:11:03Z INF Initial protocol http2 +``` + +This is an ephemeral quick-tunnel URL. It is valid only while the local `cloudflared` process keeps running. I forced HTTP/2 because the first QUIC tunnel showed degraded transport and its generated hostname did not resolve reliably. + +### Public `/health` verification from this machine + +```text +$ curl -v https://headed-approval-feels-image.trycloudflare.com/health +* Host headed-approval-feels-image.trycloudflare.com:443 was resolved. +* Connected to headed-approval-feels-image.trycloudflare.com (104.16.230.132) port 443 +* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 +* Server certificate: +* subject: CN=trycloudflare.com +* subjectAltName: host "headed-approval-feels-image.trycloudflare.com" matched cert's "*.trycloudflare.com" +* using HTTP/2 +> GET /health HTTP/2 +> Host: headed-approval-feels-image.trycloudflare.com +< HTTP/2 200 +< content-type: application/json +< cf-ray: a1797f6f7a73d7b3-MIA +< server: cloudflare +{"notes":4,"status":"ok"} +``` + +### External network verification + +I used Check-Host as an external multi-region verifier, which proves the tunnel was reachable from networks other than the laptop running `cloudflared`. + +Evidence files: + +```text +submissions/lab10-checkhost-cloudflare/summary.json +submissions/lab10-checkhost-cloudflare/metadata.json +``` + +Summary excerpt: + +```json +{ + "target": "https://headed-approval-feels-image.trycloudflare.com/health", + "regions": ["DE", "SG", "US"], + "nodes": { + "DE": {"country": "Germany", "city": "Frankfurt am Main", "errors": 0, "statuses": [200], "p50_seconds": 0.439}, + "SG": {"country": "Singapore", "city": "Singapore", "errors": 0, "statuses": [200], "p50_seconds": 0.935}, + "US": {"country": "United States", "city": "Dallas", "errors": 0, "statuses": [200], "p50_seconds": 0.387} + }, + "overall": {"checks": 3, "errors": 0, "p50_seconds": 0.439, "p95_seconds": 0.935} +} +``` + +### Warm latency measurement + +```text +$ hyperfine --warmup 3 --runs 50 "curl -o /dev/null -s https://headed-approval-feels-image.trycloudflare.com/health" +Benchmark 1: curl -o /dev/null -s https://headed-approval-feels-image.trycloudflare.com/health + Time (mean +/- sigma): 696.2 ms +/- 27.2 ms [User: 19.7 ms, System: 9.9 ms] + Range (min ... max): 661.9 ms ... 789.2 ms 50 runs +``` + +Explicit p50/p95 using the repository helper: + +```text +$ ./cloud/scripts/measure-curl-latency.sh https://headed-approval-feels-image.trycloudflare.com/health 50 +runs=50 +p50=0.675478s +p95=0.746448s +``` + +### Comparison table + +| Metric | HF Spaces (hosted) | Cloudflare Tunnel (local-via-edge) | +|--------|-------------------:|-----------------------------------:| +| Warm p50 | 0.801118 s | 0.675478 s | +| Warm p95 | 0.989296 s | 0.746448 s | +| Cold start | Not attempted: current free `cpu-basic` sleep window is 48h | N/A, continuously local while tunnel runs | +| Public URL stability | Stable for the Space name | Ephemeral on tunnel restart | +| Cost | Free | Free | + +### Design questions + +**g) Architectural difference.** + +In Hugging Face Spaces, the QuickNotes container runs in Hugging Face infrastructure and users connect to a platform-hosted service. In Cloudflare Tunnel, the container runs on my laptop and Cloudflare proxies public traffic from its edge back to the local machine over the tunnel. Both are useful cloud delivery models, but they solve different problems. For users, the distinction matters mainly through availability, latency, and operational responsibility: a hosted container survives my laptop sleeping; a quick tunnel does not. + +**h) Latency dominator.** + +For HF Spaces warm requests, the slow part is usually internet routing to HF plus the platform router and the container runtime path. If the Space has slept, cold start dominates by far. For Cloudflare Tunnel, the slow part is the round trip from user to Cloudflare edge, then from Cloudflare edge through the tunnel to my laptop and back. The QuickNotes handler itself is tiny; network path and tunnel forwarding dominate the measured latency. + +**i) When Cloudflare Tunnel is the right production pick.** + +Cloudflare Tunnel is a good production choice for controlled exposure of home lab services, on-prem internal tools, admin dashboards, webhook receivers, or private apps where the service must remain on an internal network but needs secure external ingress. It is also excellent for temporary stakeholder review URLs during development. A quick tunnel is never the right production pick for a stable public product because the URL is ephemeral and tied to a local process. Even named tunnels should not be used when the workload needs cloud autoscaling, regional high availability, managed runtime isolation, or independence from a single laptop or office network. + +--- + +## Repository Artifacts Added + +```text +.github/workflows/release.yml +cloud/huggingface-space/Dockerfile +cloud/huggingface-space/README.md +cloud/cloudflare/README.md +cloud/scripts/measure-curl-latency.sh +cloud/teardown.md +submissions/lab10.md +``` diff --git a/submissions/lab7.md b/submissions/lab7.md new file mode 100644 index 000000000..92532b6a0 --- /dev/null +++ b/submissions/lab7.md @@ -0,0 +1,689 @@ +# Lab 7 — Configuration Management: Deploy QuickNotes via Ansible + +## Objective + +Write an idempotent Ansible playbook that deploys the QuickNotes binary to the +Lab 5 Vagrant VM, installs a parameterized systemd unit, and manages the +service lifecycle through handlers. + +## Environment + +| Component | Version / value | +|-----------|-----------------| +| Host OS | macOS (authoring machine) | +| Git branch | `feature/lab7` | +| VM target | Lab 5 Vagrant VM (`127.0.0.1:18080 -> guest:8080`) | +| App artifact | `ansible/files/quicknotes` (Linux arm64 for this MacBook VM path) | +| Ansible layout | `ansible/` | + +> Note: this lab was completed on an Apple Silicon MacBook, so two small +> compatibility adjustments were required compared with the original Lab 5/7 +> assumptions: the Lab 5 `Vagrantfile` now picks the guest Go tarball by +> architecture, and the committed QuickNotes payload in `ansible/files/` was +> rebuilt for Linux arm64. + +## Implementation summary + +The `ansible/` directory contains: + +1. `inventory.ini` targeting the Lab 5 VM over the Vagrant SSH endpoint. +2. `playbook.yaml` with `become: true` and `gather_facts: false`. +3. `files/quicknotes`, a static Linux QuickNotes binary built from `app/`. +4. `templates/quicknotes.service.j2`, a Jinja2 systemd unit whose runtime + values come from playbook variables. +5. Bonus templates for `ansible-pull`: a local inventory plus systemd service + and timer units. + +The playbook performs the required idempotent steps: + +1. Creates the `quicknotes` system group and system user, with + `create_home: false` and a non-login shell. +2. Ensures `/var/lib/quicknotes` exists with owner `quicknotes:quicknotes` and + mode `0750`. +3. Copies the binary to `/usr/local/bin/quicknotes` with mode `0755`. +4. Renders `/etc/systemd/system/quicknotes.service` from a template. +5. Reloads systemd only when the rendered unit changed, then enables and starts + the service. +6. Restarts the service only when the binary or unit file changes, via a + handler. + +`gather_facts` is disabled intentionally because this playbook does not branch +on OS facts, CPU architecture, or package manager state; skipping the setup step +keeps runs shorter and makes the idempotency proof tighter. + +On this MacBook, the inventory also had to mirror `vagrant ssh-config` by +disabling strict host-key checks for the ephemeral local VM SSH endpoint. That +matched Vagrant's own SSH behavior and resolved an initial `Host key +verification failed` error from Ansible. + +## Repository layout + +```text +ansible/ +├── inventory.ini +├── playbook.yaml +├── files/ +│ └── quicknotes +└── templates/ + ├── ansible-pull-quicknotes.service.j2 + ├── ansible-pull-quicknotes.timer.j2 + ├── quicknotes-local.ini.j2 + └── quicknotes.service.j2 +``` + +## The playbook + +```yaml +- name: Deploy QuickNotes to the Lab 5 VM + hosts: quicknotes_vm + become: true + gather_facts: false + + vars: + quicknotes_user: quicknotes + quicknotes_group: quicknotes + quicknotes_home: /nonexistent + quicknotes_shell: /usr/sbin/nologin + quicknotes_binary_src: files/quicknotes + quicknotes_binary_dest: /usr/local/bin/quicknotes + quicknotes_service_name: quicknotes + quicknotes_service_unit_path: /etc/systemd/system/quicknotes.service + quicknotes_data_dir: /var/lib/quicknotes + quicknotes_data_path: /var/lib/quicknotes/notes.json + quicknotes_seed_path: /opt/quicknotes/app/seed.json + listen_addr: ":8080" + quicknotes_restart_sec: 4 + ansible_pull_enabled: true + ansible_pull_repo_url: https://github.com/Hidancloud/DevOps-Intro.git + ansible_pull_repo_branch: feature/lab7 + ansible_pull_checkout_dir: /var/lib/ansible-pull/quicknotes + ansible_pull_inventory_path: /etc/ansible/quicknotes-local.ini + ansible_pull_playbook_path: ansible/playbook.yaml + ansible_pull_service_name: ansible-pull-quicknotes.service + ansible_pull_timer_name: ansible-pull-quicknotes.timer + + tasks: + - name: Create the quicknotes system group + ansible.builtin.group: + name: "{{ quicknotes_group }}" + system: true + + - name: Create the quicknotes system user + ansible.builtin.user: + name: "{{ quicknotes_user }}" + group: "{{ quicknotes_group }}" + system: true + create_home: false + home: "{{ quicknotes_home }}" + shell: "{{ quicknotes_shell }}" + + - name: Ensure the quicknotes data directory exists + ansible.builtin.file: + path: "{{ quicknotes_data_dir }}" + state: directory + owner: "{{ quicknotes_user }}" + group: "{{ quicknotes_group }}" + mode: "0750" + + - name: Install the QuickNotes binary + ansible.builtin.copy: + src: "{{ quicknotes_binary_src }}" + dest: "{{ quicknotes_binary_dest }}" + owner: root + group: root + mode: "0755" + notify: restart quicknotes + + - name: Install the systemd unit + ansible.builtin.template: + src: templates/quicknotes.service.j2 + dest: "{{ quicknotes_service_unit_path }}" + owner: root + group: root + mode: "0644" + register: quicknotes_service_unit + notify: restart quicknotes + + - name: Reload systemd when the unit changes + ansible.builtin.systemd: + daemon_reload: true + when: quicknotes_service_unit.changed + + - name: Enable and start the quicknotes service + ansible.builtin.systemd: + name: "{{ quicknotes_service_name }}" + enabled: true + state: started + + - name: Install ansible-pull dependencies + ansible.builtin.apt: + name: + - ansible + - git + state: present + update_cache: true + cache_valid_time: 3600 + when: ansible_pull_enabled + + - name: Ensure ansible config directory exists + ansible.builtin.file: + path: /etc/ansible + state: directory + owner: root + group: root + mode: "0755" + when: ansible_pull_enabled + + - name: Install ansible-pull local inventory + ansible.builtin.template: + src: templates/quicknotes-local.ini.j2 + dest: "{{ ansible_pull_inventory_path }}" + owner: root + group: root + mode: "0644" + register: ansible_pull_local_inventory + when: ansible_pull_enabled + + - name: Ensure ansible-pull checkout directory exists + ansible.builtin.file: + path: "{{ ansible_pull_checkout_dir }}" + state: directory + owner: root + group: root + mode: "0755" + when: ansible_pull_enabled + + - name: Install ansible-pull service + ansible.builtin.template: + src: templates/ansible-pull-quicknotes.service.j2 + dest: "/etc/systemd/system/{{ ansible_pull_service_name }}" + owner: root + group: root + mode: "0644" + register: ansible_pull_service_unit + when: ansible_pull_enabled + + - name: Install ansible-pull timer + ansible.builtin.template: + src: templates/ansible-pull-quicknotes.timer.j2 + dest: "/etc/systemd/system/{{ ansible_pull_timer_name }}" + owner: root + group: root + mode: "0644" + register: ansible_pull_timer_unit + when: ansible_pull_enabled + + - name: Reload systemd when ansible-pull units change + ansible.builtin.systemd: + daemon_reload: true + when: + - ansible_pull_enabled + - ansible_pull_service_unit.changed or ansible_pull_timer_unit.changed + + - name: Prime ansible-pull once so the timer has a last-run anchor + ansible.builtin.systemd: + name: "{{ ansible_pull_service_name }}" + state: started + when: + - ansible_pull_enabled + - ansible_connection != 'local' + - ansible_pull_local_inventory.changed or ansible_pull_service_unit.changed or ansible_pull_timer_unit.changed + + - name: Enable and start the ansible-pull timer + ansible.builtin.systemd: + name: "{{ ansible_pull_timer_name }}" + enabled: true + state: started + when: ansible_pull_enabled + + handlers: + - name: restart quicknotes + ansible.builtin.systemd: + name: "{{ quicknotes_service_name }}" + state: restarted + daemon_reload: true +``` + +## Inventory + +```ini +[quicknotes_vm] +quicknotes-vm ansible_host=127.0.0.1 ansible_port=2222 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/default/virtualbox/private_key ansible_python_interpreter=/usr/bin/python3 ansible_ssh_common_args='-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa' +``` + +If `vagrant ssh-config` prints a different forwarded port or key path on your +machine, update `ansible_port` and `ansible_ssh_private_key_file` to match that +output before running the playbook. + +## Systemd unit template + +```ini +[Unit] +Description=QuickNotes API +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User={{ quicknotes_user }} +Group={{ quicknotes_group }} +WorkingDirectory={{ quicknotes_data_dir }} +Environment=ADDR={{ listen_addr }} +Environment=DATA_PATH={{ quicknotes_data_path }} +Environment=SEED_PATH={{ quicknotes_seed_path }} +ExecStart={{ quicknotes_binary_dest }} +Restart=on-failure +RestartSec={{ quicknotes_restart_sec }} + +[Install] +WantedBy=multi-user.target +``` + +## Static artifact build + +The managed binary was produced from `app/` as a static Linux executable for +the Apple Silicon VM path: + +```bash +GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \ + go build -C app -trimpath -ldflags='-s -w' \ + -o ../ansible/files/quicknotes . +``` + +## Task 1 — Run + verify + +The Task 1 dry-run and first real-run evidence below was captured before the +bonus `ansible-pull` block was appended to the playbook. The later Task 2 +idempotency proof and Bonus section show that the final extended playbook still +converges cleanly on the same VM. + +### Dry-run + +```bash +ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml --check +``` + +```text +PLAY [Deploy QuickNotes to the Lab 5 VM] *************************************** + +TASK [Create the quicknotes system group] ************************************** +changed: [quicknotes-vm] + +TASK [Create the quicknotes system user] *************************************** +changed: [quicknotes-vm] + +TASK [Ensure the quicknotes data directory exists] ***************************** +changed: [quicknotes-vm] + +TASK [Install the QuickNotes binary] ******************************************* +changed: [quicknotes-vm] + +TASK [Install the systemd unit] ************************************************ +changed: [quicknotes-vm] + +TASK [Reload systemd when the unit changes] ************************************ +ok: [quicknotes-vm] + +TASK [Enable and start the quicknotes service] ********************************* +ok: [quicknotes-vm] + +RUNNING HANDLER [restart quicknotes] ******************************************* +changed: [quicknotes-vm] + +PLAY RECAP ********************************************************************* +quicknotes-vm : ok=8 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 +``` + +### First real run + +```bash +ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml +``` + +```text +PLAY [Deploy QuickNotes to the Lab 5 VM] *************************************** + +TASK [Create the quicknotes system group] ************************************** +changed: [quicknotes-vm] + +TASK [Create the quicknotes system user] *************************************** +changed: [quicknotes-vm] + +TASK [Ensure the quicknotes data directory exists] ***************************** +changed: [quicknotes-vm] + +TASK [Install the QuickNotes binary] ******************************************* +changed: [quicknotes-vm] + +TASK [Install the systemd unit] ************************************************ +changed: [quicknotes-vm] + +TASK [Reload systemd when the unit changes] ************************************ +ok: [quicknotes-vm] + +TASK [Enable and start the quicknotes service] ********************************* +ok: [quicknotes-vm] + +RUNNING HANDLER [restart quicknotes] ******************************************* +changed: [quicknotes-vm] + +PLAY RECAP ********************************************************************* +quicknotes-vm : ok=8 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 +``` + +### Reachability proof + +```bash +curl -s http://localhost:18080/health +``` + +```text +{"notes":4,"status":"ok"} +``` + +### Design questions + +**a) `command:` vs dedicated modules.** +Dedicated modules such as `user`, `file`, `copy`, `template`, and `systemd` +understand the desired end state and compare it with the target's current +state. That makes them idempotent: if the target already matches, they return +`ok` instead of changing anything. `command:` and `shell:` just run imperative +commands; Ansible cannot reliably infer whether they changed state, so they are +usually reported as changed every run unless you add fragile guards like +`creates:` or `changed_when:`. For this lab, idempotency matters because the +second run must prove `changed=0` and the restart handler must fire only when an +actual config or binary change occurred. + +**b) `notify:` and handlers.** +A handler fires only when a task that references it reports `changed`. If the +`copy` or `template` task determines that the destination file already matches +what Ansible would write, the task stays `ok` and the handler does not run. That +is the right default because unnecessary restarts create avoidable downtime and +noise; service restarts should be coupled to real state changes, not to the mere +fact that a playbook was executed. + +**c) Variable hierarchy.** +For this lab I would use three levels: +1. Playbook vars for repo-local defaults such as `listen_addr`, + `quicknotes_data_path`, and `quicknotes_seed_path`, because they document the + intended baseline close to the tasks. +2. Inventory or `group_vars/quicknotes_vm` for environment-specific connection + values such as `ansible_host`, `ansible_port`, and `ansible_user`, because + they belong to the target environment rather than the app logic. +3. Extra vars (`-e`) only for one-off overrides during demonstrations, such as + `-e listen_addr=:9090` to prove selective template changes, because CLI vars + have high precedence and are ideal for temporary experiments without editing + committed files. + +**d) Do we need `gather_facts: true` here?** +No. This playbook does not make decisions based on Ansible facts; it copies a +binary, writes a template, and manages a service on a known Ubuntu VM. Turning +fact gathering off skips the setup phase, which usually saves a few seconds per +run and makes repeated idempotency checks faster. + +## Task 2 — Idempotency + selective re-run + +### Second run: prove `changed=0` + +```bash +ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml +``` + +```text +PLAY [Deploy QuickNotes to the Lab 5 VM] *************************************** + +TASK [Create the quicknotes system group] ************************************** +ok: [quicknotes-vm] + +TASK [Create the quicknotes system user] *************************************** +ok: [quicknotes-vm] + +TASK [Ensure the quicknotes data directory exists] ***************************** +ok: [quicknotes-vm] + +TASK [Install the QuickNotes binary] ******************************************* +ok: [quicknotes-vm] + +TASK [Install the systemd unit] ************************************************ +ok: [quicknotes-vm] + +TASK [Reload systemd when the unit changes] ************************************ +skipping: [quicknotes-vm] + +TASK [Enable and start the quicknotes service] ********************************* +ok: [quicknotes-vm] + +TASK [Install ansible-pull dependencies] *************************************** +ok: [quicknotes-vm] + +TASK [Ensure ansible config directory exists] ********************************** +ok: [quicknotes-vm] + +TASK [Install ansible-pull local inventory] ************************************ +ok: [quicknotes-vm] + +TASK [Ensure ansible-pull checkout directory exists] *************************** +ok: [quicknotes-vm] + +TASK [Install ansible-pull service] ******************************************** +ok: [quicknotes-vm] + +TASK [Install ansible-pull timer] ********************************************** +ok: [quicknotes-vm] + +TASK [Reload systemd when ansible-pull units change] *************************** +skipping: [quicknotes-vm] + +TASK [Prime ansible-pull once so the timer has a last-run anchor] ************** +skipping: [quicknotes-vm] + +TASK [Enable and start the ansible-pull timer] ********************************* +ok: [quicknotes-vm] + +PLAY RECAP ********************************************************************* +quicknotes-vm : ok=13 changed=0 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 +``` + +### Selective change: template only + handler fired + +One simple way to prove selective change without editing committed files is: + +```bash +ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml -e listen_addr=:9090 +``` + +Expected behavior: +- The `template` task reports `changed=1` +- The restart handler is invoked +- The `user`, `file`, and `copy` tasks stay `ok` + +```text +PLAY [Deploy QuickNotes to the Lab 5 VM] *************************************** + +TASK [Create the quicknotes system group] ************************************** +ok: [quicknotes-vm] + +TASK [Create the quicknotes system user] *************************************** +ok: [quicknotes-vm] + +TASK [Ensure the quicknotes data directory exists] ***************************** +ok: [quicknotes-vm] + +TASK [Install the QuickNotes binary] ******************************************* +ok: [quicknotes-vm] + +TASK [Install the systemd unit] ************************************************ +changed: [quicknotes-vm] + +TASK [Reload systemd when the unit changes] ************************************ +ok: [quicknotes-vm] + +TASK [Enable and start the quicknotes service] ********************************* +ok: [quicknotes-vm] + +RUNNING HANDLER [restart quicknotes] ******************************************* +changed: [quicknotes-vm] + +PLAY RECAP ********************************************************************* +quicknotes-vm : ok=8 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 +``` + +### Preview a third change with `--check --diff` + +```bash +ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml \ + -e quicknotes_data_path=/var/lib/quicknotes/notes-v2.json --check --diff +``` + +```diff +--- before: /etc/systemd/system/quicknotes.service ++++ after: /Users/arsenypinigin/.ansible/tmp/ansible-local-12275b4sv3mqt/tmpy1vbafoo/quicknotes.service.j2 +@@ -8,8 +8,8 @@ + User=quicknotes + Group=quicknotes + WorkingDirectory=/var/lib/quicknotes +-Environment=ADDR=:9090 +-Environment=DATA_PATH=/var/lib/quicknotes/notes.json ++Environment=ADDR=:8080 ++Environment=DATA_PATH=/var/lib/quicknotes/notes-v2.json + Environment=SEED_PATH=/opt/quicknotes/app/seed.json + ExecStart=/usr/local/bin/quicknotes + Restart=on-failure +``` + +### Design questions + +**e) Why does the second run report `changed=0`?** +Because Ansible modules compare the current remote state to the desired state. +`file` checks attributes such as existence, type, owner, group, and mode. +`copy` checks the destination file contents plus metadata. `template` renders the +Jinja2 template and compares the rendered bytes to the target file before +rewriting it. If nothing differs, the tasks are already converged and the play +reports `ok` rather than `changed`. + +**f) What would happen with `shell: 'echo "ADDR=..." > /etc/systemd/system/quicknotes.service'`?** +You would lose safe change detection, structured templating, and predictable file +metadata management. The shell task would typically report changed every run, +which would restart the service every run even when the unit content was +identical. It also makes quoting brittle, hides accidental truncation bugs, +cannot easily manage owner/mode consistently, and encourages partially written +or malformed units when variables contain spaces or special characters. In +short: more imperative code, weaker idempotency, and noisier deploys. + +**g) What bug does `--check --diff` catch that plain `--check` can miss?** +Plain `--check` tells you that a template *would* change, but not whether the +rendered result is sensible. `--diff` exposes the exact before/after lines, so +it catches bugs like accidentally changing `ADDR=:8080` to `ADDR=8080`, pointing +`SEED_PATH` at the wrong file, or rendering a typo into `ExecStart`. Those are +production-impacting config mistakes that a dry-run count alone would not make +obvious. + +## Bonus Task — `ansible-pull` GitOps loop + +### Implementation + +The bonus adds three VM-local artifacts: + +1. `/etc/ansible/quicknotes-local.ini` +2. `/etc/systemd/system/ansible-pull-quicknotes.service` +3. `/etc/systemd/system/ansible-pull-quicknotes.timer` + +The local inventory targets the VM itself via `ansible_connection=local`, so +the same `ansible/playbook.yaml` can run either from the host or from inside +the VM. The systemd service executes: + +```bash +/usr/bin/ansible-pull \ + -U https://github.com/Hidancloud/DevOps-Intro.git \ + -C feature/lab7 \ + -d /var/lib/ansible-pull/quicknotes \ + -i /etc/ansible/quicknotes-local.ini \ + ansible/playbook.yaml +``` + +The timer uses the required cadence: + +```ini +[Timer] +OnBootSec=1min +OnUnitActiveSec=5min +Unit=ansible-pull-quicknotes.service +``` + +### Timer proof + +`systemctl list-timers` on the VM: + +```text +Tue 2026-06-30 14:14:56 UTC 4min 44s Tue 2026-06-30 14:09:56 UTC 15s ago ansible-pull-quicknotes.timer ansible-pull-quicknotes.service +``` + +The local inventory installed in the VM: + +```ini +[quicknotes_vm] +quicknotes-vm ansible_connection=local ansible_python_interpreter=/usr/bin/python3 +``` + +### Convergence demonstration + +To produce a non-destructive but visible reconcile, I changed the playbook +variable `quicknotes_restart_sec` from `3` to `4`, committed it to +`feature/lab7`, and pushed to GitHub. + +Timeline: + +| Event | Time | +|-------|------| +| Git commit pushed (`5b92184`) | `2026-06-30T17:05:27+03:00` | +| Next timer fire on VM | `2026-06-30 14:09:56 UTC` | +| State observed reconciled | `2026-06-30 14:10:11 UTC` | + +State proof from the VM after the timer-fired run: + +```text +16:RestartSec=4 +``` + +Relevant `journalctl` excerpt: + +```text +Jun 30 14:10:06 quicknotes-vm ansible-pull[8484]: RUNNING HANDLER [restart quicknotes] ******************************************* +Jun 30 14:10:06 quicknotes-vm ansible-pull[8484]: changed: [quicknotes-vm] +Jun 30 14:10:06 quicknotes-vm ansible-pull[8484]: PLAY RECAP ********************************************************************* +Jun 30 14:10:06 quicknotes-vm ansible-pull[8484]: quicknotes-vm : ok=15 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 +Jun 30 14:10:06 quicknotes-vm ansible-pull[8484]: Starting Ansible Pull at 2026-06-30 14:09:56 +Jun 30 14:10:06 quicknotes-vm ansible-pull[8484]: /usr/bin/ansible-pull -U https://github.com/Hidancloud/DevOps-Intro.git -C feature/lab7 -d /var/lib/ansible-pull/quicknotes -i /etc/ansible/quicknotes-local.ini ansible/playbook.yaml +``` + +QuickNotes remained healthy after reconciliation: + +```text +{"notes":4,"status":"ok"} +``` + +### Design questions + +**h) What is the security benefit of `ansible-pull` vs push mode?** +In pull mode, the VM does not need to accept inbound SSH from a central control +node during each deploy. That shrinks the attack surface: no long-lived inbound +management path needs to stay open, and compromise of the control node does not +automatically grant shell access into every managed host. The node only needs +outbound access to fetch Git, then it reconciles locally. + +**i) What is the Kubernetes-layer equivalent, and why is this a fair simulator?** +The same pattern at the Kubernetes layer is commonly associated with **Argo CD** +or **Flux**: Git is the source of truth, and an in-cluster agent pulls desired +state and reconciles the live system to match. `ansible-pull` is a fair VM-layer +simulator because the control loop is the same even though the resource model is +different: commit to Git, agent detects new desired state, apply drift-correcting +changes locally, and repeat on a schedule. + +## Notes + +- The inventory assumes the standard Vagrant VM created from the repo root with + the Lab 5 `Vagrantfile` now restored into this branch. +- The Lab 5 `Vagrantfile` was updated to choose the correct guest Go tarball + for either `amd64` or `arm64`, because this MacBook is Apple Silicon. +- The binary is committed intentionally because the lab specification requires an + `ansible/files/quicknotes` payload that the playbook ships to the VM. diff --git a/submissions/lab8.md b/submissions/lab8.md new file mode 100644 index 000000000..259dbf353 --- /dev/null +++ b/submissions/lab8.md @@ -0,0 +1,526 @@ +# Lab 8 — SRE & Monitoring: Golden Signals Dashboard + One Good Alert + +## Objective + +Provision Prometheus and Grafana on top of the QuickNotes Compose stack, +auto-load a four-panel golden-signals dashboard, define one sustained high-error-rate alert, and write the matching runbook. + +## Environment + +| Component | Version / value | +|-----------|-----------------| +| Host OS | macOS (Apple Silicon) | +| Branch | `feature/lab8` | +| Compose stack | QuickNotes + Prometheus + Grafana | +| Prometheus image | `prom/prometheus:v3.5.0` | +| Grafana image | `grafana/grafana:13.0.3-slim` | + +> Note: the screenshot-oriented parts of the lab were validated through container and HTTP APIs in this environment. Where the spec asks for screenshots, this submission records the underlying API evidence and configuration instead. + +## Layout + +```text +monitoring/ +├── blackbox/ +│ └── blackbox.yml +├── prometheus/ +│ ├── alerts.yml +│ └── prometheus.yml +├── grafana/ +│ ├── dashboards/ +│ │ └── golden-signals.json +│ └── provisioning/ +│ ├── dashboards/ +│ │ └── dashboard.yml +│ └── datasources/ +│ └── datasource.yml +└── scripts/ + ├── checkhost-multi-region.sh + ├── generate-traffic.sh + └── trigger-high-error-rate.sh +``` + +## Task 1 — Prometheus + Grafana with a provisioned dashboard + +### Prometheus config + +```yaml +# monitoring/prometheus/prometheus.yml +global: + scrape_interval: 15s + +rule_files: + - /etc/prometheus/alerts.yml + +scrape_configs: + - job_name: blackbox-internal + metrics_path: /probe + params: + module: [http_2xx] + static_configs: + - targets: + - http://quicknotes:8080/health + relabel_configs: + - source_labels: [__address__] + target_label: __param_target + - source_labels: [__address__] + target_label: __address__ + replacement: blackbox-exporter:9115 + - source_labels: [__param_target] + target_label: instance + - target_label: job + replacement: blackbox-internal + + - job_name: quicknotes + static_configs: + - targets: + - quicknotes:8080 +``` + +### Grafana data source provisioning + +```yaml +# monitoring/grafana/provisioning/datasources/datasource.yml +apiVersion: 1 + +datasources: + - name: Prometheus + uid: prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true + editable: false +``` + +### Grafana dashboard provider + +```yaml +# monitoring/grafana/provisioning/dashboards/dashboard.yml +apiVersion: 1 + +providers: + - name: quicknotes-dashboards + orgId: 1 + folder: QuickNotes + type: file + disableDeletion: false + editable: false + options: + path: /var/lib/grafana/dashboards +``` + +### Dashboard JSON + +The provisioned dashboard lives at: +- `monitoring/grafana/dashboards/golden-signals.json` + +It contains four panels: +1. Latency proxy — `sum(rate(quicknotes_http_requests_total[1m]))` +2. Traffic — `sum(rate(quicknotes_http_requests_total[5m]))` +3. Errors — `100 * sum(rate(quicknotes_http_responses_by_code_total{code=~"4..|5.."}[5m])) / clamp_min(sum(rate(quicknotes_http_requests_total[5m])), 0.000001)` +4. Saturation — `quicknotes_notes_total` + +QuickNotes does not expose a request-duration histogram yet, so the Latency panel is explicitly labeled as a proxy until such a metric exists. + +### Compose extension + +```yaml +# compose.yaml +services: + quicknotes: + build: ./app + image: quicknotes:lab6 + ports: + - "8080:8080" + environment: + ADDR: ":8080" + DATA_PATH: "/data/notes.json" + SEED_PATH: "/seed.json" + volumes: + - quicknotes-data:/data + healthcheck: + test: ["CMD", "/healthcheck"] + interval: 10s + timeout: 3s + retries: 3 + start_period: 5s + restart: unless-stopped + + prometheus: + image: prom/prometheus:v3.5.0 + command: + - --config.file=/etc/prometheus/prometheus.yml + ports: + - "9090:9090" + volumes: + - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro + - ./monitoring/prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro + depends_on: + quicknotes: + condition: service_healthy + restart: unless-stopped + + blackbox-exporter: + image: prom/blackbox-exporter:v0.27.0 + command: + - --config.file=/etc/blackbox_exporter/blackbox.yml + ports: + - "9115:9115" + volumes: + - ./monitoring/blackbox/blackbox.yml:/etc/blackbox_exporter/blackbox.yml:ro + depends_on: + quicknotes: + condition: service_healthy + restart: unless-stopped + + grafana: + image: grafana/grafana:13.0.3-slim + environment: + GF_SECURITY_ADMIN_USER: quicknotes_admin + GF_SECURITY_ADMIN_PASSWORD: qn-lab8-monitoring + GF_USERS_ALLOW_SIGN_UP: "false" + ports: + - "3000:3000" + volumes: + - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro + - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro + depends_on: + - prometheus + restart: unless-stopped + +volumes: + quicknotes-data: +``` + +### Verification evidence + +Commands run: + +```bash +docker compose up -d --build +./monitoring/scripts/generate-traffic.sh http://localhost:8080 200 +curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health' +``` + +Live API evidence: + +```text +$ docker compose ps +NAME IMAGE COMMAND SERVICE STATUS +devops-intro-grafana-1 grafana/grafana:13.0.3-slim "/run.sh" grafana Up +devops-intro-prometheus-1 prom/prometheus:v3.5.0 "/bin/prometheus --..." prometheus Up +devops-intro-quicknotes-1 quicknotes:lab6 "/quicknotes" quicknotes Up (healthy) +``` + +```text +$ curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health' +"up" +``` + +```json +$ curl -u quicknotes_admin:qn-lab8-monitoring 'http://localhost:3000/api/search?query=golden' +[ + { + "uid": "quicknotes-golden-signals", + "title": "QuickNotes Golden Signals", + "type": "dash-db", + "url": "/d/quicknotes-golden-signals/quicknotes-golden-signals" + } +] +``` + +```json +$ curl -u quicknotes_admin:qn-lab8-monitoring http://localhost:3000/api/datasources/name/Prometheus +{ + "name": "Prometheus", + "type": "prometheus", + "url": "http://prometheus:9090", + "isDefault": true +} +``` + +Representative panel-query values after traffic generation: + +```json +$ curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=sum(rate(quicknotes_http_requests_total[1m]))' +[ + { + "value": [ + 1782829897.910, + "10.95249755577282" + ] + } +] +``` + +```json +$ curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=sum(rate(quicknotes_http_requests_total[5m]))' +[ + { + "value": [ + 1782830282.713, + "3.554535629570368" + ] + } +] +``` + +```json +$ curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=100 * sum(rate(quicknotes_http_responses_by_code_total{code=~\"4..|5..\"}[5m])) / clamp_min(sum(rate(quicknotes_http_requests_total[5m])), 0.000001)' +[ + { + "value": [ + 1782830282.720, + "47.581441263573545" + ] + } +] +``` + +```json +$ curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=quicknotes_notes_total' +[ + { + "metric": { + "__name__": "quicknotes_notes_total", + "instance": "quicknotes:8080", + "job": "quicknotes" + }, + "value": [ + 1782829897.980, + "204" + ] + } +] +``` + +### Design questions + +**a) Pull vs push.** +Prometheus must be able to reach QuickNotes over the network because the scrape model is pull-based. QuickNotes does not initiate metric delivery. If Prometheus cannot reach QuickNotes, the scrape target turns `up == 0`, Prometheus stops ingesting fresh samples, and dashboard panels either flatten at stale values or show no recent data. + +**b) Why not `5s` or `5m` scrape intervals?** +At `5s`, Prometheus stores far more samples, increases container and disk overhead, and makes noisy short-lived spikes dominate `rate()` queries that should represent service trends. At `5m`, you lose resolution so badly that short incidents disappear, alert evaluation becomes sluggish, and most panels look stair-stepped or empty over small time windows. + +**c) `rate()` vs `irate()` vs `delta()`.** +`rate()` is the right choice for the Traffic panel because it smooths a counter over a range and is stable for dashboards. `irate()` is too twitchy for a primary traffic graph because it only looks at the last two samples, while `delta()` is for absolute change over a range rather than per-second counter rates. + +**d) Why provision Grafana from files?** +Provisioning makes the dashboard and data source reproducible: a fresh `docker compose up` recreates the same monitoring state without manual clicking. It also means the dashboard lives in git, can be reviewed in PRs, and can be rebuilt on another machine or in CI exactly the same way. + +## Task 2 — One good alert + runbook + +### Alert rule + +```yaml +# monitoring/prometheus/alerts.yml +groups: + - name: quicknotes + rules: + - alert: QuickNotesHighErrorRate + expr: | + ( + sum(rate(quicknotes_http_responses_by_code_total{code=~"4..|5.."}[5m])) + / + clamp_min(sum(rate(quicknotes_http_requests_total[5m])), 0.000001) + ) > 0.05 + for: 5m + labels: + severity: page + annotations: + summary: "QuickNotes error ratio is above 5%" + description: "4xx + 5xx responses have exceeded 5% of total traffic for 5 minutes." + runbook: "docs/runbook/high-error-rate.md" +``` + +### Runbook + +```md +# docs/runbook/high-error-rate.md +# High Error Rate + +## What this alert means +QuickNotes is returning 4xx and 5xx responses for more than 5% of requests, sustained for at least 5 minutes. + +## Triage steps +1. Confirm the alert is still active by checking Prometheus `/alerts` and Grafana panels for traffic and error ratio over the last 15 minutes. +2. Compare healthy versus failing requests with `curl` against `/health`, `/notes`, and one deliberately malformed `POST /notes` to determine whether the issue is broad or isolated to a route. +3. Inspect the QuickNotes container logs with `docker compose logs --tail=200 quicknotes` and correlate timestamps with the error spike. +4. Check whether the backing data file or mounted volume is writable and present, because write failures can surface as 5xx on note creation. + +## Mitigations +1. Roll back the most recent change to the QuickNotes image or monitoring-related config if the spike started immediately after a deploy. +2. Temporarily reduce bad traffic by blocking malformed clients, rate-limiting the offending caller, or disabling the specific integration sending invalid payloads. +3. Restart the `quicknotes` container if the service is wedged but the root cause is still under investigation. + +## Post-incident +Write a short blameless postmortem using the Lecture 1 template: timeline, customer impact, root cause, contributing factors, and concrete follow-up actions. +``` + +### Trigger plan + +```bash +./monitoring/scripts/trigger-high-error-rate.sh http://localhost:8080 330 +curl http://localhost:9090/api/v1/alerts | jq '.data.alerts' +``` + +Observed state transition: + +```text +=== 2026-06-30 14:32:24 UTC === +{ + "state": "pending", + "activeAt": "2026-06-30T14:32:12.464142167Z", + "value": "6.352555131025643e-02", + "severity": "page", + "runbook": "docs/runbook/high-error-rate.md" +} +``` + +```text +=== 2026-06-30 14:37:24 UTC === +{ + "state": "firing", + "activeAt": "2026-06-30T14:32:12.464142167Z", + "value": "4.788478847884788e-01", + "severity": "page", + "runbook": "docs/runbook/high-error-rate.md" +} +``` + +Final alert API object: + +```json +$ curl http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.alertname=="QuickNotesHighErrorRate")' +{ + "labels": { + "alertname": "QuickNotesHighErrorRate", + "severity": "page" + }, + "annotations": { + "description": "4xx + 5xx responses have exceeded 5% of total traffic for 5 minutes.", + "runbook": "docs/runbook/high-error-rate.md", + "summary": "QuickNotes error ratio is above 5%" + }, + "state": "firing", + "activeAt": "2026-06-30T14:32:12.464142167Z", + "value": "4.788478847884788e-01" +} +``` + +### Design questions + +**e) Why sustained for 5 minutes?** +Because on-call alerts should represent real user pain, not single malformed requests or transient blips. A 5-minute sustain window filters noise and pages only when the error ratio is persistent enough to suggest an actual incident. + +**f) Symptom vs cause alerts.** +This alert is symptom-based because it tracks what users actually observe: error responses. A cause alert would be something like “CPU above 80%” or “container restarted once.” That is worse as a primary page because it may fire when users are fine, which trains on-call engineers to ignore alerts. + +**g) Alert fatigue threshold.** +If an alert pages roughly more than 20% of the time without meaningful user impact, it is too noisy and needs tuning. That is a practical threshold where responders start distrusting the signal instead of treating it as urgent. + +## Bonus Task — Synthetic Monitoring From the Outside + +### Public URL and probe setup + +For the external vantage point, I exposed the local QuickNotes instance through a +Cloudflare quick tunnel: + +```bash +docker run -d --name quicknotes-tunnel --network bridge \ + cloudflare/cloudflared:latest \ + tunnel --no-autoupdate --url http://host.docker.internal:8080 +``` + +Tunnel URL used for the run: + +```text +https://intensity-biological-beverages-perfectly.trycloudflare.com/health +``` + +Instead of Checkly, I used the equivalent free synthetic service +`api.check-host.cc`, which satisfies the lab's "or equivalent" clause. The +collector script `monitoring/scripts/checkhost-multi-region.sh` issued one HTTP +check per minute, treated non-`200` responses as failures, and summarized three +regions: + +- Germany: `DE-FFM-KRK` (Frankfurt am Main) +- Singapore: `SG-SIN-FDCServers` +- United States: `US-DAL-Leora` (Dallas) + +Run window: + +```text +start: 2026-06-30T15:01:43Z +end: 2026-06-30T15:35:35Z +span: 2032 s (~33.9 min) +cadence: 60 s +``` + +### External summary + +```json +$ cat monitoring/results/checkhost/summary.json +{ + "overall": { + "checks": 93, + "errors": 0, + "p50_seconds": 0.428, + "p95_seconds": 1.284 + }, + "nodes": { + "DE": { "p50_seconds": 0.389, "p95_seconds": 0.619, "errors": 0 }, + "SG": { "p50_seconds": 0.99, "p95_seconds": 1.549, "errors": 0 }, + "US": { "p50_seconds": 0.418, "p95_seconds": 0.478, "errors": 0 } + } +} +``` + +All three external regions stayed below the lab's `2 s` latency threshold at +the `p95` level, and none returned a non-`200` status during the measured +window. + +### Internal blackbox summary over the same window + +The Prometheus-side comparison used the internal blackbox probe on +`http://quicknotes:8080/health` and queried the same time window +(`2026-06-30T15:01:43Z` to `2026-06-30T15:35:35Z`) with a `15s` step. + +```json +{ + "samples": 136, + "p50_seconds": 0.002150833, + "p95_seconds": 0.002642417, + "errors": 0 +} +``` + +### Internal vs external comparison + +| Metric | Prometheus (inside the Compose net) | External synthetic (DE + SG + US) | +|--------|-------------------------------------|-----------------------------------| +| Avg latency p50 | `0.0022 s` | `0.428 s` | +| Avg latency p95 | `0.0026 s` | `1.284 s` | +| Errors observed | `0 / 136` | `0 / 93` | + +### Failure-mode analysis + +The external synthetic probe would catch failures in the public path that the +internal Prometheus probe cannot see: Cloudflare tunnel issues, DNS mistakes, +edge routing problems, TLS handshake failures, or an ISP/regional reachability +issue between users and the edge. That is why the external p50/p95 values are +two orders of magnitude slower than the internal ones even though both saw zero +errors: the external path includes Internet and edge overhead, while the +internal probe only measures an in-network container-to-container request. On +the other hand, Prometheus can catch failures that stay inside the cluster or +service mesh boundary, such as a metrics endpoint disappearing, an internal-only +health endpoint regressing, or fast transient failures sampled every 15 seconds +that a once-per-minute synthetic probe could miss. In practice, the two signals +are complementary: internal probing is sharper for service behavior, while +external probing is the better "can a real user reach me?" detector. + +## Notes + +- The bonus collector was hardened during the run to tolerate a transient + `api.check-host.cc` `502` without aborting the full measurement window.