-
Notifications
You must be signed in to change notification settings - Fork 2
docs: restructure deployment guide #310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
bef6f46
8f30ec0
7ee8f74
ada4e1a
c141369
1fb7d83
8a4954c
e715a4f
2afc027
bdb6362
92c3782
a2164b5
89fbfdd
921f06c
1f6dbc5
d3e1688
97bc638
0cd8afd
efd95f8
dbff18f
93f04c8
258acf8
f629c66
1d2d66b
a000b1e
d3cc508
32bf9ce
550275d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| --- | ||
| slug: /deployment-guide | ||
| title: Guide | ||
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| # Deployment Guide | ||
|
|
||
| metal-stack is a Metal-as-a-Service (MaaS) platform that turns on-premises bare-metal servers into an elastic, self-managed cloud infrastructure. At its core, metal-stack consists of two components: | ||
|
|
||
| - **Control Plane** — The central management layer (APIs, databases, scheduling) | ||
| - **Partitions** — The data center infrastructure (servers, leaf switches, BMCs) | ||
|
|
||
| These two components alone provide a fully functional **Bare-Metal as a Service (MaaS)** platform. You can allocate machines, manage networks, configure firewalls, and operate servers via REST/gRPC APIs and the `metalctl` CLI — without any Kubernetes cluster lifecycle management. | ||
|
|
||
| ## Kubernetes Cluster Lifecycle Management | ||
|
|
||
| If you need to provision **Kubernetes clusters** on top of your bare-metal infrastructure, metal-stack integrates with KCLM solutions like [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) or [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md). These are **optional layers** built on top of the MaaS foundation — not prerequisites. | ||
|
|
||
| :::tip | ||
| You can use metal-stack as a pure MaaS platform without any KCLM integration. The KCLM layer is only needed if you want to automate Kubernetes cluster provisioning on your bare-metal infrastructure. | ||
| ::: | ||
|
|
||
| ### Deployment Approach | ||
|
|
||
| We bootstrap the [metal control-plane](../../05-Concepts/01-architecture.mdx#metal-control-plane) as well as our [partitions](../../05-Concepts/01-architecture.mdx#partitions) with [Ansible](https://www.ansible.com/) through CI. | ||
|
|
||
| The deployment has multiple phases, resulting in a Git repository containing Ansible playbooks, inventory files and CI/CD workflows. | ||
|
|
||
| 1. **[Control Plane](./03_control-plane.mdx)** — Deploy the metal-stack API, databases, and ingress on a bootstrap Kubernetes cluster. This alone gives you a functional MaaS platform. | ||
| 2. **[Partition](./04_partition.md)** — Configure the network fabric (leaf switches, management spine, DHCP, PXE) and connect your bare-metal servers to the control plane. | ||
| 3. **[KCLM with Gardener](./05_kclm.md)** — Add Gardener to manage Kubernetes cluster lifecycle on your bare-metal infrastructure. | ||
|
|
||
| By the end of this guide, your deployment repository will look something like this: | ||
|
|
||
| ```text | ||
| my-metal-stack-deployment/ | ||
| ├── deploy_metal_control_plane.yaml # Control plane | ||
| ├── deploy_partition.yaml # Partition | ||
| ├── deploy_gardener.yaml # KCLM | ||
| ├── inventory/ | ||
| │ ├── inventory.yaml | ||
| │ ├── group_vars/ # Variables per host group | ||
| │ └── host_vars/ # Variables per host | ||
| ├── roles/ # Custom Ansible roles | ||
| ├── files/ | ||
| └── .github/workflows/ # CI/CD pipelines | ||
| ``` | ||
|
|
||
| The different phases of the deployment guide show which files to add and how they fit into this structure. | ||
|
|
||
| :::tip | ||
| You can use the [mini-lab](https://github.com/metal-stack/mini-lab) as a template project for your own deployment. It uses the same approach as described in this guide. | ||
| ::: | ||
|
|
||
| We recommend using the same Ansible roles that we use to deploy metal-stack in our own environments. They are available in the [metal-roles](https://github.com/metal-stack/metal-roles) repository. | ||
|
|
||
| To simplify dependency management, we provide a dedicated [deployment base image](https://github.com/metal-stack/metal-deployment-base) on GitHub Container Registry. This Docker image bundles all required tools and libraries, keeping your deployment environment lean and easy to maintain. | ||
|
|
||
| The remainder of this guide assumes you are using our Ansible roles and the deployment base image. Make sure you have [Docker](https://docs.docker.com/get-started/get-docker/) installed. Every deployment environment is unique, so this guide describes how we set up metal-stack in our own infrastructure. You will need to adapt the steps to match your specific requirements. | ||
|
|
||
| :::warning | ||
| You will likely need to learn to write Ansible playbooks if you want to be able to deploy the metal-stack as presented in this documentation. However, even when starting without any knowledge of Ansible it should be possible to follow these docs. In case you need further explanations regarding Ansible please refer to [docs.ansible.com](https://docs.ansible.com/). | ||
| ::: | ||
|
|
||
| :::info | ||
| If you do not want to use Ansible for deployment, you need to come up with a deployment mechanism by yourself. You can still learn from our [metal-roles](https://github.com/metal-stack/metal-roles) repository and [Helm charts](https://github.com/metal-stack/helm-charts/) — but be aware that the Ansible roles tie everything together (variable management, dependency ordering, environment-specific configurations), so building an equivalent from scratch will require significant effort. | ||
| ::: | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| --- | ||
| slug: /deployment/bootstrap-infrastructure | ||
| title: Bootstrap Infrastructure | ||
| sidebar_position: 2 | ||
| --- | ||
|
|
||
| # Bootstrap Infrastructure | ||
|
|
||
| Every metal-stack deployment starts with one or more **initial clusters** — Kubernetes clusters that host the [metal control-plane](../../05-Concepts/01-architecture.mdx#metal-control-plane). Collectively, these form the **bootstrap infrastructure** for your metal-stack platform. | ||
|
|
||
| The initial cluster(s) serve two purposes: | ||
|
|
||
| 1. **Host the metal-stack control plane** — the core platform components for bare-metal management. | ||
| 2. **Enable Kubernetes Cluster Lifecycle Management (KCLM)** — if you need to provision and manage downstream Kubernetes clusters on your bare-metal machines. | ||
|
|
||
| The number and placement of initial clusters depends on your KCLM choice, availability requirements, and autonomy needs. | ||
|
|
||
| If you only need **Bare-Metal as a Service** without KCLM, you need at least **one initial cluster** for the [Control Plane](./03_control-plane.mdx). | ||
|
|
||
| :::tip | ||
| Your control plane Kubernetes cluster can run anywhere — on a hyperscaler, in your own data center, or on [metalstack.cloud](https://metalstack.cloud). A managed cluster removes the operational burden of running Kubernetes yourself and can even strengthen fail-safe operation. Learn more about the [rationale for this approach](../../05-Concepts/01-architecture.mdx#target-deployment-platforms) and find concrete hosting suggestions below. | ||
| ::: | ||
|
|
||
| ## KCLM Solutions | ||
|
|
||
| metal-stack supports three Kubernetes Cluster Lifecycle Management solutions, each with different maturity levels and capabilities. | ||
| See the [Kubernetes Concepts Section](../../05-Concepts/04-Kubernetes/01-gardener.md) for a detailed comparison. | ||
|
|
||
| ### Gardener | ||
|
|
||
| [Gardener](../../05-Concepts/04-Kubernetes/01-gardener.md) is the **recommended** path for Kubernetes cluster lifecycle management. It is battle-tested in production for over seven years at financial-sector customers and bundles several day-2 capabilities natively (DNS, backup, audit). Gardener manages entire clusters as Kubernetes-native resources with a strong separation between platform operators and end-users. | ||
|
|
||
| :::tip | ||
| Gardener is the recommended solution for production environments. See the [Gardener concept section](../../05-Concepts/04-Kubernetes/01-gardener.md) for terminology and architecture details. | ||
| ::: | ||
|
|
||
| ### Cluster-API | ||
|
|
||
| [Cluster-API](../../05-Concepts/04-Kubernetes/02-cluster-api.md) is a CNCF project maintained by a Kubernetes SIG that provides declarative cluster management through a management cluster. The metal-stack provider (CAPMS) is **under development** and not yet production-ready. | ||
|
|
||
| :::warning | ||
| Cluster-API with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. | ||
| ::: | ||
|
|
||
| #### Kamaji | ||
|
|
||
| [Kamaji](../../05-Concepts/04-Kubernetes/02-cluster-api.md#kamaji) allows a similar control plane hosting model as Gardener, where the control plane runs on dedicated infrastructure separate from worker nodes. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The kamaji section will follow later?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it is WIP. A concepts PR is on its way. As they are heavily intertwined, we'll have to merge both at the same time. |
||
| Kamaji therefore uses ClusterAPI to support different infrastructure- and control-plane providers. | ||
| However, Kamaji integrations with metal-stack **have not been evaluated in production-grade scenarios** by metal-stack. | ||
|
ma-hartma marked this conversation as resolved.
|
||
|
|
||
| :::warning | ||
| Kamaji with metal-stack is in development and not advised for production use. Please use Gardener for production workloads. | ||
| ::: | ||
|
|
||
| ## Deployment Options | ||
|
|
||
| There are three supported approaches for hosting the initial cluster: | ||
|
|
||
| ### Option 1: Shared Initial Cluster | ||
|
|
||
| It is possible to use a **single initial cluster** for both metal-stack and the KCLM solution. This approach is technically feasible but **not recommended** for production environments. Sharing a single cluster mixes platform infrastructure with lifecycle management, which can complicate operational boundaries and failure isolation. | ||
|
|
||
| ### Option 2: Dedicated Initial Clusters | ||
|
|
||
| We recommend using **dedicated initial clusters** for metal-stack and the KCLM solution — one cluster for the metal-stack control plane and a separate cluster for the KCLM. | ||
|
|
||
| This approach provides clearer operational boundaries, better isolation and simplified failure boundaries. | ||
|
|
||
| ### Option 3: Autonomous Control Plane | ||
|
|
||
| For self-hosted deployments, metal-stack can be set up with an [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) cluster. This approach is the best choice for organizations that require full digital sovereignty and autonomy over their entire infrastructure stack. | ||
|
|
||
| The autonomous control-plane cluster serves as a minimal control plane whose sole purpose is to host the production control plane cluster (the "Matryoshka principle"). | ||
| This brings several advantages like failure isolation, separate operational responsibility, minimal resource requirements and full control and ownership. | ||
|
|
||
| The only requirement from metal-stack is that your partitions can establish network connections to the metal control-plane. | ||
|
|
||
| ## Suggestions for the Initial Cluster | ||
|
|
||
| ### For Options 1 & 2: Cloud-Hosted Clusters | ||
|
|
||
| For the shared and dedicated cluster approaches, the initial cluster can be hosted anywhere — a hyperscaler, metalstack.cloud, or any other managed Kubernetes provider. Some common options: | ||
|
|
||
| - **metalstack.cloud** — A Kubernetes cluster can be created via [UI](https://metalstack.cloud/de/documentation/UserManual#creating-a-cluster), CLI, or [Terraform](https://github.com/metal-stack-cloud/terraform-provider-metal). | ||
| - **GCP/GKE** — A GCP account is required. The Ansible [gcp-auth role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-auth) can be used for authentication, and the [gcp-create role](https://github.com/metal-stack/ansible-common/tree/master/roles/gcp-create) for creating a GKE cluster. | ||
| - Suggested defaults: `gcp_machine_type`: e2-standard-8, `gcp_autoscaling_min_nodes`: 1, `gcp_autoscaling_max_nodes`: 3 | ||
|
ma-hartma marked this conversation as resolved.
|
||
|
|
||
| ### For Option 3: Autonomous Control Plane with k3s | ||
|
|
||
| For the autonomous control plane approach, [MEP-18](/community/MEP-18-autonomous-control-plane) proposes using [k3s](https://k3s.io/) as the initial cluster. This is because KCLM solutions are not yet able to create an initial cluster themselves (though this may change with implementations like [GEP-28](https://github.com/gardener/enhancements/blob/main/geps/0028-self-hosted-shoot-clusters/README.md) for Gardener). | ||
|
|
||
| The k3s nodes can be either bare metal machines or virtual machines. For a minimal setup, a single node with 8–16 cores, 64GB RAM, and two NVMe drives of 1TB is a good starting point. For high availability, a clustered k3s configuration across multiple nodes is recommended, with ETCD replication and backup-restore mechanisms configured for metal-stack and KCLM components. | ||
|
|
||
| See the [Autonomous Control Plane](/community/MEP-18-autonomous-control-plane) proposal for detailed architecture, failure scenarios, and implementation guidance. | ||
Uh oh!
There was an error while loading. Please reload this page.