mirror of
https://github.com/openshift/installer.git
synced 2026-02-05 15:47:14 +01:00
*: remove k8s-node-bootstrap.service
This was broken by 4f8f4fc when the origin image was used instead. The
k8s-node-bootstrap service was failing to parse "latest" as valid
semver. Since we are eventually moving away from Container Linux, this
service is safe to remove in favor of some future mechanism.
This commit is contained in:
Alex Crawford
@@ -7,32 +7,26 @@ This is a development document which describes the bootstrapping flow for Contai
|
||||
When a cluster node is being bootstrapped from scratch, it goes through several phases in the following order:
|
||||
|
||||
1. first-boot OS configuration, via ignition (systemd units, node configuration, etc)
|
||||
1. provisioning of additional assets (k8s manifests, TLS material), via either of:
|
||||
2. provisioning of additional assets (k8s manifests, TLS material), via either of:
|
||||
* pushing from terraform file/remote-exec (SSH)
|
||||
* pulling from private cloud stores (S3 buckets)
|
||||
1. system-wide updates via `k8s-node-bootstrap.service`, which includes:
|
||||
* determining current kubernetes cluster version (when joining an existing cluster)
|
||||
* triggering a ContainerLinux update, via update-engine (optional)
|
||||
* downloading and deploying proper docker addon version, via tectonic-torcx
|
||||
* writing the `kubelet.env` file
|
||||
1. if needed, a node reboot is triggered to apply systemd-wide changes and to clean container runtime datadir
|
||||
1. `kubelet.service` picks up the `kubelet.env` file and actually starts the kubelet as a rkt-fly service.
|
||||
3. if needed, a node reboot is triggered to apply systemd-wide changes and to clean container runtime datadir
|
||||
4. `kubelet.service` picks up the `kubelet.env` file and actually starts the kubelet as a rkt-fly service.
|
||||
|
||||
Additionally, only on one of the master nodes the following kubernetes bootstrapping happens:
|
||||
|
||||
1. `bootkube.service` is started after `kubelet.service` start
|
||||
1. a static bootstrapping control-plane is deployed
|
||||
1. a fully self-hosted control-plane starts and takes over the previous one
|
||||
1. `bootkube.service` is completed with success
|
||||
1. `tectonic.service` is started
|
||||
1. a self-hosted tectonic control-plane is deployed
|
||||
1. `tectonic.service` is completed with success
|
||||
2. a static bootstrapping control-plane is deployed
|
||||
3. a fully self-hosted control-plane starts and takes over the previous one
|
||||
4. `bootkube.service` is completed with success
|
||||
5. `tectonic.service` is started
|
||||
6. a self-hosted tectonic control-plane is deployed
|
||||
7. `tectonic.service` is completed with success
|
||||
|
||||
## Systemd units
|
||||
|
||||
The following systemd units are deployed to a node by tectonic-installer and take part in the bootstrapping process:
|
||||
The following systemd unit is deployed to a node by tectonic-installer and take part in the bootstrapping process:
|
||||
|
||||
* `k8s-node-bootstrap.service` ensures node and assets freshness. It is automatically started on boot, can crash-loop, and it runs only during bootstrap
|
||||
* `kubelet.service` is the main kubelet daemon. It is automatically started on boot, it is crash-looping until `kubelet.env` is populated, and it runs on each boot
|
||||
|
||||
Additionally, only on one of the master nodes the following kubernetes bootstrapping happens:
|
||||
@@ -42,35 +36,12 @@ Additionally, only on one of the master nodes the following kubernetes bootstrap
|
||||
* `tectonic.service` deploys tectonic control-plane. It is started only after `bootkube.service` _has completed_. It is a oneshot unit and cannot crash, and it runs only during bootstrap
|
||||
* `bootkube.path` waits for tectonic assets/scripts to exist on disk and triggers `tectonic.service`
|
||||
|
||||
`k8s-node-bootstrap` runs [tectonic-torcx][tectonic-torcx] as a containerized service, thus relying on a container runtime being already on the node.
|
||||
It currently assumes that Docker is available and working. In case of version changes, a cleanup of the Docker datadir `/var/lib/docker` is scheduled before rebooting.
|
||||
|
||||
[tectonic-torcx]: https://github.com/coreos/tectonic-torcx
|
||||
|
||||
## Service ordering
|
||||
|
||||
Service ordering is enforced via systemd dependencies. This is the rationale for the settings, with relevant snippets:
|
||||
|
||||
### `k8s-node-bootstrap.service`
|
||||
|
||||
```
|
||||
ConditionPathExists=!/etc/kubernetes/kubelet.env
|
||||
Before=kubelet.service
|
||||
Restart=on-failure
|
||||
ExecStartPre=[...]
|
||||
ExecStart=/usr/bin/echo "node components bootstrapped"
|
||||
WantedBy=multi-user.target kubelet.service
|
||||
```
|
||||
|
||||
This service is enabled by default and can crash-loop until success.
|
||||
Main logic happens in `Pre`, before the unit is marked as started, to block further services (a synchronous reboot can happen here).
|
||||
|
||||
In particular, this blocks kubelet from starting by:
|
||||
* a `WantedBy=` and `Before=`
|
||||
* writing the actual `kubelet.env` file on success.
|
||||
|
||||
It is skipped on further boots, as the condition-path exists.
|
||||
|
||||
### `kubelet.service`
|
||||
|
||||
```
|
||||
@@ -81,7 +52,6 @@ WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
This service is enabled by default and can crash-loop until success.
|
||||
On first boot, it is initially blocked by `k8s-node-bootstrap.service`.
|
||||
It crash-loop until the `kubelet.env` file exists.
|
||||
It is started on every boot.
|
||||
|
||||
@@ -135,7 +105,6 @@ This is a visual simplified representation of the overall bootstrapping flow.
|
||||
Legend:
|
||||
* TF -> terraform provisioner
|
||||
* IGN -> ignition
|
||||
* knb.s -> k8s-node-bootstrap.service
|
||||
* k.s -> kubelet.service
|
||||
* b.p -> bootkube.path
|
||||
* b.s -> bootkube.service
|
||||
@@ -144,28 +113,28 @@ Legend:
|
||||
* rm.p -> rm-assets.path
|
||||
* rm.s -> rm-assets.service
|
||||
|
||||
.---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| |
|
||||
| Provision cloud/userdata +----------+ Provision files |
|
||||
| ,----------------------------------------------o| TF |o-----------------.------------------------.-----------------+ |
|
||||
| | +----------+ | | | |
|
||||
| | v v v |
|
||||
| | +----------+ +-----+ +-------+ +------+ |
|
||||
| | .--->| (reboot) |----. | b.p | | t.p | | rm.p | |
|
||||
| | | +----------+ | +-----+ +-------+ +------+ |
|
||||
| V | | o o o |
|
||||
| +-------+ | v Before +------------+ Before | Trigger Trigger | Trigger | |
|
||||
| | IGN | | *---------->| k.s |o--------. | | | |
|
||||
| +-------+ o ^ +------------+ | v v v |
|
||||
| | +----------+ | ^ | | +-----+ Before +-------+ Before +-----+ |
|
||||
| '------>| knb.s |o--------------' | v '--->| b.s |o--------------->| t.s |--------> |rm.s | |
|
||||
| Enable +----------+ '------' +-----+ +-------+ +-----+ |
|
||||
| ^ | |
|
||||
| | v |
|
||||
| '----' o o |
|
||||
| | | |
|
||||
| * First boot | * Each boot | * First boot |
|
||||
| * All nodes | * All nodes | * Bootkube master |
|
||||
| | | |
|
||||
'----------------------------------------------o----------------------------o-----------------------------------------------------------+
|
||||
.--------------------------------------------------------------------------------------------------------------------------------+
|
||||
| |
|
||||
| Provision cloud/userdata +----------+ Provision files |
|
||||
| ,---------------------------------------o| TF |o-----------------.------------------------.-----------------+ |
|
||||
| | +----------+ | | | |
|
||||
| | v v v |
|
||||
| | +-----+ +-------+ +------+ |
|
||||
| | | b.p | | t.p | | rm.p | |
|
||||
| | +-----+ +-------+ +------+ |
|
||||
| V o o o |
|
||||
| +-------+ Before +------------+ Before | Trigger Trigger | Trigger | |
|
||||
| | IGN | .--------------->| k.s |o--------. | | | |
|
||||
| +-------+ | +------------+ | v v v |
|
||||
| | | ^ | | +-----+ Before +-------+ Before +-----+ |
|
||||
| '----------------------' | v '--->| b.s |o--------------->| t.s |--------> |rm.s | |
|
||||
| Enable '------' +-----+ +-------+ +-----+ |
|
||||
| |
|
||||
| |
|
||||
| o o |
|
||||
| | | |
|
||||
| | * Each boot | * First boot |
|
||||
| | * All nodes | * Bootkube master |
|
||||
| | | |
|
||||
'---------------------------------------o----------------------------o-----------------------------------------------------------+
|
||||
```
|
||||
|
||||
@@ -9,7 +9,6 @@ This document gives an overview of variables used in all platforms of the Tecton
|
||||
| tectonic_admin_email | (internal) The e-mail address used to: 1. login as the admin user to the Tectonic Console. 2. generate DNS zones for some providers.<br><br>Note: This field MUST be in all lower-case e-mail address format and set manually prior to creating the cluster. | string | - |
|
||||
| tectonic_admin_password | (internal) The admin user password to login to the Tectonic Console.<br><br>Note: This field MUST be set manually prior to creating the cluster. Backslashes and double quotes must also be escaped. | string | - |
|
||||
| tectonic_base_domain | The base DNS domain of the cluster. It must NOT contain a trailing period. Some DNS providers will automatically add this if necessary.<br><br>Example: `openstack.dev.coreos.systems`.<br><br>Note: This field MUST be set manually prior to creating the cluster. This applies only to cloud platforms.<br><br>[Azure-specific NOTE] To use Azure-provided DNS, `tectonic_base_domain` should be set to `""` If using DNS records, ensure that `tectonic_base_domain` is set to a properly configured external DNS zone. Instructions for configuring delegated domains for Azure DNS can be found here: https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns | string | - |
|
||||
| tectonic_bootstrap_upgrade_cl | (internal) Whether to trigger a ContainerLinux upgrade on node bootstrap. | string | `true` |
|
||||
| tectonic_ca_cert | (optional) The content of the PEM-encoded CA certificate, used to generate Tectonic Console's server certificate. If left blank, a CA certificate will be automatically generated. | string | `` |
|
||||
| tectonic_ca_key | (optional) The content of the PEM-encoded CA key, used to generate Tectonic Console's server certificate. This field is mandatory if `tectonic_ca_cert` is set. | string | `` |
|
||||
| tectonic_ca_key_alg | (optional) The algorithm used to generate tectonic_ca_key. The default value is currently recommended. This field is mandatory if `tectonic_ca_cert` is set. | string | `RSA` |
|
||||
|
||||
@@ -323,12 +323,6 @@ variable "tectonic_networking" {
|
||||
EOF
|
||||
}
|
||||
|
||||
variable "tectonic_bootstrap_upgrade_cl" {
|
||||
type = "string"
|
||||
default = "true"
|
||||
description = "(internal) Whether to trigger a ContainerLinux upgrade on node bootstrap."
|
||||
}
|
||||
|
||||
variable "tectonic_kubelet_debug_config" {
|
||||
type = "string"
|
||||
default = ""
|
||||
|
||||
@@ -49,23 +49,6 @@ data "ignition_systemd_unit" "kubelet" {
|
||||
content = "${data.template_file.kubelet.rendered}"
|
||||
}
|
||||
|
||||
data "template_file" "k8s_node_bootstrap" {
|
||||
template = "${file("${path.module}/resources/services/k8s-node-bootstrap.service")}"
|
||||
|
||||
vars {
|
||||
bootstrap_upgrade_cl = "${var.bootstrap_upgrade_cl}"
|
||||
tectonic_torcx_image = "${var.container_images["tectonic_torcx"]}"
|
||||
torcx_skip_setup = "false"
|
||||
torcx_store_url = "${var.torcx_store_url}"
|
||||
}
|
||||
}
|
||||
|
||||
data "ignition_systemd_unit" "k8s_node_bootstrap" {
|
||||
name = "k8s-node-bootstrap.service"
|
||||
enabled = true
|
||||
content = "${data.template_file.k8s_node_bootstrap.rendered}"
|
||||
}
|
||||
|
||||
data "ignition_systemd_unit" "rm_assets" {
|
||||
name = "rm-assets.service"
|
||||
enabled = true
|
||||
|
||||
@@ -24,10 +24,6 @@ variable "ign_installer_kubelet_env_id" {
|
||||
type = "string"
|
||||
}
|
||||
|
||||
variable "ign_k8s_node_bootstrap_service_id" {
|
||||
type = "string"
|
||||
}
|
||||
|
||||
variable "ign_update_ca_certificates_dropin_id" {
|
||||
type = "string"
|
||||
}
|
||||
|
||||
@@ -32,7 +32,6 @@ output "ignition_systemd_id_list" {
|
||||
"${data.ignition_systemd_unit.docker_dropin.id}",
|
||||
"${data.ignition_systemd_unit.kubelet.id}",
|
||||
"${data.ignition_systemd_unit.locksmithd.id}",
|
||||
"${data.ignition_systemd_unit.k8s_node_bootstrap.id}",
|
||||
"${data.ignition_systemd_unit.update_ca_certificates_dropin.id}",
|
||||
"${data.ignition_systemd_unit.rm_assets.id}",
|
||||
]
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
[Unit]
|
||||
Description=Bootstrap Kubernetes Node Components
|
||||
ConditionPathExists=!/etc/kubernetes/kubelet.env
|
||||
Wants=update-ca-certificates.service
|
||||
After=update-ca-certificates.service
|
||||
Before=kubelet.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
RemainAfterExit=true
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
TimeoutStartSec=1h
|
||||
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes
|
||||
ExecStartPre=/usr/bin/docker run --rm \
|
||||
--env http_proxy \
|
||||
--env HTTP_PROXY \
|
||||
--env https_proxy \
|
||||
--env HTTPS_PROXY \
|
||||
--env no_proxy \
|
||||
--env NO_PROXY \
|
||||
--tmpfs /tmp \
|
||||
-v /usr/share:/usr/share:ro \
|
||||
-v /usr/lib/os-release:/usr/lib/os-release:ro \
|
||||
-v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro \
|
||||
-v /var/lib/torcx:/var/lib/torcx \
|
||||
-v /var/run/dbus:/var/run/dbus \
|
||||
-v /run/metadata:/run/metadata:ro \
|
||||
-v /run/torcx:/run/torcx:ro \
|
||||
-v /run/systemd:/run/systemd \
|
||||
-v /etc/coreos:/etc/coreos:ro \
|
||||
-v /etc/torcx:/etc/torcx \
|
||||
-v /etc/kubernetes:/etc/kubernetes \
|
||||
${tectonic_torcx_image} \
|
||||
/tectonic-torcx-bootstrap \
|
||||
--upgrade-os=${bootstrap_upgrade_cl} \
|
||||
--torcx-manifest-url="${torcx_store_url}" \
|
||||
--torcx-skip-setup=${torcx_skip_setup} \
|
||||
--verbose=debug
|
||||
ExecStart=/usr/bin/echo "node components bootstrapped"
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target kubelet.service
|
||||
@@ -45,12 +45,6 @@ variable "cloud_provider_config" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "bootstrap_upgrade_cl" {
|
||||
type = "string"
|
||||
description = "(optional) Whether to trigger a ContainerLinux OS upgrade during the bootstrap process."
|
||||
default = "true"
|
||||
}
|
||||
|
||||
variable "torcx_store_url" {
|
||||
type = "string"
|
||||
description = "(optional) URL template for torcx store. Leave empty to use the default CoreOS endpoint."
|
||||
|
||||
@@ -21,7 +21,6 @@ module assets_base {
|
||||
ingress_kind = "haproxy-router"
|
||||
|
||||
tectonic_base_domain = "${var.tectonic_base_domain}"
|
||||
tectonic_bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}"
|
||||
tectonic_cluster_name = "${var.tectonic_cluster_name}"
|
||||
tectonic_container_images = "${var.tectonic_container_images}"
|
||||
tectonic_image_re = "${var.tectonic_image_re}"
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
module "ignition_bootstrap" {
|
||||
source = "../../../modules/ignition"
|
||||
|
||||
bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}"
|
||||
cloud_provider = "${var.cloud_provider}"
|
||||
container_images = "${var.tectonic_container_images}"
|
||||
etcd_ca_cert_pem = "${local.etcd_ca_cert_pem}"
|
||||
|
||||
@@ -9,7 +9,6 @@ module assets_base {
|
||||
ingress_kind = "haproxy-router"
|
||||
|
||||
tectonic_base_domain = "${var.tectonic_base_domain}"
|
||||
tectonic_bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}"
|
||||
tectonic_cluster_name = "${var.tectonic_cluster_name}"
|
||||
tectonic_container_images = "${var.tectonic_container_images}"
|
||||
tectonic_image_re = "${var.tectonic_image_re}"
|
||||
|
||||
Reference in New Issue
Block a user