mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
OSDOCS:11110 On Cluster Layering: Phase 3 (GA)
This commit is contained in:
Michael Burke
committed by
openshift-cherrypick-robot
openshift-cherrypick-robot
parent
625b51cd7f
commit
aa2984b0ad
@@ -30,11 +30,6 @@ As soon as you apply the custom layered image to your cluster, you effectively _
|
||||
There are two methods for deploying a custom layered image onto your nodes:
|
||||
|
||||
On-cluster layering:: With xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on_mco-coreos-layering[on-cluster layering], you create a `MachineOSConfig` object where you include the Containerfile and other parameters. The build is performed on your cluster and the resulting custom layered image is automatically pushed to your repository and applied to the machine config pool that you specified in the `MachineOSConfig` object. The entire process is performed completely within your cluster.
|
||||
+
|
||||
--
|
||||
:FeatureName: On-cluster image layering
|
||||
include::snippets/technology-preview.adoc[]
|
||||
--
|
||||
|
||||
Out-of-cluster layering:: With xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring_mco-coreos-layering[out-of-cluster layering], you create a Containerfile that references an {product-title} image and the RPM that you want to apply, build the layered image in your own environment, and push the image to your repository. Then, in your cluster, create a `MachineConfig` object for the targeted node pool that points to the new image. The Machine Config Operator overrides the base {op-system} image, as specified by the `osImageURL` value in the associated machine config, and boots the new image.
|
||||
|
||||
@@ -182,9 +177,11 @@ It is strongly recommended that you test your images outside of your production
|
||||
include::modules/coreos-layering-configuring-on.adoc[leveloffset=+1]
|
||||
|
||||
.Additional resources
|
||||
* xref:../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling[Enabling features using feature gates]
|
||||
* xref:../updating/updating_a_cluster/update-using-custom-machine-config-pools.adoc#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools[Pausing the machine config pools]
|
||||
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-remove_mco-coreos-layering[Removing an on-cluster custom layered image]
|
||||
* xref:../updating/updating_a_cluster/update-using-custom-machine-config-pools.adoc#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools[Pausing the machine config pools]
|
||||
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-rebuild_mco-coreos-layering[Rebuilding an on-cluster custom layered image]
|
||||
* xref:../openshift_images/managing_images/using-image-pull-secrets.adoc#images-update-global-pull-secret_using-image-pull-secrets[Updating the global cluster pull secret]
|
||||
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-modifying_mco-coreos-layering[Modifying a custom layered image]
|
||||
|
||||
include::modules/coreos-layering-configuring-on-modifying.adoc[leveloffset=+2]
|
||||
|
||||
@@ -200,8 +197,7 @@ include::modules/coreos-layering-configuring-on-extensions.adoc[leveloffset=+2]
|
||||
* xref:../updating/updating_a_cluster/update-using-custom-machine-config-pools.adoc#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools[Pausing the machine config pools]
|
||||
////
|
||||
|
||||
// Not in 4.18; maybe 4.19
|
||||
// include::modules/coreos-layering-configuring-on-rebuild.adoc[leveloffset=+2]
|
||||
include::modules/coreos-layering-configuring-on-rebuild.adoc[leveloffset=+2]
|
||||
|
||||
include::modules/coreos-layering-configuring-on-revert.adoc[leveloffset=+2]
|
||||
|
||||
|
||||
@@ -24,42 +24,37 @@ include::snippets//coreos-layering-configuring-on-pause.adoc[]
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1alpha1
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineOSConfig
|
||||
metadata:
|
||||
name: layered
|
||||
name: layered-image
|
||||
spec:
|
||||
machineConfigPool:
|
||||
name: worker
|
||||
buildInputs:
|
||||
containerFile:
|
||||
- containerfileArch: noarch
|
||||
content: |- <1>
|
||||
FROM configs AS final
|
||||
containerFile:
|
||||
- containerfileArch: noarch
|
||||
content: |- <1>
|
||||
FROM configs AS final
|
||||
|
||||
RUN rpm-ostree install rng-tools && \
|
||||
systemctl enable rngd && \
|
||||
rpm-ostree cleanup -m && \
|
||||
ostree container commit
|
||||
RUN rpm-ostree install rng-tools && \
|
||||
systemctl enable rngd && \
|
||||
rpm-ostree cleanup -m && \
|
||||
ostree container commit
|
||||
|
||||
RUN rpm-ostree install tree && \
|
||||
ostree container commit
|
||||
imageBuilder:
|
||||
imageBuilderType: PodImageBuilder
|
||||
baseImagePullSecret:
|
||||
name: global-pull-secret-copy <2>
|
||||
renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:latest <3>
|
||||
renderedImagePushSecret: <4>
|
||||
name: new-secret-name
|
||||
buildOutputs:
|
||||
currentImagePullSecret:
|
||||
name: new-secret-name <5>
|
||||
RUN rpm-ostree install tree && \
|
||||
ostree container commit
|
||||
imageBuilder:
|
||||
imageBuilderType: PodImageBuilder
|
||||
baseImagePullSecret:
|
||||
name: global-pull-secret-copy <2>
|
||||
renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:latest <3>
|
||||
renderedImagePushSecret: <4>
|
||||
name: new-secret-name
|
||||
----
|
||||
<1> Optional: Modify the Containerfile, for example to add or remove packages.
|
||||
<2> Optional: Update the secret needed to pull the base operating system image from the registry.
|
||||
<3> Optional: Modify the image registry to push the newly built custom layered image to.
|
||||
<4> Optional: Update the secret needed to push the newly built custom layered image to the registry.
|
||||
<5> Optional: Update the secret needed to pull the newly built custom layered image from the registry.
|
||||
+
|
||||
When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the cluster base {op-system-first} image. If your changes modify a secret only, no new build is triggered and no reboot is performed.
|
||||
|
||||
@@ -75,9 +70,9 @@ $ oc get machineosbuild
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED
|
||||
layered-a5457b883f5239cdcb71b57e1a30b6ef False False True False False
|
||||
layered-f91f0f5593dd337d89bf4d38c877590b False True False False False <1>
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE
|
||||
layered-image-a5457b883f5239cdcb71b57e1a30b6ef False False True False False 4d17h
|
||||
layered-image-f91f0f5593dd337d89bf4d38c877590b False True False False False 2m41s <1>
|
||||
----
|
||||
<1> The value `True` in the `BUILDING` column indicates that the `MachineOSBuild` object is building. When the `SUCCEEDED` column reports `True`, the build is complete.
|
||||
|
||||
|
||||
78
modules/coreos-layering-configuring-on-rebuild.adoc
Normal file
78
modules/coreos-layering-configuring-on-rebuild.adoc
Normal file
@@ -0,0 +1,78 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * machine_configuration/coreos-layering.adoc
|
||||
|
||||
:_mod-docs-content-type: PROCEDURE
|
||||
[id="coreos-layering-configuring-on-rebuild_{context}"]
|
||||
= Rebuilding an on-cluster custom layered image
|
||||
|
||||
In situations where you want to rebuild a custom layered image, you can either modify your `MachineOSConfig` object or add an annotation to the `MachineOSConfig` object. Both of these actions trigger an automatic rebuild of the object. For example, you could perform a rebuild if the you change the Containerfile or need to update the `osimageurl` location in a machine config.
|
||||
|
||||
After you add the annotation, the Machine Config Operator (MCO) deletes the current `MachineOSBuild` object and creates a new one in its place. When the build process is complete, the MCO automatically removes the annotation.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You have opted-in to on-cluster layering by creating a `MachineOSConfig` object.
|
||||
|
||||
.Procedure
|
||||
|
||||
* Edit the `MachineOSConfig` object to add the `machineconfiguration.openshift.io/rebuild` annotation by using the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc edit MachineOSConfig <object_name>
|
||||
----
|
||||
+
|
||||
.Example `MachineOSConfig` object
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineOSConfig
|
||||
metadata:
|
||||
annotations:
|
||||
machineconfiguration.openshift.io/current-machine-os-build: layering-c26d4a003432df70ee66c83981144cfa
|
||||
machineconfiguration.openshift.io/rebuild: "" <1>
|
||||
# ...
|
||||
name: layered-image
|
||||
# ...
|
||||
----
|
||||
<1> Add this annotation to trigger a rebuild of the custom layered image.
|
||||
|
||||
.Verification
|
||||
|
||||
* Check that the `MachineOSBuild` object is building by using the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc get machineosbuild
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE
|
||||
layered-image-d6b929a29c6dbfa8e4007c8069a2fd08 False True False False False 2m41s <1>
|
||||
----
|
||||
<1> The value `True` in the `BUILDING` column indicates that the `MachineOSBuild` object is building.
|
||||
|
||||
* Edit the `MachineOSConfig` object to verify that the MCO removed the `machineconfiguration.openshift.io/rebuild` annotation by using the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc edit MachineOSConfig <object_name>
|
||||
----
|
||||
+
|
||||
.Example `MachineOSConfig` object
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineOSConfig
|
||||
metadata:
|
||||
annotations:
|
||||
machineconfiguration.openshift.io/current-machine-os-build: layering-c26d4a003432df70ee66c83981144cfa
|
||||
# ...
|
||||
name: layered-image
|
||||
# ...
|
||||
----
|
||||
@@ -6,18 +6,19 @@
|
||||
[id="coreos-layering-configuring-on-revert_{context}"]
|
||||
= Reverting an on-cluster custom layered image
|
||||
|
||||
You can revert an on-cluster custom layered image from nodes by removing the label for the machine config pool (MCP) that you specified in the `MachineOSConfig` object. After you remove the label, the Machine Config Operator (MCO) reboots the nodes in that MCP with the cluster base {op-system-first} image, along with any previously-made machine config changes, overriding the custom layered image.
|
||||
If you applied an on-cluster layered image to a node in a custom machine config pool (MCP), you can remove the custom layered image from the node and revert to the base image.
|
||||
|
||||
To revert the node, remove the node from the custom MCP by removing the custom machine config pool label from the node. After you remove the label, the Machine Config Operator (MCO) reboots the node with the cluster base {op-system-first} image, overriding the custom layered image.
|
||||
|
||||
[IMPORTANT]
|
||||
====
|
||||
If the node where the custom layered image is deployed uses a custom machine config pool, before you remove the label, make sure the node is associated with a second MCP.
|
||||
Before you remove the label, make sure the node is associated with another MCP.
|
||||
====
|
||||
|
||||
You can reapply the custom layered image to the node by using the `oc label node/<node_name> 'node-role.kubernetes.io/<mcp_name>='` label.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You have opted in to on-cluster layering by creating a `MachineOSConfig` object.
|
||||
* You have opted-in to on-cluster layering by creating a `MachineOSConfig` object.
|
||||
* You have applied a `MachineOSConfig` object to a node in a custom machine config pool.
|
||||
|
||||
.Procedure
|
||||
|
||||
@@ -32,9 +33,9 @@ When you save the changes, the MCO drains, cordons, and reboots the nodes. After
|
||||
|
||||
.Verification
|
||||
|
||||
You can verify that the custom layered image is removed by performing the following checks:
|
||||
* Verify that the custom layered image is removed by performing any of the following checks:
|
||||
|
||||
. Check that the worker machine config pool is updating with the previous machine config:
|
||||
** Check that the worker machine config pool is updating with the previous machine config:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
@@ -45,12 +46,14 @@ $ oc get mcp
|
||||
[source,terminal]
|
||||
----
|
||||
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
|
||||
master rendered-master-8332482204e0b76002f15ecad15b6c2d True False False 3 3 3 0 5h26m
|
||||
worker rendered-worker-bde4e4206442c0a48b1a1fb35ba56e85 False True False 3 2 2 0 5h26m <1>
|
||||
layered rendered-layered-e8c8bc1de69777325003e80bc0c04b82 True False False 0 0 0 0 4h20m <1>
|
||||
master rendered-master-50d7bc27ee8b9ca2250383f0647ade7f True False False 3 3 3 0 5h39m
|
||||
worker rendered-worker-e8c8bc1de69777325003e80bc0c04b82 True False False 3 3 3 0 5h39m <2>
|
||||
----
|
||||
<1> The value `FALSE` in the `UPDATED` column indicates that the `MachineOSBuild` object is building. When the `UPDATED` column reports `FALSE`, the base image has rolled out to the nodes.
|
||||
<1> The custom machine config pool no longer has any nodes.
|
||||
<2> When the `UPDATING` field is `True`, the machine config pool is updating with the previous machine config. When the field becomes `False`, the worker machine config pool has rolled out to the previous machine config.
|
||||
|
||||
. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied:
|
||||
** Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
@@ -69,36 +72,35 @@ ip-10-0-211-49.us-west-1.compute.internal Ready control-
|
||||
ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.32.3
|
||||
----
|
||||
|
||||
. When the node is back in the `Ready` state, check that the node is using the base image:
|
||||
|
||||
.. Open an `oc debug` session to the node. For example:
|
||||
** When the node is back in the `Ready` state, check that the node is using the base image:
|
||||
+
|
||||
. Open an `oc debug` session to the node. For example:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc debug node/<node_name>
|
||||
$ oc debug node/ip-10-0-155-125.us-west-1.compute.internal
|
||||
----
|
||||
|
||||
.. Set `/host` as the root directory within the debug shell:
|
||||
+
|
||||
. Set `/host` as the root directory within the debug shell:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
sh-5.1# chroot /host
|
||||
sh-4.4# chroot /host
|
||||
----
|
||||
|
||||
.. Run an `rpm-ostree status` command to view that the base image is in use:
|
||||
. Run the `rpm-ostree status` command to view that the base image is in use:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
sh-5.1# rpm-ostree status
|
||||
sh-4.4# rpm-ostree status
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
State: idle
|
||||
Deployments:
|
||||
* ostree-unverified-image:containers-storage:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76721c875a2b79688be46b1dca654c2c6619a6be28b29a2822cd86c3f9d8e3c1
|
||||
Digest: sha256:76721c875a2b79688be46b1dca654c2c6619a6be28b29a2822cd86c3f9d8e3c1
|
||||
Version: 418.94.202501300706-0 (2025-01-30T07:10:58Z)
|
||||
* ostree-unverified-registry:registry.build05.ci.openshift.org/ci-ln-qd0hmqk/stable@sha256:a8bd32573f787f6d1c23e1d669abbefd1e31339826d06e750c0ca632ad6c414f
|
||||
Digest: sha256:a8bd32573f787f6d1c23e1d669abbefd1e31339826d06e750c0ca632ad6c414f
|
||||
Version: 419.96.202501202201-0 (2025-01-20T22:06:13Z)
|
||||
----
|
||||
|
||||
@@ -6,69 +6,86 @@
|
||||
[id="coreos-layering-configuring-on_{context}"]
|
||||
= Using on-cluster layering to apply a custom layered image
|
||||
|
||||
To apply a custom layered image to your cluster by using the on-cluster build process, make a `MachineOSConfig` custom resource (CR) that specifies the following parameters:
|
||||
To apply a custom layered image to your cluster by using the on-cluster build process, create a `MachineOSConfig` custom resource (CR) that specifies the following parameters:
|
||||
|
||||
* the Containerfile to build
|
||||
* the machine config pool to associate the build
|
||||
* where the final image should be pushed and pulled from
|
||||
* the push and pull secrets to use
|
||||
|
||||
When you create the object, the Machine Config Operator (MCO) creates a `MachineOSBuild` object and a builder pod. The build process also creates transient objects, such as config maps, which are cleaned up after the build is complete. The `MachineOSBuild` object and the associated `builder-*` pod use the same naming scheme, `<MachineOSConfig_CR_name>-<hash>`, for example:
|
||||
When you create the object, the Machine Config Operator (MCO) creates a `MachineOSBuild` object and a builder pod. The process also creates transient objects, such as config maps, which are cleaned up after the build is complete. The `MachineOSBuild` object and the associated `builder-*` pod use the same naming scheme, `<MachineOSConfig_CR_name>-<hash>`, for example:
|
||||
|
||||
.Example `MachineOSBuild` object
|
||||
[source,terminal]
|
||||
----
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED
|
||||
layered-c8765e26ebc87e1e17a7d6e0a78e8bae False False True False False
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED
|
||||
layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae False False True False False
|
||||
----
|
||||
|
||||
.Example builder pod
|
||||
[source,terminal]
|
||||
----
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
build-layered-c8765e26ebc87e1e17a7d6e0a78e8bae 2/2 Running 0 11m
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
build-layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae 2/2 Running 0 11m
|
||||
----
|
||||
|
||||
When the build is complete, the MCO pushes the new custom layered image to your repository and rolled out to the nodes in the associated machine config pool. You can see the digested image pull spec for the new custom layered image in the `MachineOSBuild` object and `machine-os-builder` pod.
|
||||
You should not need to interact with these new objects or the `machine-os-builder` pod. However, you can use all of these resources for troubleshooting, if necessary.
|
||||
|
||||
When the build is complete, the MCO pushes the new custom layered image to your repository and rolls the image out to the nodes in the associated machine config pool. You can see the digested image pull spec for the new custom layered image in the `MachineOSConfig` object. This is now the active image pull spec for this `MachineOSConfig`.
|
||||
|
||||
.Example digested image pull spec
|
||||
[source,terminal]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineOSConfig
|
||||
metadata:
|
||||
annotations:
|
||||
machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45
|
||||
labels:
|
||||
machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: ""
|
||||
name: layered-image
|
||||
# ...
|
||||
status:
|
||||
currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
|
||||
----
|
||||
|
||||
[TIP]
|
||||
====
|
||||
You can test a `MachineOSBuild` object to make sure it builds correctly without rolling out the custom layered image to active nodes by using a custom machine config pool that contains non-production nodes. Alternatively, you can use a custom machine config pool that has no nodes. The `MachineOSBuild` object builds even if there are no nodes for the MCO to deploy the custom layered image onto.
|
||||
====
|
||||
|
||||
You should not need to interact with these new objects or the `machine-os-builder` pod. However, you can use all of these resources for troubleshooting, if necessary.
|
||||
|
||||
You need a separate `MachineOSConfig` CR for each machine config pool where you want to use a custom layered image.
|
||||
You can apply a custom layered image to any machine config pool in your cluster, including the control plane, worker, or custom pools.
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
For {sno} clusters, you can apply a custom layered image to the control plane node only.
|
||||
====
|
||||
|
||||
:FeatureName: On-cluster image layering
|
||||
include::snippets/technology-preview.adoc[]
|
||||
include::snippets//coreos-layering-configuring-on-pause.adoc[]
|
||||
|
||||
In the case of a build failure, for example due to network issues or an invalid secret, the MCO retries the build three additional times before the job fails. The MCO creates a different build pod for each build attempt. You can use the build pod logs to troubleshoot any build failures. However, the MCO automatically removes these build pods after a short period of time.
|
||||
In the case of a build failure, for example due to network issues or an invalid secret, the MCO retries the build three additional times before the job fails. The MCO creates a different build pod for each build attempt. You can use the build pod logs to troubleshoot any build failures. Note that the MCO automatically removes these build pods after a short period of time.
|
||||
|
||||
.Example failed `MachineOSBuild` object
|
||||
[source,terminal]
|
||||
----
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED
|
||||
layered-c8765e26ebc87e1e17a7d6e0a78e8bae False False False False True
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE
|
||||
layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae False False False False True 12m
|
||||
----
|
||||
|
||||
// Not in 4.18; maybe 4.19
|
||||
// You can manually rebuild your custom layered image by either modifying your `MachineOSConfig` object or applying an annotation to the `MachineOSConfig` object as discussed in the following section.
|
||||
You can manually rebuild your custom layered image by either modifying your `MachineOSConfig` object or applying an annotation to the `MachineOSConfig` object. For more information, see "Rebuilding an on-cluster custom layered image".
|
||||
|
||||
If you used a custom machine config pool to apply an on-cluster layered image to a node, you can remove the custom layered image from the node and revert to the base image. For more information, see "Reverting an on-cluster layered node".
|
||||
|
||||
You can modify an on-custom layered image as needed, to install additional packages, remove existing packages, change repositories, update secrets, or other similar changes, by editing the MachineOSConfig object. For more information, see "Modifying a custom layered image".
|
||||
|
||||
[discrete]
|
||||
[id="coreos-layering-configuring-on-limitations_{context}"]
|
||||
== On-cluster layering Technology Preview known limitations
|
||||
== On-cluster layering known limitations
|
||||
|
||||
Note the following limitations when working with the on-cluster layering feature:
|
||||
|
||||
* On-cluster layering is supported only for {product-title} clusters on the AMD64 architecture.
|
||||
* On-cluster layering is not supported on multi-architecture compute machines.
|
||||
* Using multiple `MachineOSConfig` objects on the same machine config pool is not supported. You need a separate `MachineOSConfig` CR for each machine config pool where you want to use a distinct custom layered image.
|
||||
* If you scale up a machine set that uses a custom layered image, the nodes reboot two times. The first, when the node is initially created with the base image and a second time when the custom layered image is applied.
|
||||
* Node disruption policies are not supported on nodes with a custom layered image. As a result the following configuration changes cause a node reboot:
|
||||
** Modifying the configuration files in the `/var` or `/etc` directory
|
||||
@@ -80,33 +97,16 @@ Note the following limitations when working with the on-cluster layering feature
|
||||
|
||||
.Prerequisites
|
||||
|
||||
* You have enabled the `TechPreviewNoUpgrade` feature set by using the feature gates. For more information, see "Enabling features using feature gates".
|
||||
* You have the pull secret in the `openshift-machine-config-operator` namespace that the MCO needs in order to pull the base operating system image from your repository. By default, the MCO uses the cluster global pull secret, which it synchronizes into the `openshift-machine-config-operator` namespace. You can add your pull secret to the {product-title} global pull secret or you can use a different pull secret. For information on modifying the global pull secret, see "Updating the global cluster pull secret".
|
||||
|
||||
* You have a copy of the pull secret in the `openshift-machine-config-operator` namespace that the MCO needs to pull the base operating system image.
|
||||
+
|
||||
For example, if you are using the global pull secret, you can run the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$oc create secret docker-registry global-pull-secret-copy \
|
||||
--namespace "openshift-machine-config-operator" \
|
||||
--from-file=.dockerconfigjson=<(oc get secret/pull-secret -n openshift-config -o go-template='{{index .data ".dockerconfigjson" | base64decode}}')
|
||||
----
|
||||
// https://issues.redhat.com/browse/OCPBUGS-42013
|
||||
|
||||
// Not in 4.18; maybe in 4.19
|
||||
// If you are using the global pull secret, the MCO automatically creates a copy when you first create a `MachineOSconfig` object.
|
||||
|
||||
* You have the push secret of the registry that the MCO needs to push the new custom layered image to.
|
||||
* You have the push secret of the registry that the MCO needs to push the new custom layered image to. The credentials provided by the secret must also grant permission to delete an image from the registry.
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
In a disconnected environment, ensure that the disconnected cluster can access the registry where you want to push the image. Image mirroring applies only to pulling images.
|
||||
====
|
||||
|
||||
* You have the push secret of the registry that the MCO needs to push the new custom layered image to. If you want the custom layered image deleted when you delete the associated the `MachineOSBuild` object, the credentials provided by the secret must also grant permission to delete an image from the registry.
|
||||
|
||||
* You have a pull secret that your nodes need to pull the new custom layered image from your registry. This should be a different secret than the one used to push the image to the repository.
|
||||
* You have the pull secret that your nodes need to pull the new custom layered image from your registry. This should be a different secret than the one used to push the image to the repository.
|
||||
|
||||
* You are familiar with how to configure a Containerfile. Instructions on how to create a Containerfile are beyond the scope of this documentation.
|
||||
|
||||
@@ -120,42 +120,37 @@ In a disconnected environment, ensure that the disconnected cluster can access t
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: machineconfiguration.openshift.io/v1alpha1 <1>
|
||||
apiVersion: machineconfiguration.openshift.io/v1 <1>
|
||||
kind: MachineOSConfig
|
||||
metadata:
|
||||
name: layered <2>
|
||||
name: layered-image <2>
|
||||
spec:
|
||||
machineConfigPool:
|
||||
name: worker <3>
|
||||
buildInputs:
|
||||
containerFile: <4>
|
||||
- containerfileArch: noarch
|
||||
content: |-
|
||||
FROM configs AS final
|
||||
RUN rpm-ostree install tree && \
|
||||
ostree container commit
|
||||
imageBuilder: <5>
|
||||
imageBuilderType: PodImageBuilder
|
||||
baseImagePullSecret: <6>
|
||||
name: global-pull-secret-copy
|
||||
renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest <7>
|
||||
renderedImagePushSecret: <8>
|
||||
name: builder-dockercfg-7lzwl
|
||||
buildOutputs: <9>
|
||||
currentImagePullSecret:
|
||||
name: builder-dockercfg-mtcl23
|
||||
name: layered <3>
|
||||
containerFile: # <4>
|
||||
- containerfileArch: NoArch <5>
|
||||
content: |-
|
||||
FROM configs AS final
|
||||
RUN dnf install -y cowsay && \
|
||||
dnf clean all && \
|
||||
ostree container commit
|
||||
imageBuilder: # <6>
|
||||
imageBuilderType: Job
|
||||
baseImagePullSecret: # <7>
|
||||
name: global-pull-secret-copy
|
||||
renderedImagePushSpec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest # <8>
|
||||
renderedImagePushSecret: # <9>
|
||||
name: builder-dockercfg-mtcl23
|
||||
----
|
||||
<1> Specifies the `machineconfiguration.openshift.io/v1` API that is required for `MachineConfig` CRs.
|
||||
<2> Specifies a name for the `MachineOSConfig` object. This name is used with other on-cluster layering resources. The examples in this documentation use the name `layered`.
|
||||
<3> Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. The examples in this documentation use the `worker` machine config pool.
|
||||
<2> Specifies a name for the `MachineOSConfig` object. This name is used with other on-cluster layering resources. The examples in this documentation use the name `layered-image`.
|
||||
<3> Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. The examples in this documentation use the `layered` machine config pool.
|
||||
<4> Specifies the Containerfile to configure the custom layered image.
|
||||
<5> Specifies the name of the image builder to use. This must be `PodImageBuilder`.
|
||||
<6> Specifies the name of the pull secret that the MCO needs in order to pull the base operating system image from the registry.
|
||||
<7> Specifies the image registry to push the newly built custom layered image to. This can be any registry that your cluster has access to. This example uses the internal {product-title} registry.
|
||||
<8> Specifies the name of the push secret that the MCO needs in order to push the newly built custom layered image to that registry.
|
||||
<9> Specifies the secret required by the image registry that the nodes need in order to pull the newly built custom layered image. This should be a different secret than the one used to push the image to your repository.
|
||||
// +
|
||||
// https://github.com/openshift/openshift-docs/pull/87486/files has the v1 api versions
|
||||
<5> Specifies the architecture this containerfile is to be built for: `ARM64`, `AMD64`, `PPC64LE`, `S390X`, or `NoArch`. The default is `NoArch`, which defines a Containerfile that can be applied to any architecture.
|
||||
<6> Specifies the name of the image builder to use. This must be `Job`, which is a reference to the `job` object that is managing the image build.
|
||||
<7> Optional: Specifies the name of the pull secret that the MCO needs to pull the base operating system image from the registry. By default, the global pull secret is used.
|
||||
<8> Specifies the image registry to push the newly-built custom layered image to. This can be any registry that your cluster has access to in the `host[:port][/namespace]/name` or `svc_name.namespace.svc[:port]/repository/name:<tag>` format. This example uses the internal {product-title} registry. You can specify a mirror registry if you cluster is properly configured to use a mirror registry.
|
||||
<9> Specifies the name of the push secret that the MCO needs to push the newly-built custom layered image to that registry.
|
||||
|
||||
.. Create the `MachineOSConfig` object:
|
||||
+
|
||||
@@ -176,8 +171,8 @@ $ oc get machineosbuild
|
||||
.Example output showing that the `MachineOSBuild` object is ready
|
||||
[source,terminal]
|
||||
----
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED
|
||||
layered-ad5a3cad36303c363cf458ab0524e7c0-builder False False True False False
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE
|
||||
layered-image-ad5a3cad36303c363cf458ab0524e7c0-builder False False True False False 43s
|
||||
----
|
||||
|
||||
.. Edit the nodes where you want to deploy the custom layered image by adding a label for the machine config pool you specified in the `MachineOSConfig` object:
|
||||
@@ -207,15 +202,15 @@ $ oc get pods -n openshift-machine-config-operator
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
build-layered-ad5a3cad36303c363cf458ab0524e7c0-hxrws 2/2 Running 0 2m40s # <1>
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
build-layered-image-ad5a3cad36303c363cf458ab0524e7c0-hxrws 2/2 Running 0 2m40s # <1>
|
||||
# ...
|
||||
machine-os-builder-6fb66cfb99-zcpvq 1/1 Running 0 2m42s # <2>
|
||||
machine-os-builder-6fb66cfb99-zcpvq 1/1 Running 0 2m42s # <2>
|
||||
----
|
||||
<1> This is the build pod where the custom layered image is building, named in the `build-<MachineOSConfig_CR_name>-<hash>` format.
|
||||
<2> This pod can be used for troubleshooting.
|
||||
|
||||
. Verify that the `MachineOSConfig` object contains a reference to the new custom layered image by running the following command:
|
||||
. Verify the current stage of your layered build by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
@@ -225,40 +220,32 @@ $ oc get machineosbuilds
|
||||
.Example output
|
||||
[source,terminal]
|
||||
----
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED
|
||||
layered-ad5a3cad36303c363cf458ab0524e7c0 False True False False False <1>
|
||||
NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE
|
||||
layered-image-ad5a3cad36303c363cf458ab0524e7c0 False True False False False 12m <1>
|
||||
----
|
||||
<1> The `MachineOSBuild` is named in the `<MachineOSConfig_CR_name>-<hash>` format.
|
||||
|
||||
. Verify that the `MachineOSBuild` object contains a reference to the new custom layered image by running the following command:
|
||||
. Verify that the `MachineOSConfig` object contains a reference to the new custom layered image by running the following command:
|
||||
+
|
||||
[source,terminal]
|
||||
----
|
||||
$ oc describe machineosbuild <object_name>
|
||||
$ oc describe machineosconfig <object_name>
|
||||
----
|
||||
+
|
||||
.Example output
|
||||
[source,yaml]
|
||||
.Example digested image pull spec
|
||||
[source,terminal]
|
||||
----
|
||||
Name: layered-ad5a3cad36303c363cf458ab0524e7c0
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineOSConfig
|
||||
metadata:
|
||||
annotations:
|
||||
machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45
|
||||
labels:
|
||||
machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: ""
|
||||
name: layered-image
|
||||
# ...
|
||||
API Version: machineconfiguration.openshift.io/v1alpha1
|
||||
Kind: MachineOSBuild
|
||||
# ...
|
||||
Spec:
|
||||
Config Generation: 1
|
||||
Desired Config:
|
||||
Name: rendered-layered-ad5a3cad36303c363cf458ab0524e7c0
|
||||
Machine OS Config:
|
||||
Name: layered
|
||||
Rendered Image Pushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:layered-ad5a3cad36303c363cf458ab0524e7c0
|
||||
# ...
|
||||
Last Transition Time: 2025-02-12T19:21:28Z
|
||||
Message: Build Ready
|
||||
Reason: Ready
|
||||
Status: True
|
||||
Type: Succeeded
|
||||
Final Image Pullspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b <1>
|
||||
status:
|
||||
currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 <1>
|
||||
----
|
||||
<1> Digested image pull spec for the new custom layered image.
|
||||
|
||||
@@ -290,8 +277,8 @@ sh-5.1# rpm-ostree status
|
||||
----
|
||||
# ...
|
||||
Deployments:
|
||||
* ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b
|
||||
Digest: sha256:312e48825e074b01a913deedd6de68abd44894ede50b2d14f99d722f13cda04b <1>
|
||||
Version: 418.94.202502100215-0 (2025-02-12T19:20:44Z)
|
||||
* ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754
|
||||
Digest: sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 <1>
|
||||
Version: 419.94.202502100215-0 (2025-02-12T19:20:44Z)
|
||||
----
|
||||
<1> Digested image pull spec for the new custom layered image.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
:_mod-docs-content-type: SNIPPET
|
||||
|
||||
Making certain changes to a `MachineOSConfig` object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools." While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build will still run regardless of whether the pool is paused or not. For example, if you want to remove and replace a `MachineOSCOnfig` object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed.
|
||||
Making certain changes to a `MachineOSConfig` object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools". While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build runs regardless of whether the pool is paused or not. For example, if you want to remove and replace a `MachineOSCOnfig` object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed.
|
||||
|
||||
When a machine config pool is paused, the `oc get machineconfigpools` reports the following status:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user