openshift-docs/machine_configuration/mco-coreos-layering.adoc

:_mod-docs-content-type: ASSEMBLY
include::_attributes/common-attributes.adoc[]
[id="mco-coreos-layering"]
= Image mode for OpenShift
:context: mco-coreos-layering

toc::[]

[role="_abstract"]
You can extend the functionality of your base {op-system} image by layering additional images onto the base image. This layering does not modify the base {op-system} image. Instead, it creates a _custom layered image_ that includes all {op-system} functionality and adds additional functionality to specific nodes in the cluster.

// Paragraph taken from Mark Russell draft (6/2/25) blog: https://docs.google.com/document/d/1DMaBg--3ljVlv0jqLJ4KjjFvGRA6J9gmF67Yq37uKdQ/edit?tab=t.0
Image mode is a cloud-native approach to operating system management that treats your OS like a container image. You define your operating system configuration as code, build it as a unified image, and deploy it consistently across your entire fleet.

[id="coreos-layering-about_{context}"]
== About {image-mode-os-lower}

{image-mode-os-caps} allows you to customize the underlying node operating system on any of your cluster nodes. This helps keep everything up-to-date, including the node operating system and any added customizations such as specialized software.

You create a custom layered image by using a Containerfile and applying it to nodes by using a custom object. At any time, you can remove the custom layered image by deleting that custom object.

With {image-mode-os-lower}, you can install RPMs into your base image, and your custom content will be booted alongside {op-system}. The Machine Config Operator (MCO) can roll out these custom layered images and monitor these custom containers in the same way it does for the default {op-system} image. {image-mode-os-caps} gives you greater flexibility in how you manage your {op-system} nodes.

// NOTE from https://issues.redhat.com/browse/OCPBUGS-2214?focusedCommentId=21430101&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-21430101

[IMPORTANT]
====
Installing realtime kernel and extensions RPMs as custom layered content is not recommended. This is because these RPMs can conflict with RPMs installed by using a machine config. If there is a conflict, the MCO enters a `degraded` state when it tries to install the machine config RPM. You need to remove the conflicting extension from your machine config before proceeding.
====

As soon as you apply the custom layered image to your cluster, you effectively _take ownership_ of your custom layered images and those nodes. While Red Hat remains responsible for maintaining and updating the base {op-system} image on standard nodes, you are responsible for maintaining and updating images on nodes that use a custom layered image. You assume the responsibility for the package you applied with the custom layered image and any issues that might arise with the package.

There are three methods for deploying a custom layered image onto your nodes:

{image-mode-os-on-caps}:: With xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on_mco-coreos-layering[{image-mode-os-on-lower}], you create a `MachineOSConfig` object where you include the Containerfile and other parameters. The build is performed on your cluster and the resulting custom layered image is automatically pushed to your repository and applied to the machine config pool that you specified in the `MachineOSConfig` object. The entire process is performed completely within your cluster.

{image-mode-os-out-caps}:: With xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring_mco-coreos-layering[{image-mode-os-out-lower}], you create a Containerfile that references an {product-title} image and the RPM that you want to apply, build the layered image in your own environment, and push the image to your repository. Then, in your cluster, create a `MachineConfig` object for the targeted node pool that points to the new image. The Machine Config Operator overrides the base {op-system} image, as specified by the `osImageURL` value in the associated machine config, and boots the new image.

During {product-title} installation:: You can apply a pre-built custom layered image to specific nodes during {product-title} installation. 

[IMPORTANT]
====
For these methods, use the same base {op-system} image installed on the rest of your cluster. Use the `oc adm release info --image-for rhel-coreos` command to obtain the base image used in your cluster.
====

[id="coreos-layering-examples_{context}"]
== Example Containerfiles

{image-mode-os-caps} allows you to use the following types of images to create custom layered images:

* *{product-title} Hotfixes*. You can work with Customer Experience and Engagement (CEE) to obtain and apply link:https://access.redhat.com/solutions/2996001[Hotfix packages] on top of your {op-system} image. In some instances, you might want a bug fix or enhancement before it is included in an official {product-title} release. {image-mode-os-caps} allows you to easily add the Hotfix before it is officially released and remove the Hotfix when the underlying {op-system} image incorporates the fix.
+
[IMPORTANT]
====
Some Hotfixes require a Red Hat Support Exception and are outside of the normal scope of {product-title} support coverage or life cycle policies.
====
+
Hotfixes are provided to you based on link:https://access.redhat.com/solutions/2996001[Red Hat Hotfix policy]. Apply it on top of the base image and test that new custom layered image in a non-production environment. When you are satisfied that the custom layered image is safe to use in production, you can roll it out on your own schedule to specific node pools. For any reason, you can easily roll back the custom layered image and return to using the default {op-system}.
+
.Example on-cluster Containerfile to apply a Hotfix
[source,yaml]
----
containerfileArch: noarch
content: |-
  FROM configs AS final
  #Install hotfix package
  RUN dnf update -y https://example.com/files/systemd-252-46.el9_4.x86_64.rpm \
                    https://example.com/files/systemd-journal-remote-252-46.el9_4.x86_64.rpm \
                    https://example.com/files/systemd-libs-252-46.el9_4.x86_64.rpm  \
                    https://example.com/files/systemd-pam-252-46.el9_4.x86_64.rpm \
                    https://example.com/files/systemd-udev-252-46.el9_4.x86_64.rpm \
                    https://example.com/files/systemd-rpm-macros-252-46.el9_4.noarch.rpm && \
      dnf clean all && \
      ostree container commit
----
+
.Example out-of-cluster Containerfile to apply a Hotfix
[source,yaml]
----
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256...
#Install hotfix package
RUN dnf update -y https://example.com/files/systemd-252-46.el9_4.x86_64.rpm \
                  https://example.com/files/systemd-journal-remote-252-46.el9_4.x86_64.rpm \
                  https://example.com/files/systemd-libs-252-46.el9_4.x86_64.rpm  \
                  https://example.com/files/systemd-pam-252-46.el9_4.x86_64.rpm \
                  https://example.com/files/systemd-udev-252-46.el9_4.x86_64.rpm \
                  https://example.com/files/systemd-rpm-macros-252-46.el9_4.noarch.rpm && \
    dnf clean all && \
    ostree container commit
----
// https://issues.redhat.com/browse/OCPBUGS-42838

* *{op-system-base} packages*. You can download {op-system-base-full} packages from the link:https://access.redhat.com/downloads/content/479/ver=/rhel---9/9.1/x86_64/packages[Red Hat Customer Portal], such as chrony, firewalld, and iputils.
+
.Example out-of-cluster Containerfile to apply the rsyslog utility
[source,yaml,subs="attributes+"]
----
# Using a 4.18.0 image
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256...
# Install rsyslog package
RUN dnf install -y rsyslog && \
    ostree container commit
# Copy your custom configuration in
ADD remote.conf /etc/rsyslog.d/remote.conf
----

* *Third-party packages*. You can download and install RPMs from third-party organizations, such as the following types of packages:
+
--
** Bleeding edge drivers and kernel enhancements to improve performance or add capabilities.
** Forensic client tools to investigate possible and actual break-ins.
** Security agents.
** Inventory agents that provide a coherent view of the entire cluster.
** SSH Key management packages.
--
+
.Example on-cluster Containerfile to apply a third-party package from EPEL
[source,yaml]
----
FROM configs AS final

#Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop
RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
    dnf install -y htop && \
    dnf clean all && \
    ostree container commit
----
+
.Example out-of-cluster Containerfile to apply a third-party package from EPEL
[source,yaml,subs="attributes+"]
----
# Get {op-system} base image of target cluster `oc adm release info --image-for rhel-coreos`
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256...

#Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop
RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
    dnf install -y htop && \
    dnf clean all && \
    ostree container commit
----
+
This Containerfile installs the {op-system-base} fish program. Because fish requires additional {op-system-base} packages, the image must be built on an entitled {op-system-base} host. For {op-system-base} entitlements to work, you must copy the `etc-pki-entitlement` secret into the `openshift-machine-config-operator` namespace. 
+
.Example on-cluster Containerfile to apply a third-party package that has {op-system-base} dependencies
[source,yaml]
----
FROM configs AS final

# RHEL entitled host is needed here to access RHEL packages
# Install fish as third party package from EPEL
RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && \
    dnf clean all && \
    ostree container commit
----
+
.Example out-of-cluster Containerfile to apply a third-party package that has {op-system-base} dependencies
[source,yaml,subs="attributes+"]
----
# Get {op-system} base image of target cluster `oc adm release info --image-for rhel-coreos`
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256...

# {op-system-base} entitled host is needed here to access {op-system-base} packages
# Install fish as third party package from EPEL
RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && \
    dnf clean all && \
    ostree container commit
----

After you create the machine config, the Machine Config Operator (MCO) performs the following steps:

. Renders a new machine config for the specified pool or pools.
. Performs cordon and drain operations on the nodes in the pool or pools.
. Writes the rest of the machine config parameters onto the nodes.
. Applies the custom layered image to the node.
. Reboots the node using the new image.

[IMPORTANT]
====
It is strongly recommended that you test your images outside of your production environment before rolling out to your cluster.
====

include::modules/coreos-layering-configuring-on.adoc[leveloffset=+1]

.Additional resources
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-proc_mco-coreos-layering[Using the {image-mode-os-on-lower} to apply a custom layered image]
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-remove_mco-coreos-layering[Removing an on-cluster custom layered image]
* xref:../updating/updating_a_cluster/update-using-custom-machine-config-pools.adoc#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools[Pausing the machine config pools]
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-rebuild_mco-coreos-layering[Rebuilding an on-cluster custom layered image]
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-revert_mco-coreos-layering[Reverting an on-cluster custom layered image]
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-modifying_mco-coreos-layering[Modifying a custom layered image]
* xref:../machine_configuration/index.adoc#checking-mco-node-status_machine-config-overview[About checking machine config node status]

include::modules/coreos-layering-configuring-on-proc.adoc[leveloffset=+2]

.Additional resources
* xref:../openshift_images/managing_images/using-image-pull-secrets.adoc#images-update-global-pull-secret_using-image-pull-secrets[Updating the global cluster pull secret]
* xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-configuring-on-revert_mco-coreos-layering[Reverting an on-cluster custom layered image]
* xref:../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features[Enabling features using feature gates]

include::modules/coreos-layering-configuring-on-modifying.adoc[leveloffset=+2]

.Additional resources
* xref:../updating/updating_a_cluster/update-using-custom-machine-config-pools.adoc#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools[Pausing the machine config pools]

////
Hiding extensions, not in 4.18. Maybe 4.19?
include::modules/coreos-layering-configuring-on-extensions.adoc[leveloffset=+2]

.Additional resources
* xref:../machine_configuration/machine-configs-configure.html#rhcos-add-extensions_machine-configs-configure[Adding extensions to RHCOS]
* xref:../updating/updating_a_cluster/update-using-custom-machine-config-pools.adoc#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools[Pausing the machine config pools]
////

include::modules/coreos-layering-configuring-on-rebuild.adoc[leveloffset=+2]

include::modules/coreos-layering-configuring-on-revert.adoc[leveloffset=+2]

include::modules/coreos-layering-configuring-on-remove.adoc[leveloffset=+2]

include::modules/coreos-layering-configuring.adoc[leveloffset=+1]

.Additional resources
xref:../machine_configuration/mco-coreos-layering.adoc#coreos-layering-updating_mco-coreos-layering[Updating with a {op-system} custom layered image]

include::modules/coreos-layering-removing.adoc[leveloffset=+2]

include::modules/coreos-layering-install-time.adoc[leveloffset=+1]

include::modules/coreos-layering-updating.adoc[leveloffset=+1]

////
Sources:
https://docs.google.com/document/d/1Eow2IReNWqnIh5HvCfcKV2MWgHUmFKSnBkt2rH6_V_M/edit
https://hackmd.io/OKc5ZnN7SQm3myHaGqksBg
https://github.com/openshift/enhancements/blob/master/enhancements/ocp-coreos-layering/ocp-coreos-layering.md
https://docs.google.com/document/d/1RbfCJuL_NBaWvUmd9nmiG8-7MwzsvWHjrIS2LglCYM0/edit
////