diff --git a/_topic_maps/_topic_map_ms.yml b/_topic_maps/_topic_map_ms.yml index e723cd99a6..3414f596df 100644 --- a/_topic_maps/_topic_map_ms.yml +++ b/_topic_maps/_topic_map_ms.yml @@ -393,6 +393,8 @@ Topics: File: microshift-using-config-tools - Name: Cluster access with kubeconfig File: microshift-cluster-access-kubeconfig +- Name: Using custom certificate authorities + File: microshift-custom-ca - Name: Checking the status of Greenboot health checks File: microshift-greenboot-checking-status --- diff --git a/microshift_configuring/microshift-custom-ca.adoc b/microshift_configuring/microshift-custom-ca.adoc new file mode 100644 index 0000000000..0de387f98b --- /dev/null +++ b/microshift_configuring/microshift-custom-ca.adoc @@ -0,0 +1,27 @@ +:_mod-docs-content-type: ASSEMBLY +[id="microshift-custom-ca"] += Configuring custom certificate authorities +include::_attributes/attributes-microshift.adoc[] +:context: microshift-custom-ca + +toc::[] + +You can encrypt connections by using custom certificate authorities (CAs) with the {microshift-short} service. + +include::modules/microshift-custom-ca-con.adoc[leveloffset=+1] + +include::modules/microshift-custom-ca-proc.adoc[leveloffset=+1] + +include::modules/microshift-custom-ca-reserved-names.adoc[leveloffset=+1] + +include::modules/microshift-custom-ca-troubleshooting.adoc[leveloffset=+1] + +[id="Additional-resources_microshift-custom-ca_{context}"] +== Additional resources +* link:https://docs.openshift.com/container-platform/{ocp-version}/security/certificates/api-server.html#customize-certificates-api-add-named_api-server-certificates[OpenShift: Add an API server named certificate] + +* link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/creating-and-managing-tls-keys-and-certificates_securing-networks#doc-wrapper[RHEL: Creating and managing TLS keys and certificates] + +* link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/using-shared-system-certificates_securing-networks#the-system-wide-trust-store_using-shared-system-certificates[The system-wide truststore] for details. + +* link:https://docs.openshift.com/container-platform/{ocp-version}/cli_reference/openshift_cli/managing-cli-profiles.html[OpenShift CLI Reference: oc login] \ No newline at end of file diff --git a/modules/microshift-custom-ca-con.adoc b/modules/microshift-custom-ca-con.adoc new file mode 100644 index 0000000000..2975d3e3c3 --- /dev/null +++ b/modules/microshift-custom-ca-con.adoc @@ -0,0 +1,45 @@ +// Module included in the following assemblies: +// +// * microshift_security_compliance/microshift-custom-ca.adoc + +:_mod-docs-content-type: CONCEPT +[id="microshift-custom-cas_{context}"] += How custom certificate authorities work in {microshift-short} + +The default API server certificate is issued by an internal {microshift-short} cluster certificate authority (CA). Clients outside of the cluster cannot verify the API server certificate by default. This certificate can be replaced by a custom server certificate that is issued externally by a custom CA that clients trust. The following steps illustrate the workflow in {microshift-short}: + +. Copy the certificates and keys to the preferred directory in the host operating system. Ensure that the files are accessible by root only. + +. Update the {microshift-short} configuration for each custom CA by specifying the certificate names and new fully qualified domain name (FQDN) in the {microshift-short} `/etc/microshift/config.yaml` configuration file. ++ +Each certificate configuration can contain the following values: + +** The certificate file location is a required value. +** A single common name containing the API server DNS and IP address or IP address range. ++ +-- +[TIP] +==== +In most cases, {microshift-short} generates a new `kubeconfig` for your custom CA that includes the IP address or range that you specify. The exception is when wildcards are specified for the IP address. In this case, {microshift-short} generates a `kubeconfig` with the public IP address of the server. To use wildcards, you must update the `kubeconfig` file with your specific details. +==== +-- +** Multiple Subject Alternative Names (SANs) containing the API server DNS and IP addresses or a wildcard certificate. +** You can provide additional DNS names for each certificate. + +. After the {microshift-short} service restarts, you must copy the generated `kubeconfig` files to the client. + +. Configure additional CAs on the client system. For example, you can update CA bundles in the {op-system-base-full} truststore. + +. The certificates and keys are read from the specified file location on the host. Testing and validation of configuration is done from the client. + +. External server certificates are not automatically renewed. You must manually rotate your external certificates. + +[NOTE] +==== +If any validation fails, the {microshift-short} service skips the custom configuration and uses the default certificate to start. The priority is to continue the service uninterrupted. {microshift-short} logs errors when the service starts. Common errors include expired certificates, missing files, or incorrect IP addresses. +==== + +[IMPORTANT] +==== +Custom server certificates have to be validated against CA data configured in the trust root of the host operating system. +==== diff --git a/modules/microshift-custom-ca-proc.adoc b/modules/microshift-custom-ca-proc.adoc new file mode 100644 index 0000000000..5147152795 --- /dev/null +++ b/modules/microshift-custom-ca-proc.adoc @@ -0,0 +1,141 @@ +// Module included in the following assemblies: +// +// * microshift_security_compliance/microshift-custom-ca.adoc + +:_mod-docs-content-type: PROCEDURE +[id="microshift-custom-cas-configuring_{context}"] += Configuring custom certificate authorities + +To configure externally generated certificates and domain names using custom certificate authorities (CAs), add them to the {microshift-short} `/etc/microshift/config.yaml` configuration file. You must also configure the host operating system trust root. + +[NOTE] +==== +Externally generated `kubeconfig` files are created in the `/var/lib/microshift/resources/kubeadmin//kubeconfig` directory. If you need to use `localhost` in addition to externally generated configurations, retain the original `kubeconfig` file in its default location. The `localhost` `kubeconfig` file uses the self-signed certificate authority. +==== + +.Prerequisites +* The OpenShift CLI (`oc`) is installed. +* You have access to the cluster as a user with the cluster administration role. +* The certificate authority has issued the custom certificates. +* A {microshift-short} `/etc/microshift/config.yaml` configuration file exists. + +.Procedure + +. Copy the custom certificates you want to add to the trust root of the {microshift-short} host. Ensure that the +certificate and private keys are only accessible to {microshift-short}. + +. For each custom CA that you need, add an `apiServer` section called `namedCertificates` to the `/etc/microshift/config.yaml` {microshift-short} configuration file by using the following example: ++ +[source,yaml] +---- +apiServer: + namedCertificates: + - certPath: ~/certs/api_fqdn_1.crt <1> + keyPath: ~/certs/api_fqdn_1.key <2> + - certPath: ~/certs/api_fqdn_2.crt + keyPath: ~/certs/api_fqdn_2.key + names: <3> + - api_fqdn_1 + - *.apps.external.com +---- +<1> Add the full path to the certificate. +<2> Add the full path to the certificate key. +<3> Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If no names are provided, the implicit names are extracted from the certificates. + +. Restart the {microshift-service} to apply the certificates by running the following command: ++ +[source,terminal] +---- +$ systemctl microshift restart +---- + +. Wait a few minutes for the system to restart and apply the custom server. New `kubeconfig` files are generated in the `/var/lib/microshift/resources/kubeadmin/` directory. + +. Copy the `kubeconfig` files to the client. If you specified wildcards for the IP address, update the `kubeconfig` to remove the public IP address of the server and replace that IP address with the specific wildcard range you want to use. + +. From the client, use the following steps: + +.. Specify the `kubeconfig` to use by running the following command: ++ +[source,terminal] +---- +$ export KUBECONFIG=~/custom-kubeconfigs/kubeconfig <1> +---- +<1> Use the location of the copied `kubeconfig` file as the path. + +.. Check that the certificates are applied by using the following command: ++ +[source,terminal] +---- +$ oc --certificate-authority ~/certs/ca.ca get node +---- ++ +.Example output +[source,terminal] +---- +oc get node +NAME STATUS ROLES AGE VERSION +dhcp-1-235-195.arm.example.com Ready control-plane,master,worker 76m v1.29.2 +---- + +.. Add the new CA file to the $KUBECONFIG environment variable by running the following command: ++ +[source,terminal] +---- +$ oc config set clusters.microshift.certificate-authority /tmp/certificate-authority-data-new.crt +---- + +.. Verify that the new `kubeconfig` file contains the new CA by running the following command: ++ +[source,terminal] +---- +$ oc config view --flatten +---- ++ +.Example externally generated `kubeconfig` file ++ +[source,yaml] +---- +apiVersion: v1 +clusters: +- cluster: + certificate-authority: /tmp/certificate-authority-data-new.crt <1> + server: https://api.ci-ln-k0gim2b-76ef8.aws-2.ci.openshift.org:6443 + name: ci-ln-k0gim2b-76ef8 +contexts: +- context: + cluster: ci-ln-k0gim2b-76ef8 + user: + name: +current-context: +kind: Config +preferences: {} +---- +<1> The `certificate-authority-data` section is not present in externally generated `kubeconfig` files. It is added with the `oc config set` command used previously. + +.. Verify the `subject` and `issuer` of your customized API server certificate authority by running the following command: ++ +[source,terminal] +---- +$ curl --cacert /tmp/caCert.pem https://${fqdn_name}:6443/healthz -v +---- ++ +.Example output +---- +Server certificate: + subject: CN=kas-test-cert_server + start date: Mar 12 11:39:46 2024 GMT + expire date: Mar 12 11:39:46 2025 GMT + subjectAltName: host "dhcp-1-235-3.arm.eng.rdu2.redhat.com" matched cert's "dhcp-1-235-3.arm.eng.rdu2.redhat.com" + issuer: CN=kas-test-cert_ca + SSL certificate verify ok. +---- ++ +[IMPORTANT] +==== +Either replace the `certificate-authority-data` in the generated `kubeconfig` file with the new `rootCA` or add the `certificate-authority-data` to the trust root of the operating system. Do not use both methods. +==== + +.. Configure additional CAs in the trust root of the operating system. For example, in the RHEL Client truststore on the client system. See link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/using-shared-system-certificates_securing-networks#the-system-wide-trust-store_using-shared-system-certificates[The system-wide truststore] for details. +** Updating the certificate bundle with the configuration that contains the CA is recommended. +** If you do not want to configure your certificate bundles, you can alternately use the `oc login localhost:8443 --certificate-authority=/path/to/cert.crt` command, but this method is not preferred. diff --git a/modules/microshift-custom-ca-reserved-names.adoc b/modules/microshift-custom-ca-reserved-names.adoc new file mode 100644 index 0000000000..30160e5bb9 --- /dev/null +++ b/modules/microshift-custom-ca-reserved-names.adoc @@ -0,0 +1,27 @@ +// Module included in the following assemblies: +// +// * microshift_security_compliance/microshift-custom-ca.adoc + +:_mod-docs-content-type: REFERENCE +[id="microshift-custom-ca-reserved-name-values_{context}"] += Custom certificates reserved name values + +The following certificate problems cause {microshift-short} to ignore certificates dynamically and log an error: + +* The certificate files do not exist on the disk or are not readable. +* The certificate is not parsable. +* The certificate overrides the internal certificates IPAddress/DNSNames in a `SubjectAlternativeNames` (SAN) field. Do not use a reserved name when configuring SANs. + +.Reserved Names values +[cols="<,<,<",options="header",] +|=== +|Address |Type |Comment +|`localhost` |DNS | +|`127.0.0.1` |IP Address | +|`10.42.0.0` |IP Address |Cluster Network +|`10.43.0.0/16,10.44.0.0/16` |IP Address |Service Network +|169.254.169.2/29 |IP Address |br-ex Network +|`kubernetes.default.svc` |DNS | +|`openshift.default.svc` |DNS | +|`svc.cluster.local` |DNS | +|=== diff --git a/modules/microshift-custom-ca-troubleshooting.adoc b/modules/microshift-custom-ca-troubleshooting.adoc new file mode 100644 index 0000000000..c78543dfce --- /dev/null +++ b/modules/microshift-custom-ca-troubleshooting.adoc @@ -0,0 +1,31 @@ +// Module included in the following assemblies: +// +// * microshift_security_compliance/microshift-custom-ca.adoc + +:_mod-docs-content-type: PROCEDURE +[id="microshift-custom-ca-troubleshootin_{context}"] += Troubleshooting custom certificates + +To troubleshoot the implementation of custom certificates, you can take the following steps. + +.Procedure + +. From {microshift-short}, ensure that the certificate is served by the `kube-apiserver` and verify that the certificate path is appended to the `--tls-sni-cert-key` FLAG by running the following command: ++ +[source,terminal] +---- +$ journalctl -u microshift -b0 | grep tls-sni-cert-key +---- ++ +.Example output +[source,terminal] +---- +Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099 45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key +---- + +. From the client, ensure that the `kube-apiserver` is serving the correct certificate by running the following command: ++ +[source,terminal] +---- +$ openssl s_client -connect :6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN" +----